Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition (1402.1128v1)

Abstract: Long Short-Term Memory (LSTM) is a recurrent neural network (RNN) architecture that has been designed to address the vanishing and exploding gradient problems of conventional RNNs. Unlike feedforward neural networks, RNNs have cyclic connections making them powerful for modeling sequences. They have been successfully used for sequence labeling and sequence prediction tasks, such as handwriting recognition, LLMing, phonetic labeling of acoustic frames. However, in contrast to the deep neural networks, the use of RNNs in speech recognition has been limited to phone recognition in small scale tasks. In this paper, we present novel LSTM based RNN architectures which make more effective use of model parameters to train acoustic models for large vocabulary speech recognition. We train and compare LSTM, RNN and DNN models at various numbers of parameters and configurations. We show that LSTM models converge quickly and give state of the art speech recognition performance for relatively small sized models.

• The paper introduces novel LSTM architectures that overcome traditional RNN limitations in large vocabulary speech recognition tasks.
• It employs recurrent and non-recurrent projection layers to optimize parameter usage, yielding improved frame accuracy and reduced word error rates.
• Experimental results demonstrate faster convergence and enhanced stability, highlighting the approach's promise for real-world speech applications.

Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition

The paper by Sak, Senior, and Beaufays introduces novel Long Short-Term Memory (LSTM) based Recurrent Neural Network (RNN) architectures aimed at enhancing large vocabulary speech recognition (LVSR) systems. The focus is on overcoming limitations of conventional RNNs and demonstrating that LSTMs can be effectively scaled for complex, high-dimensional tasks such as LVSR.

Introduction and Background

LSTMs are a specialized type of RNN designed to mitigate the issues of vanishing and exploding gradients, which affect traditional RNNs during back-propagation through time (BPTT). While RNNs have shown promise in various sequence modeling tasks, their use in large-scale speech recognition has been limited. This paper proposes new LSTM architectures to make better use of model parameters and improve computational efficiency. The authors argue that previous applications of LSTMs to speech recognition were restricted to smaller problems like phone recognition on the TIMIT database and required supplementary models to outperform Deep Neural Networks (DNNs).

Proposed LSTM Architectures

The paper introduces two new LSTM architectures involving projection layers to mitigate the computational expense of scaling LSTMs:

1. LSTM with Recurrent Projection Layer: This architecture includes an intermediate recurrent projection layer between the LSTM layer and the output layer. The recurrent projection layer helps maintain the efficacy of the LSTM model while enhancing computational efficiency.
2. LSTM with Recurrent and Non-Recurrent Projection Layers: This model incorporates both recurrent and non-recurrent projection layers, providing more flexibility by increasing projection layer size without inflating the number of recurrent connections. This decoupling optimizes parameter usage and enables the model to handle more complex tasks.

The LSTM models utilize specific units such as input, forget, and output gates, alongside memory cells and recurrent/non-recurrent projection layers, leading to improved temporal context capturing and output precision.

Implementation Details

The researchers adopted a CPU-based, multi-core implementation using the Eigen matrix library, which facilitated efficient matrix operations through vectorization (SIMD). They utilized asynchronous stochastic gradient descent (ASGD) to optimize the training process, effectively managing computational resources and enhancing parallelism. The model employed truncated BPTT for sequence processing, further optimizing learning efficiency.

Experimental Setup and Results

The authors conducted extensive experiments comparing DNN, RNN, and LSTM architectures on the Google English Voice Search task, which includes a large dataset of 3 million utterances (about 1900 hours). Evaluation metrics included frame accuracies and word error rates (WERs) for various context-dependent (CD) state inventories: 126, 2000, and 8000 states.

Key Findings:

• Stability and Convergence: LSTMs demonstrated superior stability and faster convergence compared to conventional RNNs, particularly mitigating the exploding gradient problem.
• Frame Accuracy: LSTM models consistently outperformed both DNNs and RNNs across all state inventories. Architectures with projection layers showed significant accuracy improvements.
• Word Error Rate (WER): LSTM architectures provided significant reductions in WER, outperforming DNNs across different state configurations. This highlights their potential for real-world large vocabulary speech recognition applications.

Conclusions and Future Work

The paper establishes that LSTM architectures can be effectively scaled for large vocabulary speech recognition, demonstrating superior performance over traditional DNN models. The introduction of recurrent and non-recurrent projection layers significantly enhances the computational efficiency and accuracy of LSTM models.

Future directions could involve exploring GPU-based and distributed CPU-based implementations to further scale the LSTM models for larger datasets and more complex tasks. This approach could enable more extensive and real-time applications of LSTMs in speech recognition and other sequence modeling domains.

In summary, the proposed LSTM-based RNN architectures represent a significant advancement in the field of speech recognition, providing a robust model that can handle the complexities and large output spaces required for high-accuracy, large vocabulary tasks.