End-to-end speech recognition system based on improved CLDNN structure

Abstract

In the field of end-to-end speech recognition technology based on deep learning, CLDNN (Convolutional Long Short-Term Memory Fully Connected Deep Neural Network) is a commonly used model structure. The fully connected LSTM (Long Short Term Memory) model is used in the traditional CLDNN structure to process the timing information in the speech signal, which is prone to over-fitting during the training process and affects the learning effect. Deeper models tend to perform better, but increasing the model depth by simply stacking the network layers can cause gradient disappearance, gradient explosion, and degeneration problems. Aiming at the above phenomena and problems, this paper proposes an improved CLDNN structure. It combines the residual network and ConvLSTM to establish the residual ConvLSTM model, and replaces the fully connected LSTM model in the traditional CLDNN structure. The model structure solves the problems of the traditional CLDNN model, and can increase the model depth by stacking residual ConvLSTM blocks without gradient disappearance, gradient explosion and degeneration problems, which makes the speech recognition system perform better. The experimental results show that the model structure has a word error rate (WER) decrease of more than 8% in both Chinese and English speech recognition tasks compared to the traditional CLDNN structure.