Abstract
In recent years, end-to-end automatic speech recognition (ASR) based on deep neural networks become popular because of its simple pipeline and excellent performance. However, there exists a main mismatch between its training and testing that might lead to performance degradation: in the training stage, existing method use the maximum likelihood criterion which aims to maximize log-likelihood of the training data, while in the testing stage the performance is evaluated by word error rate (WER), not log-likelihood. In this paper, we propose an alternative method based on reinforcement learning to make the goals of training and testing more consistent. Viewing speech recognition as a sequence decision process, the encoder-decoder based neural network is used as the policy function. The encoder is a pre-trained speech representation model (Wav2vec2.0), which generates the environment state encoding. The decoder is trained using a policy gradient algorithm based on a mix reward function which reflects both the word error rate and language model score. Experimental results on the LibriSpeech corpus show that our proposed method achieves 4% relative improvements over the baseline with a language model in terms of WER.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.