Abstract
In recent years, deep learning has already been applied to English lip-reading. However, Chinese lip-reading starts late and lacks relevant dataset, and the recognition accuracy is not ideal. Therefore, this paper proposes a new hybrid neural network model to establish a Chinese lip-reading system. In this paper, we integrate the attention mechanism into both CNN and RNN. Specifically, we add the convolutional block attention module (CBAM) to the ResNet50 neural network, which enhances its ability to capture the small differences among the mouth patterns of similarly pronounced words in Chinese, improving the performance of feature extraction in the convolution process. We also add the time attention mechanism to the GRU neural network, which helps to extract the features among consecutive lip motion images. Considering the effects of the moments before and after on the current moment in the lip-reading process, we assign more weights to the key frames, which makes the features more representative. We further validate our model through experiments on our self-built dataset. Our experiments show that using convolutional block attention module (CBAM) in the Chinese lip-reading model can accurately recognize Chinese numbers 0–9 and some frequently used Chinese words. Compared with other lip-reading systems, our system has better performance and higher recognition accuracy.
Highlights
Lip-reading is a human-computer interaction technology based on AI [1] that has been widely used
Because the details of Chinese lip pronunciation are not obvious, it is difficult to extract image features. e traditional convolutional neural networks (CNNs) networks cannot extract all the information from the lip image. erefore, in order to solve this problem and improve the accuracy of the Chinese lipreading, in this work, we propose a Chinese lip-reading system based on the convolutional block attention module. is system consists of three parts: First, the ResNet50 network with a convolutional block attention module[16]. is module (CBAM) consists of two parts, the channel attention and the spatial attention. e channel attention is mainly to compress the feature map in the spatial dimension
We find that the output of the convolutional neural network is H(x) F(x) + x, so the mapping of the residual neural network is F(x) H(x) − x; the fitting performance of F(x) is better. e residual structure of identity mapping can effectively avoid the problems of gradient disappearance and performance degradation during training
Summary
Lip-reading is a human-computer interaction technology based on AI [1] that has been widely used. En, Petajan [7] firstly proposed the concept of lip-reading system. Based on his theory, Goldeschen [8] combined Petajan’s work with the Hidden Markov Model and proposed a method of lipreading using the dynamic characteristics of lips as input to a Markov chain. Artificial neural network (ANN) [9] based on deep learning, which is growing increasingly popular, has gradually been introduced into the field of lip-reading. In 2018, Burton et al [11] and others used CNN and LSTM as a deep learning network for lip-reading to solve the complex speech recognition problem that the HMM network cannot. In 2021, Hussein et al [13] improved this model and proposed an HLR-Net model mainly composed of the Inception, BiGRU, and Attention Mechanism. e author used the Mathematical Problems in Engineering
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
