Continuous sign language recognition based on 3DCNN and BLSTM

Abstract

Sign language recognition can make the communication between deaf mutes and healthy people more convenient and fast. In recent years, with the continuous development of deep learning, the research method of deep learning has also been introduced into the field of sign language recognition. Compared with the recognition of isolated words, the recognition of continuous sign language is more time-dependent. The current research still has shortcomings in recognition accuracy. Therefore, we proposed a continuous sign language recognition method based on 3DCNN and BLSTM. Based on the spatial feature information extracted by 3DCNN and the short-term temporal relationship established, the global temporal modeling of the video information of continuous sign language is carried out by using the bidirectional semantic mining ability of BLSTM. The CTC loss function is constructed to solve the problem of time series label misalignment. At the same time, we add the calculation of auxiliary loss function and auxiliary classifier. Experiments show that the auxiliary loss function and classifier can effectively reduce the error rate of the network. The word error rate of the continuous sign language recognition algorithm proposed in this paper on the large continuous sign language dataset RWTH-PHONEIX-Weather 2014 is as low as 23.5%, which is lower than the previous algorithm.