Research of a Sign Language Translation System Based on Deep Learning

Abstract

Sign language is a visual language that uses hand gesture, change of hand shape and track information to express meaning, and is the main communication tool for people with hearing and language impairment. Sign language recognition can improve the problem that the number of people who need to use sign language is large but the popularity of sign language is poor, and provide a more convenient way of study, work and life for people with hearing and language impairment. Hand locating and sign language recognition methods can generally be divided into traditional methods and deep learning methods. In recent years, with the brilliant achievements of deep learning in the field of computer vision, it has been proved that the method based on deep learning has many advantages, such as rich feature extraction, strong modeling ability and intuitive training. Therefore, this paper studies hand locating and sign language recognition of common sign language based on neural network, and the main research contents include: 1. A hand locating network based on the Faster R-CNN is established to recognize the sign language video or the part of the hand in the picture, and the result of recognition is handed over to subsequent processing; 2. A 3D CNN feature extraction network and a sign-language recognition framework based on long and short time memory (LSTM) coding and decoding network are constructed for the sign language images of sequence sequences. The framework can improve the recognition accuracy by learning the context of sign language; 3. In order to solve the problem of RGB sign language image or video recognition in practical problems, this paper combines hand locating network, 3D CNN feature extraction network and LSTM encoding and decoding to build the recognition algorithm. Experimental results show that the recognition rate of this method is up to 99% in common vocabulary data set, which is better than other known methods.