Sign Language Recognition Based on Computer Vision

Abstract

The study of sign language involves the intersection of many fields and disciplines. At present, the two mainstream research directions of sign language recognition are data gloves <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[1]</sup> and visual sign language recognition <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[2]</sup> . The former uses the data collected by the sensor for sign language recognition and translation, while the latter uses the camera to capture the user's hand characteristics for sign language recognition and translation. In this paper, an improved convolutional neural network (CNN) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[3]</sup> and long short-term memory(LSTM) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[4]</sup> neural network combined sign language recognition system, which is different from the current only for sign language recognition and translation, but also for sign language generation function is designed. For the first time, this system uses a PyQt designed GUI interface. Once in the system, users can select sign language recognition and translation capabilities, capture images via OpenCV, and then use the trained CNN neural network for special processing. The model can then identify American sign language through LSTM decisions. The user can also click the voice button, the system will be based on the user's voice to convert the corresponding gesture image into the same pixels, and write to the video file. Experimental results show that sign language recognition rate is 95.52% compared with similar algorithms [5], and sign language <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[6]</sup> (American sign language and Arabic numerals) is 90.3%.