Cross-data recognition on sign language letters based on convolutional neural networks

Abstract

With an estimated 1.5 billion hearing-impaired people globally, sign language is a vital means of communication among them. Meanwhile, the complexity and diversity of sign languages across different regions bring challenges to researchers. As existing studies in this field usually lack the implementation of recognition models with multiple datasets, which limits their practical value in real-life scenarios, this study intends to get around this constraint. This work constructs a baseline American Sign Language Letter Recognition model using Convolutional Neural Network (CNN) and then optimizes it to enhance its ability. Finally, cross-data recognition is carried out. In order to train and evaluate the model, this study gathers data from several sources, including the MNIST sign language set and real-life photos. It also investigates how data augmentation affects recognition ability. Consequently, the CNN model is able to recognize hand gestures for different alphabetic letters in solid backgrounds. Its accurate rate of it reaches about 99.83%. For the extra real-life dataset, which contains 96 images, the model could still detect the majority of the images with sign language equivalents, despite some accuracy loss in more crowded backgrounds. This work, in general, concentrates on the potential of CNNs for sign language identification and highlights the significance of cross-data recognition in creating useful recognition models.