A multi-lingual sign language recognition system using machine learning

Abstract

AbstractRecently, automatic sign language recognition field gets a great attention. Machine learning algorithms are mainly used to recognize hand gestures. Most of recent studies train their machine learning model using a specific sign language of a specific country such as the American Sign Language. In this paper, we propose a multi-lingual sign language system based machine learning that is called Multi-lingual Sign Languages Interpreter (MSLI) system. MSLI trains a machine learning model based on hand signs of multiple languages. It can detect the language of the input signs and their labels. In a case of input testing signs with the same language, the proposed system can provide two-steps recognition, where it only detects the language of the first sign, and then the rest signs are tested according to the recognized language. Also, MSLI can provide separate classification of signs per each language. Experiments were performed using 11 datasets with different languages. Separate and combined classification was performed on the input data. Experimental results show the accuracy of the proposed system. Training accuracy of the proposed system over most of the used separate different sign language datasets is approximately ranged from 90 to 100%. Also, most classification accuracy results of the test data of the separate datasets exceeded 90%. The combined classification of proposed MSLI archived training accuracy of 95.87% and testing accuracy of 92.33%.