Vision-based Sign Language Recognition System Research Articles

Sign language recognition from digital videos using feature pyramid network with detection transformer

Sign language recognition is one of the fundamental ways to assist deaf people to communicate with others. An accurate vision-based sign language recognition system using deep learning is a fundamental goal for many researchers. Deep convolutional neural networks have been extensively considered in the last few years, and a slew of architectures have been proposed. Recently, Vision Transformer and other Transformers have shown apparent advantages in object recognition compared to traditional computer vision models such as Faster R-CNN, YOLO, SSD, and other deep learning models. In this paper, we propose a Vision Transformer-based sign language recognition method called DETR (Detection Transformer), aiming to improve the current state-of-the-art sign language recognition accuracy. The DETR method proposed in this paper is able to recognize sign language from digital videos with a high accuracy using a new deep learning model ResNet152 + FPN (i.e., Feature Pyramid Network), which is based on Detection Transformer. Our experiments show that the method has excellent potential for improving sign language recognition accuracy. For instance, our newly proposed net ResNet152 + FPN is able to enhance the detection accuracy up to 1.70% on the test dataset of sign language compared to the standard Detection Transformer models. Besides, an overall accuracy 96.45% was attained by using the proposed method.

Optimization of Modified Hidden Markov Model for Vision-Based Indonesian Sign Languages Recognition

Sign language is one of the most popular language which is used as a communication bridge that depends on hand movement. As it is used worldwide, the variety of sign languages is very large and most of the people doesn’t understand it which makes the communication between deaf and normal people interrupted. This makes sign language recognition popular as it makes people doesn’t need to understand the sign but still understand the meaning of the sign. But sign language itself has many problems such as the possibility of different dataset has the same movement but different meaning, the method used of each dataset could be ineffective in other dataset, and many other else which makes it difficult to be implemented. Other than that, vision-based recognition is not as popular as sensor-based recognition because of the difference in feature accuracy even though it could give more area of improvement. That’s why the aim of this paper is to presents the combination of methods used to recognize vision-based Indonesian Sign Language and enhance the method using optimization technique. The methodology used in this study follows four steps framework of sign language recognition which is dataset collection, preprocessing, feature extraction, and recognition. Each method is improved in order to be compared and checked what is the method combination and optimization method is the best for sign language recognition. For dataset collection, the dataset that is used is formal Indonesian Sign Language which is called Sistem Isyarat Bahasa Indonesia (SIBI) with some constraint in order to make sure the quality of the dataset is good. The preprocessing methods are differentiated into two categories which is image enhancement and hand detection. Image enhancement methods include no image enhancement, Gaussian blur, Bilateral blur, and Contrast Limited Adaptive Histogram Equalization and hand detection methods include skin detection using YCbCr color space and edge detection using Canny algorithm. Feature that is used in this study is pair of left and right-side movement for each frame that is extracted by calculating the average value of each pixels from left and right-side image that could be a representative value of each frame. Lastly the recognition system that is used is Gaussian Hidden Markov Model and its’ state optimization which includes no optimization, Latin Hypercube optimization, Hill Climbing optimization, and Combination of Latin Hypercube and Hill Climbing optimization. The experiment result of the proposed method could recognize up to 82% accuracy rate. The improvement of this research could be implemented on educational studies or game development that need vision-based sign language recognition system.

Sign language recognition in complex background scene based on adaptive skin colour modelling and support vector machine

With the advances of wearable cameras, the user can record the first-person view videos for gesture recognition or even sign language recognition to help the deaf or hard of hearing people communicate with others. In this paper, we propose a purely vision-based sign language recognition system which can be used in complex background scene. We design an adaptive skin colour modelling method for hand segmentation so that the hand contour can be derived more accurately even when different users use our system in various light conditions. Four kinds of feature descriptors are integrated to describe the contours and the salient points of hand gestures, and support vector machine (SVM) is applied to classify hand gestures. Our recognition method is evaluated by two datasets: 1) the CSL dataset collected by ourselves in which images were captured in three different environments including complex background; 2) the public ASL dataset, in which images of the same gesture were captured in different lighting conditions. The proposed recognition method achieves acceptable accuracy rates of 100.0% and 94.0% for the CSL and ASL datasets, respectively.

Sign language recognition in complex background scene based on adaptive skin colour modelling and support vector machine

With the advances of wearable cameras, the user can record the first-person view videos for gesture recognition or even sign language recognition to help the deaf or hard of hearing people communicate with others. In this paper, we propose a purely vision-based sign language recognition system which can be used in complex background scene. We design an adaptive skin colour modelling method for hand segmentation so that the hand contour can be derived more accurately even when different users use our system in various light conditions. Four kinds of feature descriptors are integrated to describe the contours and the salient points of hand gestures, and support vector machine (SVM) is applied to classify hand gestures. Our recognition method is evaluated by two datasets: 1) the CSL dataset collected by ourselves in which images were captured in three different environments including complex background; 2) the public ASL dataset, in which images of the same gesture were captured in different lighting conditions. The proposed recognition method achieves acceptable accuracy rates of 100.0% and 94.0% for the CSL and ASL datasets, respectively.