Signer-independent Sign Language Recognition Research Articles

Signer-independent sign language recognition with feature disentanglement

Signer-independent sign language recognition with feature disentanglement

A signer-independent sign language recognition method for the single-frequency dataset

Currently, there are over 70 million people worldwide using more than 300 sign languages for communication, resulting in a vast number of sign language categories. Sign language recognition faces two main challenges. Firstly, in real-world applications, sign language users may not be represented in the dataset, leading to weak recognition capabilities of the models. Secondly, constructing large-scale sign language datasets is time-consuming and labor-intensive. Additionally, existing sign language recognition models have complex structures, leading to severe overfitting issues in the single-frequency dataset. To address these challenges, we construct a signer-independent learning sign language recognition method for the single-frequency dataset. In this work, a SwC GR-MMixer model, which relies on Gated Recurrent Unit (GRU) and Multi-Layer Perceptron (MLP) is developed, resulting in significantly reduced model complexity. After extensive ablation experiments, the most suitable structure for the SwC GR-MMixer model in the single-frequency dataset, as well as data augmentation methods, were determined. We achieved the best performance so far on the CSL-500 dataset and tackled the challenges of recognizing sign language, including the limitations imposed by the independence of the signer and the paucity of data on single-frequency datasets (e.g., CSL-500 dataset with only one demonstration per signer) by using a mask replacement method. Leveraging spatial feature extraction method, signer-independent sign language recognition tasks are accomplished, achieving a 6.95% improvement in the signer-independent method on the LSA64 dataset.

Signer-Independent Sign Language Recognition with Adversarial Neural Networks

Sign Language Recognition (SLR) has become an appealing topic in modern societies because such technology can ideally be used to bridge the gap between deaf and hearing people. Although important steps have been made towards the development of real-world SLR systems, signer-independent SLR is still one of the bottleneck problems of this research field. In this regard, we propose a deep neural network along with an adversarial training objective, specifically designed to address the signer-independent problem. Specifically, the proposed model consists of an encoder, mapping from input images to latent representations, and two classifiers operating on these underlying representations: (i) the sign-classifier, for predicting the class/sign labels, and (ii) the signer-classifier, for predicting their signer identities. During the learning stage, the encoder is simultaneously trained to help the sign-classifier as much as possible while trying to fool the signer-classifier. This adversarial training procedure allows learning signer-invariant latent representations that are in fact highly discriminative for sign recognition. Experimental results demonstrate the effectiveness of the proposed model and its capability of dealing with the large inter-signer variations.

DeepArSLR: A Novel Signer-Independent Deep Learning Framework for Isolated Arabic Sign Language Gestures Recognition

Hand gesture recognition has attracted the attention of many researchers due to its wide applications in robotics, games, virtual reality, sign language and human-computer interaction. Sign language is a structured form of hand gestures and the most effective communication way among hear-impaired people. Developing an efficient sign language recognition system to recognize dynamic isolated gestures encounters three major challenges, namely, hand segmentation, hand shape feature representation and gesture sequence recognition. Traditional sign language recognition methods utilize color-based hand segmentation algorithms to segment hands, hand-crafted feature extraction for hand shape representation and Hidden Markov Model (HMM) for sequence recognition. In this paper, a novel framework is proposed for signer-independent sign language recognition using multiple deep learning architectures comprising hand semantic segmentation, hand shape feature representation and deep recurrent neural network. The recently developed semantic segmentation method called DeepLabv3+ is trained using a set of pixel-labeled hand images to extract hand regions from each frame of the input video. Then, the extracted hand regions are cropped and scaled to a fixed size to alleviate hand scale variations. Extracting hand shape features is achieved using a single layer Convolutional Self-Organizing Map (CSOM) instead of relying on transfer learning of pre-trained deep convolutional neural networks. The sequence of extracted feature vectors are then recognized using deep Bi-directional Long Short-Term Memory (BiLSTM) recurrent neural network. BiLSTM network contains three BiLSTM layers, one fully connected and softmax layers. The performance of the proposed method is evaluated using a challenging Arabic sign language database containing 23 isolated words captured from three different users. Experimental results show that the performance of proposed framework outperforms with large margin the state-of-the-art methods for signer-independent testing strategy.

A Signer Independent Sign Language Recognition with Co-articulation Elimination from Live Videos: An Indian Scenario

Due to the high population of hearing impaired and vocal disabled people in India, a sign language interpretation system is becoming highly important for minimizing their isolation in society. This paper proposes a signer independent novel vision-based gesture recognition system which is capable of recognizing single handed static and dynamic gestures, double-handed static gestures and finger spelling words of Indian Sign Language (ISL) from live video. The use of Zernike moments for key frame extraction reduces the computation speed to a large extent. It also proposes an improved method for co-articulation elimination in fingerspelling alphabets. The gesture recognition module comprises mainly three steps – Preprocessing, Feature Extraction, and Classification. In the preprocessing phase, the signs are extracted from a real-time video using skin color segmentation. An appropriate feature vector is extracted from the gesture sequence after co-articulation elimination phase. The obtained features are then used for classification using Support Vector Machine(SVM). The system successfully recognized finger spelling alphabets with 91% accuracy and single-handed dynamic words with 89% accuracy. The experimental results show that the system has a better recognition rate compared to some of the existing methods.

An improved adaptation algorithm for signer-independent sign language recognition

An improved adaptation algorithm for signer-independent sign language recognition

An improved adaptation algorithm for signer-independent sign language recognition

Sign language recognition is a technology that can present sign language in a understandable form, in order to achieve barrier-free communication between deaf and normal. To solve the differences in sign language data issues and the lack of training samples of manpower caused by low recognition of non-specific language, presents MLLR\MAP adaptive progressive non-specific integrated manpower language recognition framework. This approach optimises the division MLLR regression class to provide more accurate initial MAP model, which give full play to the rapidity and the MAP MLLR progressive. Then introduced MCE model parameter estimation algorithm to compensate for the limitations of the model parameters adaptive method to further reduce the system error rate and accelerate the recognition speed. Meanwhile, for the MCE algorithm computationally intensive problems proposed improvements. Experimental results show that the adaptive sign language data required for this algorithm is less than traditional MLLR and MAP methods, while improved average recognition rate by 15.6%.

Online Early-Late Fusion Based on Adaptive HMM for Sign Language Recognition

In sign language recognition (SLR) with multimodal data, a sign word can be represented by multiply features, for which there exist an intrinsic property and a mutually complementary relationship among them. To fully explore those relationships, we propose an online early-late fusion method based on the adaptive Hidden Markov Model (HMM). In terms of the intrinsic property, we discover that inherent latent change states of each sign are related not only to the number of key gestures and body poses but also to their translation relationships. We propose an adaptive HMM method to obtain the hidden state number of each sign by affinity propagation clustering. For the complementary relationship, we propose an online early-late fusion scheme. The early fusion (feature fusion) is dedicated to preserving useful information to achieve a better complementary score, while the late fusion (score fusion) uncovers the significance of those features and aggregates them in a weighting manner. Different from classical fusion methods, the fusion is query adaptive. For different queries, after feature selection (including the combined feature), the fusion weight is inversely proportional to the area under the curve of the normalized query score list for each selected feature. The whole fusion process is effective and efficient. Experiments verify the effectiveness on the signer-independent SLR with large vocabulary. Compared either on different dataset sizes or to different SLR models, our method demonstrates consistent and promising performance.

Signer-Independent SLR System Using PCA and Multi-Class SVM

The aim of Sign Language Recognition (SLR) is to provide accurate and an efficient mechanism to transliterate sign language into a text or speech. State of the art SLR must be able to solve the signer-independent problem for realistic applications. This paper presents a novel method to recognize 32 unique American Sign Language (ASL) letters and numbers from images of signs, independent of signer and environment of image capture. Input images are mapped to the YCbCr colour space, binarized and resized to 70x70 pixels. Principal Component Analysis (PCA) is then performed on these binary images using their pixels as features. This method recognized signer-dependent signs with an accuracy of 100% and signer-independent signs with an accuracy of 62.37% for 32 unique classes and which increases to 78.49% when considering 23 classes, with very similar signs grouped together. Multi class Support Vector Machine (SVM) were employed as a classification method after processing the images, giving a result of 39.78% for 32 unique classes and 55.91% for 23 classes. This paper evaluates the efficiency of feature extraction method PCA on ASL hand gestures. To evaluate the impact of features on sign recognition rate, classifiers such as minimum distance, SVM is used. Experimental trials indicate higher recognition rate for PCA in comparison to those of other techniques and also previous works on ASL recognition.

Synthetic data generation technique in Signer-independent sign language recognition

The lack of training samples is an important problem in the field of sign language recognition. This paper presents a method of generating synthetic multi-stream samples so as to enlarge the training set of sign. The mean shift algorithm is able to obtain the directions of maximum increase and decrease in the density function, so it is used to control the direction and the intensity of synthetic data generation. The synthetic data generation proposed in this paper satisfies the need of the synthetic samples, which must include a large amount of effective information of unspecific signers. The proposed method is evaluated under different experimental conditions, such as the generating strategy, the capacity of the model, as well as the intensity and direction of the generating process. The results show that in most cases recognition accuracy is improved; and in some, even greatly improved.

Effort analysis in signer-independent sign gestures

A dilemma, caused by data variation, exists in research into signer-independent sign language recognition. An effective way to solve this dilemma, and thereby help push the research forward, is to understand sign language from the perspectives of human kinesics and linguistics. This paper, based on the principles of movement observation science, specifically Laban Movement Analysis (LMA), presents a summary of the factors causing sign language data variation, proposes the definition of, and a method for describing, sign language effort elements, and then provides a strategy for standardizing signer-independent sign language data. The standardized data are to be used for training and recognition. The method presented in this paper has been assessed under different experimental conditions, and the results show that the recognition accuracy is greatly increased.

Development in Signer-Independent Sign Language Recognition and the Ideas of Solving Some Key Problems

Signer-Independent sign language recognition is an unavoidable problem that must be solved in order to promote the practicality of sign language systems. In the signer-independent sign language recognition research, the lack of training data and the signer-independent sign language data variation bring a challenge to the effectivity of the existent research frame. This paper proposes a new research frame for signer-independent sign language recognition, and provides the strategy and ideas to solve the problem. Finding resolution to these problems is significant not only to the research on Chinese sign language recognition but also to other related fields.