Field Of Sign Language Recognition Research Articles

Enhancing communication: Deep learning for Arabic sign language translation

Abstract This study explores the field of sign language recognition through machine learning, focusing on the development and comparative evaluation of various algorithms designed to interpret sign language. With the prevalence of hearing impairment affecting millions globally, efficient sign language recognition systems are increasingly critical for enhancing communication for the deaf and hard-of-hearing community. We review several studies, showcasing algorithms with accuracies ranging from 63.5 to 99.6%. Building on these works, we introduce a novel algorithm that has been rigorously tested and has demonstrated a perfect accuracy of 99.7%. Our proposed algorithm utilizes a sophisticated convolutional neural network architecture that outperforms existing models. This work details the methodology of the proposed system, which includes preprocessing, feature extraction, and a multi-layered CNN approach. The remarkable performance of our algorithm sets a new benchmark in the field and suggests significant potential for real-world application in assistive technologies. We conclude by discussing the impact of these findings and propose directions for future research to further improve the accessibility and effectiveness of sign language recognition systems.

Isolated sign language recognition through integrating pose data and motion history images

This article presents an innovative approach for the task of isolated sign language recognition (SLR); this approach centers on the integration of pose data with motion history images (MHIs) derived from these data. Our research combines spatial information obtained from body, hand, and face poses with the comprehensive details provided by three-channel MHI data concerning the temporal dynamics of the sign. Particularly, our developed finger pose-based MHI (FP-MHI) feature significantly enhances the recognition success, capturing the nuances of finger movements and gestures, unlike existing approaches in SLR. This feature improves the accuracy and reliability of SLR systems by more accurately capturing the fine details and richness of sign language. Additionally, we enhance the overall model accuracy by predicting missing pose data through linear interpolation. Our study, based on the randomized leaky rectified linear unit (RReLU) enhanced ResNet-18 model, successfully handles the interaction between manual and non-manual features through the fusion of extracted features and classification with a support vector machine (SVM). This innovative integration demonstrates competitive and superior results compared to current methodologies in the field of SLR across various datasets, including BosphorusSign22k-general, BosphorusSign22k, LSA64, and GSL, in our experiments.

Advancements in Vision-Based Sign Language Recognition: A Comprehensive Review

As nearly 70 million individuals suffer from disabling hearing loss, sign language serves as an important means of communication. Unfortunately, the lack of proficiency in sign language among the general population hinders meaningful interactions with those who rely on it. This paper presents an extensive analysis of the cutting-edge methodologies in sign language translation, with the ultimate goal of facilitating effective communication between sign language users and the broader community. In addition to reviewing state-of-the-art approaches, this work also investigates into the challenges and limitations faced by gesture recognition research. Overall, it is expected that the study may provide readers and researchers with a guide for future research and creation in the field of sign language recognition.

SignSense: AI Framework for Sign Language Recognition

Sign Language recognition is a pioneering framework designed to advance the field of Sign Language Recognition (SLR) through the innovative application of ensemble deep learning models. The primary goal of this research is to significantly improve the accuracy, resilience and interpretability of SLR systems. Leveraging the unique features of ResNet within an ensemble learning paradigm. The key component of InceptionResNetv2 architecture is its deep and effective feature extraction capabilities. The utilization of InceptionResNet model enhances the model ability to capture intricate details crucial for accurate sign language recognition. This framework is also to scale seamlessly, accommodating an expanding vocabulary of signs, diverse users and dynamic environmental conditions without compromising performance.

A Review on Sign Language Recognition and Learning System

This paper presents innovative approaches to various aspects of sign language recognition and learning, catering to the diverse needs within this field. It introduces a system utilizing 2D image sampling and concatenation, trained with convolutional neural networks, achieving accurate and robust sign recognition even with low-cost cameras. Another system, SignQuiz, offers a cost-effective web-based solution for learning finger-spelled signs in Indian Sign Language (ISL), outperforming traditional printed mediums. Additionally, a dynamic hand gesture recognition system employing deep learning architectures demonstrates superior performance over existing methods, addressing challenges such as hand segmentation and sequence modeling. Furthermore, an efficient deep convolutional neural network approach for hand gesture recognition, including transfer learning, achieves high recognition rates across various datasets. Finally, a robust model for static sign recognition in ISL, utilizing CNNs and evaluating on a diverse dataset, attains exceptional accuracy and outperforms previous works. These contributions collectively advance the field of sign language recognition and learning, offering solutions that are accurate, cost-effective, and efficient, thereby facilitating better communication and interaction for the hearing-impaired community. Key Words: CNN, Sign language recognition, Hand gesture recognition, Machine learning, Transfer Learning, SVM.

A Systematic Literature Review on Vision-Based Hand Gesture for Sign Language Translation

Deaf and hard of hearing people use sign language to communicate. People around mute and deaf people have difficulty communicating with each other if they do not understand sign language. This problem has prompted many researchers to conduct studies on sign language translation. However, there is a lack of compilation of SLR on this topic. Therefore, this paper aims to provide a thorough literature review of previous studies on sign language to text translation based on the vision method. PRISMA (Preferred Reporting Items to writing a standard Systematic Review and Meta-Analyses) is used in this systematic review. Two primary databases, Web of Science and Scopus, have been used to search for relevant articles and resources included in this systematic literature review. Based on the outcome of the systematic review of the topic, the primary studies on sign language translation systems were conducted using self-generated datasets more than public datasets. More static action sign language was studied compared to dynamic action sign language. For the type of recognition, more alphabet sign language was studied compared to digit, word, or sentence sign language. Other than that, most studies used digital cameras rather than Microsoft Kinect or a webcam. The most used classification method was Convolution Neural Network (CNN). The study is intended to guide readers and researchers for future research and knowledge enhancement in the field of sign language recognition.

A review on computational methods based automated sign language recognition system for hearing and speech impaired community

SummaryThe recent advancements in computer vision and deep learning have led to promising progress in various motion detection and gesture recognition methods. Thriving efforts in the field of sign language recognition (SLR) during recent years led to interaction between humans and computer systems. Contributing a real‐time automated Sign Language recognition system will be a remarkable treasure for hearing and speech‐impaired people that will break the barriers of interaction with the real world. Albeit several research works are accomplished in sign language recognition, there is still a demand for developing a real‐time automated sign language recognition system. Compared to other methodologies, the techniques adopted may have advantages and disadvantages, and they vary from researcher to researcher. There are still some issues with employing these SLR models and procedures regularly, even though numerous research studies have been conducted to determine the most acceptable methods and models for sign language recognition. It gets more expensive and difficult in terms of resources when the developed automated SLR system becomes available as a product. Hence, for the welfare of the hearing and speech‐impaired community, the researchers are still endeavoring way to find a cost‐effective method. This work brings forth the challenges faced by scientists in developing a cost‐effective commercial prototype for the hearing and speech‐impaired community. This paper also explores and analyses various deep learning techniques and methods used in developing a sign language recognition system. The objective behind this work is to identify the best method which produces high accuracy in developing a cost‐effective sign language recognition system aiding communication between signers and non‐signers so that they can be a part of growing technology.

DeepASLR: A CNN based human computer interface for American Sign Language recognition for hearing-impaired individuals

BackgroundSign language is an essential means of communication for hearing-impaired individuals. ObjectiveWe aimed to develop an American sign language recognition dataset and use it in the deep learning model which depends on neural networks to interpret gestures of sign language and hand poses to natural language. MethodsIn this study, we developed a dataset and a Convolutional Neural Network-based sign language interface system to interpret gestures of sign language and hand poses to natural language. The neural network developed in this study is a Convolutional Neural Network (CNN) which enhances the predictability of the American Sign Language alphabet (ASLA). This research establishes a new dataset of the American Sign Language alphabet which takes into consideration various conditions such as lighting and distance. ResultsThe dataset created in this study is a new addition in the field of sign language recognition (SLR). This dataset may be used to develop SLR systems. Furthermore, our research compares the results of our dataset with two different datasets from other studies. The other datasets have invariant scene conditions, but our suggested CNN model demonstrated high accuracy for all the tested datasets. Despite the different conditions and volume of the new dataset, it achieved 99.38% accuracy with excellent prediction and small loss (0.0250). ConclusionsThe proposed system may be considered a promising solution in medical applications that use deep learning with superior accuracy. Moreover, our dataset was created under variable conditions which increases the number of contributions, comparisons, results and conclusions in the field of SLR and may enhance such systems.

Continuous Sign Language Recognition through a Context-Aware Generative Adversarial Network.

Continuous sign language recognition is a weakly supervised task dealing with the identification of continuous sign gestures from video sequences, without any prior knowledge about the temporal boundaries between consecutive signs. Most of the existing methods focus mainly on the extraction of spatio-temporal visual features without exploiting text or contextual information to further improve the recognition accuracy. Moreover, the ability of deep generative models to effectively model data distribution has not been investigated yet in the field of sign language recognition. To this end, a novel approach for context-aware continuous sign language recognition using a generative adversarial network architecture, named as Sign Language Recognition Generative Adversarial Network (SLRGAN), is introduced. The proposed network architecture consists of a generator that recognizes sign language glosses by extracting spatial and temporal features from video sequences, as well as a discriminator that evaluates the quality of the generator’s predictions by modeling text information at the sentence and gloss levels. The paper also investigates the importance of contextual information on sign language conversations for both Deaf-to-Deaf and Deaf-to-hearing communication. Contextual information, in the form of hidden states extracted from the previous sentence, is fed into the bidirectional long short-term memory module of the generator to improve the recognition accuracy of the network. At the final stage, sign language translation is performed by a transformer network, which converts sign language glosses to natural language text. Our proposed method achieved word error rates of 23.4%, 2.1% and 2.26% on the RWTH-Phoenix-Weather-2014 and the Chinese Sign Language (CSL) and Greek Sign Language (GSL) Signer Independent (SI) datasets, respectively.

Application of Tensor Train Decomposition in S2VT Model for Sign Language Recognition

Sign language recognition is a conversion of sign language into text or speech, bridging the communication between the hearing and society. Recently, sequence-to-sequence video to text (S2VT) models has been employed in the field of sign language recognition as an effective method. However, more than 20 million parameters trained in S2VT models will result in a huge consumption in memory and computational resources, making it hard to be applied in mobile devices. In order to overcome this issue, we proposed to employ tensor-train decomposition in S2VT models to reduce the parameters. First, the impact of parameters of tensor-train factorization on the model performance was investigated systematically. After that, we applied tensor-train decomposition in different layers of a S2VT model to establish 6 tensor-train S2VT models for Chinese sign language recognition. The experimental results demonstrated that when the fully-connected layer and the first LSTM layer in S2VT was represented with tensor-train format, the model could obtain the best performance, remaining high accuracy and reducing parameters and memory significantly. The proposed tensor-train S2VT models can also be applied in other sequence-to-sequence problems to improve the performance.

Sign Language Recognition Systems: A Decade Systematic Literature Review

Despite the importance of sign language recognition systems, there is a lack of a Systematic Literature Review and a classification scheme for it. This is the first identifiable academic literature review of sign language recognition systems. It provides an academic database of literature between the duration of 2007–2017 and proposes a classification scheme to classify the research articles. Three hundred and ninety six research articles were identified and reviewed for their direct relevance to sign language recognition systems. One hundred and seventeen research articles were subsequently selected, reviewed and classified. Each of 117 selected papers was categorized on the basis of twenty five sign languages and were further compared on the basis of six dimensions (data acquisition techniques, static/dynamic signs, signing mode, single/double handed signs, classification technique and recognition rate). The Systematic Literature Review and classification process was verified independently. Literature findings of this paper indicate that the major research on sign language recognition has been performed on static, isolated and single handed signs using camera. Overall, it will be hoped that the study may provide readers and researchers a roadmap to guide future research and facilitate knowledge accumulation and creation into the field of sign language recognition.

Survey of Sign Language Recognition System

According to WHO’s survey over 5% of the total population is deaf. Deafness impacts on individual’s ability to communicate with others. Deaf people communicate among themselves using sign language. Sign languages is combination of hand movements and facial expressions. However normal person rarely knows these signs and this becomes barrier in real world communication. This paper is discusses the research work progress in the field of Sign language Recognition which aids Deaf and Dumb Community in the Society for communication. This paper discusses the research work of different countries sign languages. The discussion not limited to single sign but word and sentences recognition. Signs are in the form of hand gestures and these gestures are identified from images, videos and glove/sensor based system. The paper discuss about sign gesture dataset which can be created or referred from standard sign library. The review shows there are more than one color model available which distinguish skin and non skin color in a given image. Hand gestures are identified and classified using features. The features are like Orientation Histogram, Haar like, Fourier descriptors, DCT etc. Classification algorithms are included: Support Vector Machines, k-Nearest Neighbor, Naive Bayes Classifier (NBC) Feed forward, back propagation neural network, Distance Based Classifier, among others. Recognition accuracy is more than 90%, few achieve around 80%. However, most experiments were performed indoor under controlled environments, with specific lighting conditions, simple background, and were only using a small set of gestures. Comprehensive study provide introduction to automated Sign Language Recognition System reader and researcher and further facilitates future research efforts in this area.

Synthetic data generation technique in Signer-independent sign language recognition

The lack of training samples is an important problem in the field of sign language recognition. This paper presents a method of generating synthetic multi-stream samples so as to enlarge the training set of sign. The mean shift algorithm is able to obtain the directions of maximum increase and decrease in the density function, so it is used to control the direction and the intensity of synthetic data generation. The synthetic data generation proposed in this paper satisfies the need of the synthetic samples, which must include a large amount of effective information of unspecific signers. The proposed method is evaluated under different experimental conditions, such as the generating strategy, the capacity of the model, as well as the intensity and direction of the generating process. The results show that in most cases recognition accuracy is improved; and in some, even greatly improved.

Recent developments in visual sign language recognition

Research in the field of sign language recognition has made significant advances in recent years. The present achievements provide the basis for future applications with the objective of supporting the integration of deaf people into the hearing society. Translation systems, for example, could facilitate communication between deaf and hearing people in public situations. Further applications, such as user interfaces and automatic indexing of signed videos, become feasible. The current state in sign language recognition is roughly 30 years behind speech recognition, which corresponds to the gradual transition from isolated to continuous recognition for small vocabulary tasks. Research efforts were mainly focused on robust feature extraction or statistical modeling of signs. However, current recognition systems are still designed for signer-dependent operation under laboratory conditions. This paper describes a comprehensive concept for robust visual sign language recognition, which represents the recent developments in this field. The proposed recognition system aims for signer-independent operation and utilizes a single video camera for data acquisition to ensure user-friendliness. Since sign languages make use of manual and facial means of expression, both channels are employed for recognition. For mobile operation in uncontrolled environments, sophisticated algorithms were developed that robustly extract manual and facial features. The extraction of manual features relies on a multiple hypotheses tracking approach to resolve ambiguities of hand positions. For facial feature extraction, an active appearance model is applied which allows identification of areas of interest such as the eyes and mouth region. In the next processing step, a numerical description of the facial expression, head pose, line of sight, and lip outline is computed. The system employs a resolution strategy for dealing with mutual overlapping of the signer’s hands and face. Classification is based on hidden Markov models which are able to compensate time and amplitude variances in the articulation of a sign. The classification stage is designed for recognition of isolated signs, as well as of continuous sign language. In the latter case, a stochastic language model can be utilized, which considers uni- and bigram probabilities of single and successive signs. For statistical modeling of reference models each sign is represented either as a whole or as a composition of smaller subunits—similar to phonemes in spoken languages. While recognition based on word models is limited to rather small vocabularies, subunit models open the door to large vocabularies. Achieving signer-independence constitutes a challenging problem, as the articulation of a sign is subject to high interpersonal variance. This problem cannot be solved by simple feature normalization and must be addressed at the classification level. Therefore, dedicated adaptation methods known from speech recognition were implemented and modified to consider the specifics of sign languages. For rapid adaptation to unknown signers the proposed recognition system employs a combined approach of maximum likelihood linear regression and maximum a posteriori estimation.