Sign Language Recognition System Research Articles

Traffic Sign Detection and Recognition Using YOLO Object Detection Algorithm: A Systematic Review

Context: YOLO (You Look Only Once) is an algorithm based on deep neural networks with real-time object detection capabilities. This state-of-the-art technology is widely available, mainly due to its speed and precision. Since its conception, YOLO has been applied to detect and recognize traffic signs, pedestrians, traffic lights, vehicles, and so on. Objective: The goal of this research is to systematically analyze the YOLO object detection algorithm, applied to traffic sign detection and recognition systems, from five relevant aspects of this technology: applications, datasets, metrics, hardware, and challenges. Method: This study performs a systematic literature review (SLR) of studies on traffic sign detection and recognition using YOLO published in the years 2016–2022. Results: The search found 115 primary studies relevant to the goal of this research. After analyzing these investigations, the following relevant results were obtained. The most common applications of YOLO in this field are vehicular security and intelligent and autonomous vehicles. The majority of the sign datasets used to train, test, and validate YOLO-based systems are publicly available, with an emphasis on datasets from Germany and China. It has also been discovered that most works present sophisticated detection, classification, and processing speed metrics for traffic sign detection and recognition systems by using the different versions of YOLO. In addition, the most popular desktop data processing hardwares are Nvidia RTX 2080 and Titan Tesla V100 and, in the case of embedded or mobile GPU platforms, Jetson Xavier NX. Finally, seven relevant challenges that these systems face when operating in real road conditions have been identified. With this in mind, research has been reclassified to address these challenges in each case. Conclusions: This SLR is the most relevant and current work in the field of technology development applied to the detection and recognition of traffic signs using YOLO. In addition, insights are provided about future work that could be conducted to improve the field.

Isolated Arabic Sign Language Recognition Using a Transformer-based Model and Landmark Keypoints

Pose-based approaches for sign language recognition provide light-weight and fast models that can be adopted in real-time applications. This article presents a framework for isolated Arabic sign language recognition using hand and face keypoints. We employed MediaPipe pose estimator for extracting the keypoints of sign gestures in the video stream. Using the extracted keypoints, three models were proposed for sign language recognition: Long-Term Short Memory, Temporal Convolution Networks, and Transformer-based models. Moreover, we investigated the importance of non-manual features for sign language recognition systems and the obtained results showed that combining hand and face keypoints boosted the recognition accuracy by around 4% compared with only hand keypoints. The proposed models were evaluated on Arabic and Argentinian sign languages. Using the KArSL-100 dataset, the proposed pose-based Transformer achieved the highest accuracy of 99.74% and 68.2% in signer-dependent and -independent modes, respectively. Additionally, the Transformer was evaluated on the LSA64 dataset and obtained an accuracy of 98.25% and 91.09% in signer-dependent and -independent modes, respectively. Consequently, the pose-based Transformer outperformed the state-of-the-art techniques on both datasets using keypoints from the signer’s hands and face.

MEN: Mutual Enhancement Networks for Sign Language Recognition and Education.

The performance of existing sign language recognition approaches is typically limited by the scale of training data. To address this issue, we propose a mutual enhancement network (MEN) for joint sign language recognition and education. First, a sign language recognition system built upon a spatial-temporal network is proposed to recognize the semantic category of a given sign language video. Besides, a sign language education system is developed to detect the failure modes of learners and further guide them to sign correctly. Our theoretical contribution lies in formulating the above two systems as an estimation-maximization (EM) framework, which can progressively boost each other. The recognition system could become more robust and accurate with more training data collected by the education system, while the education system could guide the learners to sign more precisely, benefiting from the hand shape analysis module of the recognition system. Experimental results on three large-scale sign language recognition datasets validate the superiority of the proposed framework.

Uyghur Sign Language Recognition Based on Improved YOLOv7

Sign language is an important way for deaf-mutes to communicate, and sign language recognition plays an important role in solving the communication problems between able-bodied people and deaf-mutes. In order to help them communicate better with others and gradually switch to universal sign language, this paper proposes a design of Uyghur sign language recognition system based on the improved YOLOv7. Through the self-made 2568 Uyghur sign language datasets, a total of 35 categories such as ‘time’, ‘you’, ‘morning’, etc. were involved. In terms of network improvement, the backbone network is improved, and the improved convolution structure of SimAM attention + PConv is used; Second, replace the EIoU loss function. The sign language recognition model used in this paper has achieved significant improvement in Uyghur sign language recognition, which fully demonstrates its potential for in-depth research and wide application.

Sign Language Recognition using Spiking Neural Networks

In recent years, research in automatic Sign Language Recognition (SLR) has undergone significant progress, serving as a founda-tional base for developing applications that aim to promote the integration of deaf individuals into society. Most of this progress is owed to the recent developments in deep learning. However, the deployment of conventional Artificial Neural Networks (ANNs) can be hindered by their requirements in terms of computational power and energy consumption. Therefore, to improve the ef-ciency of current SLR systems, in this work, we propose the use of the increasingly popular Spiking Neural Networks (SNNs), which, on the one hand, provide more energy-efficient computations than conventional ANNs and, on the other hand, are able to process temporal sequences with simpler architectures thanks to their temporal dynamics. To evaluate our method, we utilize WLASL300, the 300-word (300 classes of signs) dataset from Word-Level American Sign Language, and achieve an improvement in accuracy with the SNN (+2.70%) over the previous state-of-the-art, when working with energy-efficient spiking neurons. Furthermore, we construct a non-spiking version of the same network and evaluate it in a similar manner. Our results demonstrate how the SNN has sparser activations (25% less), thanks to the use of spiking neurons, and therefore can be implemented with a lower power requirement than an ANN version of the same architecture. This work thus demonstrates the possibility of performing SLR in a very effective and efficient way, thus opening up the development of applications that span from the automatic real-time translation of dynamic signs to remote control utilizing sign languages.

Intelligent real-life key-pixel image detection system for early Arabic sign language learners.

Lack of an effective early sign language learning framework for a hard-of-hearing population can have traumatic consequences, causing social isolation and unfair treatment in workplaces. Alphabet and digit detection methods have been the basic framework for early sign language learning but are restricted by performance and accuracy, making it difficult to detect signs in real life. This article proposes an improved sign language detection method for early sign language learners based on the You Only Look Once version 8.0 (YOLOv8) algorithm, referred to as the intelligent sign language detection system (iSDS), which exploits the power of deep learning to detect sign language-distinct features. The iSDS method could overcome the false positive rates and improve the accuracy as well as the speed of sign language detection. The proposed iSDS framework for early sign language learners consists of three basic steps: (i) image pixel processing to extract features that are underrepresented in the frame, (ii) inter-dependence pixel-based feature extraction using YOLOv8, (iii) web-based signer independence validation. The proposed iSDS enables faster response times and reduces misinterpretation and inference delay time. The iSDS achieved state-of-the-art performance of over 97% for precision, recall, and F1-score with the best mAP of 87%. The proposed iSDS method has several potential applications, including continuous sign language detection systems and intelligent web-based sign recognition systems.

A Comprehensive Survey and Analysis of Traffic Sign Recognition Systems With Hardware Implementation

A Comprehensive Survey and Analysis of Traffic Sign Recognition Systems With Hardware Implementation

Sign Language Classifier based on Machine Learning

Sign language represents an efficient way for individuals with hearing impairments to communicate. We propose a sign recognition system into which several tools are integrated to help with the image pre-processing part. By doing so, a machine learning model was developed that does not require a lot of processing power because instead of using the images themselves, it uses extracted data from them to connect this model to a mobile interface that the users will use to recognise signed letters successfully. The communication between the client and the model is sustained through a local server.

Dynamic Sign Language Recognition Using Mediapipe Library and Modified LSTM Method

Hand gesture recognition (HGR) is a primary mode of communication and human involvement. While HGR can be used to enhance user interaction in human-computer interaction (HCI), it can also be used to overcome language barriers. For example, HGR could be used to recognize sign language, which is a visual language expressed by hand movements, poses, and faces, and used as a basic communication mode by deaf people around the world. This research aims to create a new method to detect dynamic hand movements, poses, and faces in sign language translation systems. The Long Short-Term Memory Modification (LSTM) approach and the Mediapipe library are used to recognize dynamic hand movements. In this study, twenty dynamic movements that match the context were designed to solve the challenge of identifying dynamic signal movements. Sequences and image processing data are collected using MediaPipe Holistic, processed, and trained using the LSTM Modification method. This model is practiced using training and validation data and a test set to evaluate it. The training evaluation results using the confusion matrix achieved an average accuracy of twenty words trained, which was 99.4% with epoch 150. The results of experiments per word showed detection accurateness of 85%, while experiments using sentences only reached 80%. The research carried out is a significant step forward in advancing the accuracy and practice of the dynamic sign language recognition system, promising better communication and accessibility for deaf people.

Dynamic Sign Language Recognition Using Mediapipe Library and Modified LSTM Method

Hand gesture recognition (HGR) is a primary mode of communication and human involvement. While HGR can be used to enhance user interaction in human-computer interaction (HCI), it can also be used to overcome language barriers. For example, HGR could be used to recognize sign language, which is a visual language expressed by hand movements, poses, and faces, and used as a basic communication mode by deaf people around the world. This research aims to create a new method to detect dynamic hand movements, poses, and faces in sign language translation systems. The Long Short-Term Memory Modification (LSTM) approach and the Mediapipe library are used to recognize dynamic hand movements. In this study, twenty dynamic movements that match the context were designed to solve the challenge of identifying dynamic signal movements. Sequences and image processing data are collected using MediaPipe Holistic, processed, and trained using the LSTM Modification method. This model is practiced using training and validation data and a test set to evaluate it. The training evaluation results using the confusion matrix achieved an average accuracy of twenty words trained, which was 99.4% with epoch 150. The results of experiments per word showed detection accurateness of 85%, while experiments using sentences only reached 80%. The research carried out is a significant step forward in advancing the accuracy and practice of the dynamic sign language recognition system, promising better communication and accessibility for deaf people.

Computer vision-based six layered ConvNeural network to recognize sign language for both numeral and alphabet signs

People who have trouble communicating verbally are often dependent on sign language, which can be difficult for most people to understand, making interaction with them a difficult endeavor. The Sign Language Recognition (SLR) system takes an input expression from a hearing or speaking-impaired person and outputs it in the form of text or voice to a normal person. The existing study related to the Sign Language Recognition system has some drawbacks, such as a lack of large datasets and datasets with a range of backgrounds, skin tones, and ages. This research efficiently focuses on Sign Language Recognition to overcome previous limitations. Most importantly, we use our proposed Convolutional Neural Network (CNN) model, “ConvNeural”, in order to train our dataset. Additionally, we develop our own datasets, “BdSL_OPSA22_STATIC1” and “BdSL_OPSA22_STATIC2”, both of which have ambiguous backgrounds. “BdSL_OPSA22_STATIC1” and “BdSL_OPSA22_STATIC2” both include images of Bangla characters and numerals, a total of 24,615 and 8437 images, respectively. The “ConvNeural” model outperforms the pre-trained models with accuracy of 98.38% for “BdSL_OPSA22_STATIC1” and 92.78% for “BdSL_OPSA22_STATIC2”. For “BdSL_OPSA22_STATIC1” dataset, we get precision, recall, F1-score, sensitivity and specificity of 96%, 95%, 95%, 99.31% , and 95.78% respectively. Moreover, in case of “BdSL_OPSA22_STATIC2” dataset, we achieve precision, recall, F1-score, sensitivity and specificity of 90%, 88%, 88%, 100%, and 100% respectively.

Multi-modal Deep Learning Approach to Improve Sentence level Sinhala Sign Language Recognition

Sign language is used across the world for communication purposes within hearing-impaired communities. Hearing people are not well versed in sign language and most hearing-impaired are not good in general text, creating a communication barrier. Research on Sign Language Recognition (SLR) systems have shown admirable solutions for this issue. In Sri Lanka, machine learning along with neural networks has been the prominent domain of research in Sinhala SLR. All previous research is mainly focused on word-level SLR using hand gestures for translation. While this works for a certain vocabulary, there are many signs interpreted through other spatial cues like lip movements and facial expressions. Therefore, translation is limited and sometimes the interpretations can be misleading. In this research, we propose a multi-modal Deep Learning approach that can effectively recognize sentence-level sign gestures using hand and lip movements and translate to Sinhala text. The model consists of modules for visual feature extraction (ResNet), contextual relationship modeling (transformer encoder with multi-head attention), alignment (CTC) and decoding (Prefix beam search). A dataset consisting 22 of sentences used for evaluations was collected under controlled conditions for a specific day-to-day scenario (a conversation between a vendor and a customer in a shop). The proposed model achieves a best Word Error Rate (WER) of 12.70 on the testing split, improving over the single-stream model which shows a best WER of 17.41, suggesting a multi-modal approach improves overall SLR.

Advancements in Indian Sign Language Recognition Systems: Enhancing Communication and Accessibility for the Deaf and Hearing Impaired

Sign language is an ocular-mobile language utilized by deaf or hearing-impaired individuals. It conveys a combination of handshapes, motions, facial expressions, and body postures. Sign language is an essential form of communication with its unique grammar and syntax. The communication barrier between those who use sign language and those who do not is substantial. India has approximately 63 million people in the deaf or hearing-impaired community (DHH). Therefore, research within the realm of sign language recognition (SLR) shows great potential in enhancing the quality of life for individuals facing hearing disabilities and promoting better communication and integration into society. In recent years, sign language recognition has attracted considerable attention due to its significant role in human-machine interaction, accessibility, real-time interpretation, educational tools, and communication aids. This study reviews the most current developments in Indian Sign Language Recognition Systems (ISLRS). It discusses the commonly used algorithms, standard datasets, and performance characteristics of these systems in detail. Lastly, it highlights the challenges and future perspectives of these emerging technologies.

Instance Segmentation Evaluation For Traffic Signs

This research paper focuses on developing a traffic sign recognition system based on the You Only Look At Coefficients (YOLACT) model, a one-stage instance segmentation model that offers high performance in terms of accuracy and reliability. However, the performance of YOLACT is influenced by various conditions such as day/night and different angles of objects. Therefore, this study aims to evaluate the impact of different angles and environments on the performance of the system. The paper discusses the framework, backbone structure, prototype generation branch, mask coefficient, and mask assembly used in the system. ResNet-101 and ResNet-50 were used as the backbone structure to extract feature maps of objects in the input image. The prototype generation branch generates prototype masks using fully convolutional networks (FCN), and the mask coefficient branch generates the Mask assembly using the sigmoid nonlinearity. Two models, YOLACT and Mask-RCNN, were evaluated in terms of mean precision (mAP) and frames per second (FPS) with the front view dataset. The results show that YOLACT outperforms Mask-RCNN in terms of accuracy and speed. For an image resolution of 550x550, YOLACT with Resnet-101 was considered the best model in this article since it achieves over 80% precision, recall, specificity, and accuracy in various conditions such as day, night, left and right, and forward-looking angles.

Traffic Sign Detection and Recognition System for Autonomous Vehicle

Abstract: Traffic Sign Recognition and detection system is specifically useful for drivers and driver-less cars. By identifying traffic signs accurately and effectively it improves the driving safety of autonomous cars. This research paper presents a comprehensive approach for the development of a traffic sign detection system using Convolution Neural Network, TensorFlow and OpenCV to classify the traffic signs in real-time effectively. TensorFlow and OpenCV play an important role in shaping effective traffic detection and recognition system. We have explained the process of data collection, preparation, model architecture and the integration of TensorFlow for training and interference. OpenCV for image processing and real time feed processing, ensuring seamless implementation on various hardware platform. The model uses German Traffic Sign Recognition dataset. The results show that the proposed system achieves high accuracy in detecting and recognizing traffic sign making it a valuable system for both autonomous vehicles and human drivers

Performance Evaluation of Feature Fusion Approaches for Indian Sign Language Recognition System

Performance Evaluation of Feature Fusion Approaches for Indian Sign Language Recognition System

SIGN LANGUAGE DETECTION AND TEXT TO SPEECH GENERATION

A vital technique for bridging the communication gap between hearing-impaired and normal individuals is sign language. But given the diversity of today's approximately 7000 sign languages, which vary in hand shapes, body part positions, and motion positions, automated sign language recognition (ASLR) is a challenging method. Researchers are looking into more effective ways to build ASLR systems to find intelligent solutions in order to get around this complexity, and they have shown impressive results. This purpose of this work is to examine the literature on intelligent systems for sign language recognition spanning the previous 20 years. 649 publications in all about intelligent systems and decision assistance on the recognition of sign language (SLR) are taken out of the Scopus database and examined. The taken out Using bibliometric VOS Viewer software, articles are evaluated in order to: (1) determine the temporal and spatial distributions of the publications; (2) establish cooperation networks between authors and affiliations; and (3) identify productive institutions within this framework. Additionally, methods for vision-based sign language recognition are reviewed. There is a discussion of the many feature extraction and classification methods utilized in SLR to get high-quality results. This paper's literature assessment highlights the value of integrating intelligent solutions into sign language recognition systems and illustrates that the development of flawless intelligent systems for sign language recognition remains an unsolved challenge. Overall, it is anticipated that this study will aid in the production of intelligent based SLR and the collection of information while offering readers, researchers, and practitioners a road map for future direction.

Machine Learning-Based Gesture Recognition for Communication with the Deaf and Dumb

A deep learning model specifically designed to recognize signs in sign language is the foundation of the Sign Language Recognition system. Sign Language is a visual language used by the deaf and hard of hearing community to communicate with one another and the general public. Sign language is a kind of nonverbal communication based on the use of hand gestures. The ability to communicate socially and emotionally is greatly aided when the speech and hearing challenged have access to sign language. The model developed in this paper captures the images through live web cam and displays the sign language meaning on the screen as text output. The model is trained and built by deep learning framework using Convolution Neural Networks (CNN) in this work. The model is trained with images of hand gestures captured through webcam using Computer Vision and then after successful training, the system performs recognition process through matching parameters for a given input gesture and finally displays the sign language meaning of the gesture as text output on the screen.

Sign Language Recognition Using the Electromyographic Signal: A Systematic Literature Review.

The analysis and recognition of sign languages are currently active fields of research focused on sign recognition. Various approaches differ in terms of analysis methods and the devices used for sign acquisition. Traditional methods rely on video analysis or spatial positioning data calculated using motion capture tools. In contrast to these conventional recognition and classification approaches, electromyogram (EMG) signals, which measure muscle electrical activity, offer potential technology for detecting gestures. These EMG-based approaches have recently gained attention due to their advantages. This prompted us to conduct a comprehensive study on the methods, approaches, and projects utilizing EMG sensors for sign language handshape recognition. In this paper, we provided an overview of the sign language recognition field through a literature review, with the objective of offering an in-depth review of the most significant techniques. These techniques were categorized in this article based on their respective methodologies. The survey discussed the progress and challenges in sign language recognition systems based on surface electromyography (sEMG) signals. These systems have shown promise but face issues like sEMG data variability and sensor placement. Multiple sensors enhance reliability and accuracy. Machine learning, including deep learning, is used to address these challenges. Common classifiers in sEMG-based sign language recognition include SVM, ANN, CNN, KNN, HMM, and LSTM. While SVM and ANN are widely used, random forest and KNN have shown better performance in some cases. A multilayer perceptron neural network achieved perfect accuracy in one study. CNN, often paired with LSTM, ranks as the third most popular classifier and can achieve exceptional accuracy, reaching up to 99.6% when utilizing both EMG and IMU data. LSTM is highly regarded for handling sequential dependencies in EMG signals, making it a critical component of sign language recognition systems. In summary, the survey highlights the prevalence of SVM and ANN classifiers but also suggests the effectiveness of alternative classifiers like random forests and KNNs. LSTM emerges as the most suitable algorithm for capturing sequential dependencies and improving gesture recognition in EMG-based sign language recognition systems.

Real Time Sign Language Recognition Using Deep Learning

Abstract: Sign language is the only tool of communication for the person who is not able to speak and hear anything. Sign language is a boon for verbally challenged people to express their thoughts and emotion. Using this Sign Language Recognition system, the communication gap between people with hearing impairments and the general public can be cleared. In this work, a scheme of sign language recognition has been proposed for identifying the gestures in sign language. With the help of computer vision and neural networks, system can detect the signs and give the respective text as output. The major aim of this work is to build a neural network using a Long Short-Term Memory (LSTM) deep learning model using the video frames which offer the translation of gestures into text. The model is trained with the dataset that is collected using MediaPipe holistic key points from the video of the person which detects the pose, face and hand landmarks. after building the neural network, real-time sign language recognition is performed using OpenCV and a user interface is developed using Streamlit where the gestures are recognized and displayed as text within the highlighted section on the screen