Sign Language Dataset Research Articles

Artificial Intelligence based Automated Sign Gesture Recognition Solutions for Visually Challenged People

Gesture recognition is employed in human-machine communications, enhancing human life with impairments or who depend on non-verbal instructions. Hand gestures role an important role in the field of assistive technology for persons with visual impairments, whereas an optimum user communication design is of major importance. Many authors with substantial development for gesture recognition modeled several methods by using deep learning (DL) methods. This article introduces a Robust Gesture Sign Language Recognition Using Chicken Earthworm Optimization with Deep Learning (RSLR-CEWODL) approach. The projected RSLR-CEWODL algorithm majorly focuses on the recognition and classification of sign language. To accomplish this, the presented RSLR-CEWODL technique utilizes a residual network (ResNet-101) model for feature extraction. For optimal hyper parameter tuning process, the presented RSLR-CEWODL algorithm exploits the CEWO algorithm. Besides, the RSLR-CEWODL technique uses a whale optimization algorithm (WOA) with deep belief network (DBN) method for the sign language recognition method. The simulation result of the RSLR-CEWODL algorithm is tested using sign language datasets and the outcome was measured under various measures. The simulation values demonstrated the enhancements of the RSLR-CEWODL technique over other methodologies.

Investigating Active Learning based on Dynamic Data Selection techniques for Image Classification

This paper explores the efficacy of Active Learning (AL) techniques, specifically focusing on Dynamic Data Selection (DDS), for improving image classification tasks. AL is a machine learning paradigm that enables the automatic selection of the most informative data samples for annotation, thereby reducing the annotation burden and enhancing model performance. In this study, we investigate the integration of DDS techniques with AL strategies to iteratively select the most informative image samples for model training. We use a fine-tuned VGG16 as the underlying classification model due to their effectiveness in image analysis tasks. Our experimental evaluation involves comparing the performance of fine-tuned VGG16 trained with three AL-based DDS techniques on Arabic sign language dataset. We analyze various DDS strategies, including Random selection, Entropy-based selection, and margin selection to determine their impact on model accuracy and annotation efficiency. The results of our study demonstrate the effectiveness of margin selection method-based AL approach in improving the performance of recognition of 32 hand gestures for Arabic sign language (95.3 \%) while minimizing the annotation effort.

High-Precision Traffic Sign Detection and Recognition Using an Enhanced YOLOv5

High-precision traffic sign detection plays an important role in enhancing road traffic safety, ensuring traffic smoothness, supporting the development of intelligent transport systems and promoting the standardization and normalization of traffic facilities. To overcome the limitations of traditional methods, this study proposes an enhanced YOLOv5 algorithm for complex road environments. First, the K-Means clustering algorithm is used to cluster and analyze the boundary boxes of traffic signs in the training dataset, obtaining anchor box sizes that are closer to the distribution of the dataset to improve detection accuracy. Then, a genetic algorithm was used to further optimize the initial anchor box, and a global search strategy was used to find the optimal anchor box configuration, further improving detection performance. In terms of model structure, a bidirectional feature pyramid network (Bi-FPN) is introduced, which effectively utilizes multi-scale feature information and enhances the adaptability of the model to traffic signs of different sizes through top-down and bottom-up feature fusion paths, as well as cross-scale connections. In addition, a global attention mechanism (GAM) was introduced to recalibrate the feature maps through channel attention and spatial attention dimensions, improving the robustness and detection accuracy of the model for complex environments. Finally, a loss function called Focal-EIoU was used to solve the problems of class imbalance and sample imbalance, which improved the stability and performance of the object detection model. Experiments on a Chinese traffic sign dataset demonstrate significant improvements, with an increase in mAP by 10.03%, precision by 4.7%, recall by 2.6% and F1-score by 3.48%, respectively. The results prove the validity of the proposed traffic sign recognition method, especially in complex road environments. This study provides new ideas and methods for the field of traffic sign detection, which has important theoretical significance and application value.

A real-time platform for Spanish Sign Language interpretation

AbstractMore than 13% of the Spanish population suffers from hearing impairment. Despite the existence of several hearing aids and implants making sounds clearer and louder, there are some people unable to use them and, consequently, their only means of communication is Sign Language. However, this language is not widely known in the society. Consequently, those who are deaf or hard of hearing may be socially excluded and experience frustration as a result of the lack of communication. In this context, Sign Language Recognition and interpretation would help to break down the existing communication barriers and facilitate the creation of inclusive environments. With that aim, this paper presents a real-time platform to recognize and interpret finger-spelt words in Spanish Sign Language (Lengua de Signos Española). As finger spelling implies the recognition of each signed letter, a comparative analysis of different deep learning techniques to properly recognize the Spanish Sign Language alphabet has been carried out. For that, due to the lack of Spanish Sign Language datasets, the first step was to capture and build an image dataset representing its 30 letters. As there are static and in-motion letters, spatial and temporal analysis has been conducted by considering different kind of neural networks (Convolutional Neural Networks, Recurrent Neural Networks, and Vision Transformers). The experimental results highlight the good performance of the studied architectures, obtaining a maximum accuracy of 79.96% on previously unseen data. Finally, a real-time platform for the recognition and interpretation of finger-spelt words in Spanish Sign Language has been implemented making the communication possible.

Model GHT-SVM for Traffic Sign Detection Using Support Vector Machine Algorithm Based On Gabor Filter and Top-Black Hat Transform

A factor that can hinder the detection and recognition of traffic signs is the variation in lighting in the image of traffic signs. This study aims to detect traffic symbols using Gabor Filter (GFT), Top Hat Transform (THT), and Black Hat Transform (BHT) methods on the Support Vector Machine (SVM) algorithm for traffic sign dataset images with data problems that tend to have dark backgrounds at night and bright backgrounds during the day. From the experimental results, GHT-SVM gets the highest accuracy compared to HSV-SVM, HSV-RF, HSV-KNN, and H2T-SVM models. Based on experimental results, H2T-SVM from HOG ⊕ ENT ⊕ BHT ⊕ SVM results get the best accuracy of 86.42%. The Gabor Filter (GFT) parameters used are the number of filters with a value of 16, ksize with a value of 30, sigma with a standard deviation value of 3.0, lambd with a sinusoidal factor value of 10.0, gamma with a spatial aspect ratio value of 0.5 and psi with a phase offset value of 0 while the Top Hat Transform (THT) and Black Hat Transform (BHT) methods use filterSize sizes with values (3, 3).

A Comparative Analysis of Convolutional Neural Networks for American Sign Language Recognition

Abstract. Sign language recognition is an important technology that makes it possible for ordinary people to be able to communicate with deaf people, fostering inclusivity and accessibility. The came out of deep learning technology has completely changed the field by enabling the automatic extraction and learning of hierarchical features from raw data, leading to significant improvements in recognition accuracy. This paper presents a comprehensive comparative analysis of different Convolutional Neural Network (CNN) architectures for recognizing American Sign Language (ASL) signs. Utilizing the sign language dataset, which contains 24 classes of ASL letters represented by 28x28 grayscale images, the author evaluated the performance of a Basic CNN, a Modified Residual Network (ResNet)-50, and a LeNet-5 model. This study emphasizes the impact of architectural choices on recognition accuracy and computational efficiency. Results indicate that while ResNet-50 demonstrates superior accuracy, fluctuating significantly during initial training, the Basic CNN and LeNet-5 models offer greater stability with slightly lower accuracy. This work concludes that despite the initial challenges, deep learning models, particularly ResNet-50, show promise for ASL recognition, highlighting the need for diverse and enriched datasets to improve model reliability in real-world scenarios.

A Static Sign Language Recognition Method Enhanced with Self-Attention Mechanisms.

For the current wearable devices in the application of cross-diversified user groups, it is common to face the technical difficulties of static sign language recognition accuracy attenuation, weak anti-noise ability, and insufficient system robustness due to the differences in the use of users. This paper proposes a novel static sign language recognition method enhanced by a self-attention mechanism. The key features of sign language gesture classification are highlighted by the weight function, and then the self-attention mechanism is combined to pay more attention to the key features, and the convolutional neural network is used to extract the features and classify them, which realizes the accurate recognition of different types of static sign language under standard gestures and non-standard gestures. Experimental results reveal that the proposed method achieves an average accuracy of 99.52% in the standard static sign language recognition task when tested against the standard 36 static gestures selected within the reference American Sign Language dataset. By imposing random angular bias conditions of ±(0°-9°] and ±(9°-18°], the average recognition rates in this range were 98.63% and 86.33%. These findings indicate that, compared to existing methods, the proposed method not only maintains a high recognition rate for standard static gestures but also exhibits superior noise resistance and robustness, rendering it suitable for static sign language recognition among diverse user populations.

Enhancing small target traffic sign detection with ML_SAP in YOLOv5s

Detecting small traffic signs poses significant challenges due to the complex nature and dynamic conditions of real-world traffic scenarios. In response to these challenges, we propose an improved YOLOv5s target detection model incorporating the multilevel squeeze feature perception (ML_SAP) mechanism, aiming to increase the accuracy of small traffic sign detection. First, an additional detection layer is incorporated to enhance the model’s capacity for detecting small-scale traffic signs. This improvement is accompanied by the adoption of a WIoU loss function, which evaluate the quality of anchor boxes. Moreover, the ML_SAP mechanism is designed to promotes the fusion and extraction of features at different levels. This mechanism effectively increases the network model’s proficiency in identifying small targets under varying environmental conditions. To verify the effectiveness of the improved method, we conducted extensive experiments on two public transportation sign datasets. Notably, on the challenging samples in the CCTSDB-2021 dataset, the improved model achieves a detection recall of 77.1%, which is 5.4% higher than that of YOLOv5s, and a mean average precision (mAP) of 82.7%, which is 3.9% higher than that of the base model. Furthermore, the model achieves a detection recall of 91.3% on the TT100K dataset, which is 3.7% higher than the performance of YOLOv5s, and a mean precision (mAP) of 91.5%, which is 4.6% higher than that of the base model.

A lightweight network-based sign language robot with facial mirroring and speech system

Human–robot interaction is an essential capability for humanoid robots to enter the physical world and become companions in people’s lives, learning, and work. While the majority of current research focuses on the voice-based interactions of robots, yet over 60% of communication occurs through nonverbal behaviors, such as facial expressions and hand gestures. Endowing robots with the ability to communicate through nonverbal behavior not only enhances the interactive experience with robots but also provides a potential communication tool for individuals with hearing or speech impairments. Here, we develop a humanoid robot capable of adjusting facial movements by driving servos, and design a novel framework for the robot to integrate sign language recognition and facial landmark detection algorithms. This framework facilitates the robot recognize sign language and translate it into spoken language, while also imitating the facial expressions of the signers. To achieve this, we also propose a lightweight deep learning network called RealTimeSignNet for real-time sign language recognition. Leveraging lightweight 3D convolution modules and time-dependent constraints, this model adapts to various time scales, ensuring efficient processing of sign language recognition tasks. Experimental results demonstrate the outstanding performance of the RealTimeSignNet model on mainstream sign language datasets, achieving an accuracy of 88.1% on the large continuous sign language dataset (continuous SLR), 98.2% on the isolated sign language dataset (SLR 500), and 91.50% on the English sign language dataset (WLAS). The overall assessment demonstrates that our humanoid robot is capable of recognizing sign language and translating it into spoken language, while imitating the facial emotions, providing a comprehensive solution to the communication challenges faced by individuals with hearing and speech impairments.

Study on pose-based deep learning models for gloss-free Sign Language Translation

Sign Language Translation (SLT) is a challenging task due to its cross-domain nature, different grammars and lack of data. Currently, many SLT models rely on intermediate gloss annotations as outputs or latent priors. Glosses can help models to correctly segment and align signs to better understand the video. How- ever, the use of glosses comes with significant limitations, since obtaining annotations is quite difficult. Therefore, scaling gloss-based models to millions of samples remains impractical, specially considering the scarcity of sign language datasets. In a similar fashion, many models use video data that requires larger models which typically only work on high end GPUs, and are less invariant to signers appearance and context. In this work we propose a gloss-free pose-based SLT model. Using the extracted pose as feature allow fora sign significant reduction in the dimensionality of the data and the size of the model. We evaluate the state of the art, compare available models and develop a keypoint-based Transformer model for gloss-free. SLT, trained on RWTH-Phoenix, a standard dataset for benchmarking SLT models alongside GSL, a simpler laboratory-made Greek Sign Language dataset.

SWL-LSE: A Dataset of Health-Related Signs in Spanish Sign Language with an ISLR Baseline Method

Progress in automatic sign language recognition and translation has been hindered by the scarcity of datasets available for the training of machine learning algorithms, a challenge that is even more acute for languages with smaller signing communities, such as Spanish. In this paper, we introduce a dataset of 300 isolated signs in Spanish Sign Language, collected online via a web application with contributions from 124 participants, resulting in a total of 8000 instances. This dataset, which is openly available, includes keypoints extracted using MediaPipe Holistic. The goal of this paper is to describe the construction and characteristics of the dataset and to provide a baseline classification method using a spatial–temporal graph convolutional network (ST-GCN) model, encouraging the scientific community to improve upon it. The experimental section offers a comparative analysis of the method’s performance on the new dataset, as well as on two other well-known datasets. The dataset, code, and web app used for data collection are freely available, and the web app can also be used to test classifier performance on-line in real-time.

Hardware acceleration of Tiny YOLO deep neural networks for sign language recognition: A comprehensive performance analysis

In this paper, we benchmark two automation frameworks, Vitis AI and FINN, for sign language recognition on a Field Programmable Gate Array (FPGA). We conducted an in-depth exploration of both frameworks using Tiny YOLOv2 networks by varying design parameters such as precision, parallelism ratio, etc. Further, a fair baseline comparison is made based on accuracy, speed, and hardware resources. Experimental findings demonstrate that the Vitis AI outperforms the FINN framework and traditional GPU and CPU platforms by achieving significant improvements of 1.08x, 1.7x, and 2.9x in terms of latency. Leveraging Vitis AI, our system achieved a detection speed of 32.7 frames per second (FPS) on the Kria KV260 FPGA with a power consumption rate of 5.6 W and an impressive mean Average Precision (mAP) score of 61.2% on the Hindi Indian Sign Language (ISL) dataset.

A multi-lingual sign language recognition system using machine learning

AbstractRecently, automatic sign language recognition field gets a great attention. Machine learning algorithms are mainly used to recognize hand gestures. Most of recent studies train their machine learning model using a specific sign language of a specific country such as the American Sign Language. In this paper, we propose a multi-lingual sign language system based machine learning that is called Multi-lingual Sign Languages Interpreter (MSLI) system. MSLI trains a machine learning model based on hand signs of multiple languages. It can detect the language of the input signs and their labels. In a case of input testing signs with the same language, the proposed system can provide two-steps recognition, where it only detects the language of the first sign, and then the rest signs are tested according to the recognized language. Also, MSLI can provide separate classification of signs per each language. Experiments were performed using 11 datasets with different languages. Separate and combined classification was performed on the input data. Experimental results show the accuracy of the proposed system. Training accuracy of the proposed system over most of the used separate different sign language datasets is approximately ranged from 90 to 100%. Also, most classification accuracy results of the test data of the separate datasets exceeded 90%. The combined classification of proposed MSLI archived training accuracy of 95.87% and testing accuracy of 92.33%.

Instant Sign Language Recognition by WAR Strategy Algorithm Based Tuned Machine Learning

Sign language serves as the primary means of communication utilized by individuals with hearing and speech disabilities. However, the comprehension of sign language by those without disabilities poses a significant challenge, resulting in a notable disparity in communication across society. Despite the utilization of numerous effective Machine learning techniques, there remains a minor compromise between accuracy rate and computing time when it comes to sign language recognition. A novel sign language recognition system is presented in this paper with an exceptionally accurate and expeditious, which is developed upon the recently devised metaheuristic WAR Strategy optimization algorithm. Following the preprocessing, both of spatial and temporal features has been extracted using the Linear Discriminant Analysis (LDA) and Gray-level cooccurrence matrix (GLCM) methods. Afterward, the WAR Strategy optimization algorithm has been adopted in two procedures, first in optimizing the extracted set of features, and second to fine-tune the hyperparameters of six standard machine learning models in order to achieve precise and efficient sign language recognition. The proposed system was assessed on sign language datasets of different languages (American, Arabic, and Malaysian) containing numerous variations. The proposed system attained a recognition accuracy ranging from 93.11% to 100% by employing multiple optimized machine learning classifiers and training time of 0.038–10.48 s. As demonstrated by the experimental outcomes, the proposed system is exceptionally efficient regarding time, complexity, generalization, and accuracy.

Research on traffic sign detection algorithm based on improved SSD in complex environments

Abstract In complex traffic sign environments, detection challenges include uneven size distribution, insufficient valid information, and difficulties in identifying targets under resource constraints, leading to missed and false detections. This study proposes an enhanced lightweight traffic sign detection algorithm based on SSD (Single Shot MultiBox Detector). By replacing the original backbone network with MobileNetV2, the model is streamlined to have fewer parameters, which improves generalization in complex environments. This modification significantly boosts the recall rate, achieving a better balance between precision and recall. By introducing the FPN(Feature Pyramid Network) combined with the CBAM(Convolutional Block Attention Module) attention mechanism, the detailed and semantic information between deep and shallow layers is fully integrated, reducing the loss of feature information, thus enhancing the strengthening of key information of traffic signs and the adaptability to different scales of traffic signs. Finally, by integrating the cross-attention mechanism, the algorithm's anti-interference ability in complex environments is improved, and the positioning accuracy of traffic signs is enhanced by capturing the dependency between different positions. Through ablation experiments and comparative experiments on a public traffic sign dataset, our improved SSD algorithm achieved an mAP of 89.97%. Compared with the original algorithm, the mAP increased by 12.41%, the recall rate increased by 18.38%, and the sum of precision and recall F1 increased by 14.6%. These improvements significantly enhance the performance of traffic sign detection in complex environments, thereby meeting the performance requirements of traffic sign detection.

StarCAN-PFD: An Efficient and Simplified Multi-Scale Feature Detection Network for Small Objects in Complex Scenarios

Small object detection in traffic sign applications often faces challenges like complex backgrounds, blurry samples, and multi-scale variations. Existing solutions tend to complicate the algorithms. In this study, we designed an efficient and simple algorithm network called StarCAN-PFD, based on the single-stage YOLOv8 framework, to accurately recognize small objects in complex scenarios. We proposed the StarCAN feature extraction network, which was enhanced with the Context Anchor Attention (CAA). We designed the Pyramid Focus and Diffusion Network (PFDNet) to address multi-scale information loss and developed the Detail-Enhanced Conv Shared Detect (DESDetect) module to improve the recognition of complex samples while keeping the network lightweight. Experiments on the CCTSDB dataset validated the effectiveness of each module. Compared to YOLOv8, our algorithm improved mAP@0.5 by 4%, reduced the model size to less than half, and demonstrated better performance on different traffic sign datasets. It excels at detecting small traffic sign targets in complex scenes, including challenging samples such as blurry, low-light night, occluded, and overexposed conditions, showcasing strong generalization ability.

Innovative healthcare solutions: robust hand gesture recognition of daily life routines using 1D CNN.

Hand gestures are an effective communication tool that may convey a wealth of information in a variety of sectors, including medical and education. E-learning has grown significantly in the last several years and is now an essential resource for many businesses. Still, there has not been much research conducted on the use of hand gestures in e-learning. Similar to this, gestures are frequently used by medical professionals to help with diagnosis and treatment. We aim to improve the way instructors, students, and medical professionals receive information by introducing a dynamic method for hand gesture monitoring and recognition. Six modules make up our approach: video-to-frame conversion, preprocessing for quality enhancement, hand skeleton mapping with single shot multibox detector (SSMD) tracking, hand detection using background modeling and convolutional neural network (CNN) bounding box technique, feature extraction using point-based and full-hand coverage techniques, and optimization using a population-based incremental learning algorithm. Next, a 1D CNN classifier is used to identify hand motions. After a lot of trial and error, we were able to obtain a hand tracking accuracy of 83.71% and 85.71% over the Indian Sign Language and WLASL datasets, respectively. Our findings show how well our method works to recognize hand motions. Teachers, students, and medical professionals can all efficiently transmit and comprehend information by utilizing our suggested system. The obtained accuracy rates highlight how our method might improve communication and make information exchange easier in various domains.

Cross-modal knowledge distillation for continuous sign language recognition

Continuous Sign Language Recognition (CSLR) is a task which converts a sign language video into a gloss sequence. The existing deep learning based sign language recognition methods usually rely on large-scale training data and rich supervised information. However, current sign language datasets are limited, and they are only annotated at sentence-level rather than frame-level. Inadequate supervision of sign language data poses a serious challenge for sign language recognition, which may result in insufficient training of sign language recognition models. To address above problems, we propose a cross-modal knowledge distillation method for continuous sign language recognition, which contains two teacher models and one student model. One of the teacher models is the Sign2Text dialogue teacher model, which takes a sign language video and a dialogue sentence as input and outputs the sign language recognition result. The other teacher model is the Text2Gloss translation teacher model, which targets to translate a text sentence into a gloss sequence. Both teacher models can provide information-rich soft labels to assist the training of the student model, which is a general sign language recognition model. We conduct extensive experiments on multiple commonly used sign language datasets, i.e., PHOENIX 2014T, CSL-Daily and QSL, the results show that the proposed cross-modal knowledge distillation method can effectively improve the sign language recognition accuracy by transferring multi-modal information from teacher models to the student model. Code is available at https://github.com/glq-1992/cross-modal-knowledge-distillation_new.

Impact of face swapping and data augmentation on sign language recognition

AbstractThis study addresses the challenge of improving communication between the deaf and hearing community by exploring different sign language recognition (SLR) techniques. Due to privacy issues and the need for validation by interpreters, creating large-scale sign language (SL) datasets can be difficult. The authors address this by presenting a new Spanish isolated sign language recognition dataset, CALSE-1000, consisting of 5000 videos representing 1000 glosses, with various signers and scenarios. The study also proposes using different computer vision techniques, such as face swapping and affine transformations, to augment the SL dataset and improve the accuracy of the model I3D trained using them. The results show that the inclusion of these augmentations during training leads to an improvement in accuracy in top-1 metrics by up to 11.7 points, top-5 by up to 8.8 points and top-10 by up to 9 points. This has great potential to improve the state of the art in other datasets and other models. Furthermore, the analysis confirms the importance of facial expressions in the model by testing with a facial omission dataset and shows how face swapping can be used to include new anonymous signers without the costly and time-consuming process of recording.

Classification of Traffic Signs Using Transfer Learning Methods

Transportation refers to a process based on the movement of people or vehicles from one place to another. Sea routes and roads have existed for centuries. They generally play a very important role in people's daily life, trade and industrial activities. Highway, a mode of transportation, is the first preferred mode of transportation worldwide. However, various signs and rules have been set by the authorities to prevent chaos on the highways. Traffic signs are the most important of these rules. In this study, transfer learning models (VGG16, VGG19, Xception and EfficientNet) are used to classify traffic signs using a state-of-art traffic signs dataset (German Traffic Sign Detection Benchmark-GTSDB). Accuracy was used as the classification evaluation criterion. The CNN model designed for the study gave the best result with an accuracy rate of 98% and a model competing with the literature was proposed.