Sign Language Dataset Research Articles

Exploring Explainable Artificial Intelligence Techniques for Interpretable Neural Networks in Traffic Sign Recognition Systems

Traffic Sign Recognition (TSR) plays a vital role in intelligent transportation systems (ITS) to improve road safety and optimize traffic management. While existing TSR models perform well in challenging scenarios, their lack of transparency and interpretability hinders reliability, trustworthiness, validation, and bias identification. To address this issue, we propose a Convolutional Neural Network (CNN)-based model for TSR and evaluate its performance on three benchmark datasets: German Traffic Sign Recognition Benchmark (GTSRB), Indian Traffic Sign Dataset (ITSD), and Belgian Traffic Sign Dataset (BTSD). The proposed model achieves an accuracy of 98.85% on GTSRB, 94.73% on ITSD, and 92.69% on BTSD, outperforming several state-of-the-art frameworks, such as VGG19, VGG16, ResNet50V2, MobileNetV2, DenseNet121, DenseNet201, NASNetMobile, and EfficientNet, while also providing faster training and response times. We further enhance our model by incorporating explainable AI (XAI) techniques, specifically, Local Interpretable Model-Agnostic Explanations (LIME) and Gradient-weighted Class Activation Mapping (Grad-CAM), providing clear insights of the proposed model decision-making process. This integration allows the extension of our TSR model to various engineering domains, including autonomous vehicles, advanced driver assistance systems (ADAS), and smart traffic control systems. The practical implementation of our model ensures real-time, accurate recognition of traffic signs, thus optimizing traffic flow and minimizing accident risks.

Enhancing Sign Language Recognition: Leveraging EfficientNet-B0 with Transformer-based Decoding

Sign language recognition (SLR) plays a crucial role in facilitating communication for the hearing-impaired community. Conventional methods for SLR have encountered difficulties in attaining both high precision and efficiency because of the intricate characteristics of sign language motions and the variability in articulation. We propose a novel framework for enhancing SLR by leveraging the efficiency of EfficientNet-B0 as a feature extractor and incorporating a transformer-based decoding mechanism for classification. The objective of our method is to enhance the precision and computational effectiveness of SLR systems, thereby making them more viable for real-world applications. Experimental results on two standard, commonly used sign language datasets: American Sign Language (ASL) and ASL with Digits. The proposed model achieves accuracies of 99.59% on the ASL dataset and an outstanding accuracy of 99.97% on the ASL with Digits dataset, outperforming all other state-of-the-art methods. These results highlight the effectiveness of our framework in accurately recognizing sign language gestures, making it highly suitable for realworld applications. Our study contributes to the advancement of SLR research by introducing a novel methodology that combines the efficiency of EfficientNet-B0 with the expressive capabilities of transformer-based decoding, ultimately improving communication accessibility for individuals who rely on sign language.

Machine Learning Based Customized Solution for Deafness and Mute People

Individuals with hearing or speech impairments often possess a higher degree of familiarity with sign language due to its widespread usage. Consequently, a noticeable communication divide exists between those with such impairments and the general populace. A pivotal solution in bridging this gap involves employing human sign language interpreters. Regrettably, the scarcity of sign language interpreters worldwide in comparison to the number of individuals with hearing or speech impairments has resulted in certain individuals being unable to bear the cost of a human interpreter each time they engage in conversation. To address this issue, it is imperative to automate communication in a manner that liberates the deaf community from dependence on human translators. This article centers on the development of an application capable of real-time conversion between American Sign Language and text, along with supplementary functionalities aimed at dismantling the communication barriers faced by individuals with hearing or speech challenges when interacting with the broader public. the primary objective of this application is to accurately perceive and interpret the user's sign language. The initial stage involves training the system to recognize and interpret signs utilizing algorithms for object recognition and motion tracking. For this purpose, a convolutional neural network model was trained employing a meticulously curated American Sign Language dataset. Subsequently, the recognized signs are translated into English, forming grammatically sound words. Incorporated within the system are a language translator and a virtual sign keyboard, augmenting its capabilities. Moreover, a text-to-text transformer, structured on an encoder-decoder framework, is harnessed to identify grammatical inaccuracies and generate coherent phrases. For a comprehensive understanding of the process, detailed elaboration will follow in subsequent sections.

Sign Language Recognition by using Spatio-Temporal Features

The recognition of sign language has significant importance as an application that addresses the communication challenges faced by those within the Deaf and hard-of-hearing populations. This study introduces a methodology for the identification of sign language using spatio-temporal attributes, specifically concentrating on two distinct sign language datasets: the CSL (Chinese Sign Language) alphabet and the ArSL (Arabic Sign Language) alphabet. The study's objective is to create a powerful recognition system capable of reliably identifying sign language motions. A combination of spatio-temporal characteristics collected from video sequences of sign language motions is used in the suggested technique. These characteristics capture the spatial layout as well as the temporal dynamics of the motions, allowing the model to recognise signals with more precision. Experiments were carried out on the CSL and ArSL alphabet datasets to validate the method's efficacy. The findings of the trials are encouraging. The suggested system recognized CSL alphabet signs with an accuracy of 90.87% and ArSL alphabet signs with an accuracy of 89.46%. These high accuracy rates show the power of the spatiotemporal feature-based method for sign language identification. The system's success on two different sign languages implies that it is adaptable and useful in a wide variety of sign language applications. This study contributes to the evolution of assistive technology, making sign language recognition more accessible and efficient for those who communicate using sign language. Furthermore, the findings pave the way for additional research into spatiotemporal feature-based approaches in sign language recognition, with potential applications in real-world settings such as sign language interpretation and communication support.

Toward Robust Arabic Sign Language Recognition via Vision Transformers and Local Interpretable Model-agnostic Explanations Integration

People with severe or substantial hearing loss find it difficult to communicate with others. Poor communication can have a significant impact on the mental health of deaf people. For individuals who are deaf or hard of hearing, sign language (SL) is the major mode of communication in their daily life. Motivated by the need to develop robust and interpretable models for the deaf community, this study presents a computer-aided diagnosis (CAD) framework for Arabic SL recognition. The interpretability and management of complicated spatial connections in SL images have been limited by prior studies using convolutional neural networks. To improve accuracy and offer model transparency, the proposed CAD framework incorporates state-of-the-art technologies such as local interpretable model-agnostic explanations (LIME) and vision transformers (ViTs). ViTs use self-attention mechanisms to interpret visuals in SL, capturing global dependencies. A stacking/voting strategy is then used to aggregate predictions from many ViT models, further optimizing the system. Two large datasets, the “ArSL21L: Arabic Sign Language Letter Dataset” and the “RGB Arabic Alphabets Sign Language Dataset,” totaling over 22,000 pictures, were used to validate this approach. Metrics including intersection over union, balanced accuracy, Youden’s index, Yule’s Q, F1 score, accuracy, precision, recall, and specificity were used to assess performance. The results show that the stacking method, which makes use of many ViT models, outperforms traditional models in every performance indicator and achieves an impressive accuracy of 99.46% and 99.88% on the ArSL21L and RGB datasets, respectively. For practical applications, interpretability is ensured by using LIME, which offers clear visual explanations for the model’s predictions.

Uyghur Sign Language Recognition Based on Improved YOLOv7

Sign language is an important way for deaf-mutes to communicate, and sign language recognition plays an important role in solving the communication problems between able-bodied people and deaf-mutes. In order to help them communicate better with others and gradually switch to universal sign language, this paper proposes a design of Uyghur sign language recognition system based on the improved YOLOv7. Through the self-made 2568 Uyghur sign language datasets, a total of 35 categories such as ‘time’, ‘you’, ‘morning’, etc. were involved. In terms of network improvement, the backbone network is improved, and the improved convolution structure of SimAM attention + PConv is used; Second, replace the EIoU loss function. The sign language recognition model used in this paper has achieved significant improvement in Uyghur sign language recognition, which fully demonstrates its potential for in-depth research and wide application.

Super-resolution reconstruction algorithm for dim and blurred traffic sign images in complex environments

<abstract> <p>In poor lighting and rainy and foggy bad weather environments, road traffic signs are blurred and have low recognition, etc. A super-resolution reconstruction algorithm for complex lighting and bad weather traffic sign images was proposed. First, a novel attention residual module was designed to incorporate an aggregated feature attention mechanism on the jump connection side of the base residual module so that the deep network can obtain richer detail information; second, a cross-layer jump connection feature fusion mechanism was adopted to enhance the flow of information across layers as well as to prevent the problem of gradient disappearance of the deep network to enhance the reconstruction of the edge detail information; and lastly, a positive-inverse dual-channel sub-pixel convolutional up-sampling method was designed to reconstruct super-resolution images to obtain better pixel and spatial information expression. The evaluation model was trained on the Chinese traffic sign dataset in a natural scene, and when the scaling factor is 4, the average values of PSNR and SSIM are improved by 0.031 when compared with the latest release of the deep learning-based super-resolution reconstruction algorithm for single-frame images, MICU (Multi-level Information Compensation and U-net), the average values of PSNR and SSIM are improved by 0.031 dB and 0.083, and the actual test average reaches 20.946 dB and 0.656. The experimental results show that the reconstructed image quality of this paper's algorithm is better than the mainstream algorithms of comparison in terms of objective indexes and subjective feelings. The super-resolution reconstructed image has a higher peak signal-to-noise ratio and perceptual similarity. It can provide certain technical support for the research of safe driving assistive devices in natural scenes under multi-temporal varying illumination conditions and bad weather.</p> </abstract>

Smart Glove for Sign Language Translation

Sign language is a vital mode of communication for deaf people, yet it presents a significant barrier when interacting with those who do not understand it. The advent of technology has paved the way for innovative solutions to bridge this communication gap. This abstract explores the development and implications of a smart glove designed for sign language translation (SLT). The primary aim of this study is to create a wearable device, the Smart Glove, capable of recognizing and translating sign language gestures into text or speech. Key objectives include designing a lightweight and ergonomic glove prototype, developing machine learning algorithms for sign language recognition, implementing real-time translation capabilities, evaluating the glove's accuracy and usability, and assessing the potential impact on facilitating communication for deaf people. The Smart Glove utilizes only one sensor, flex sensors, to capture hand movements and gestures. These data inputs are processed through a custom-built machine learning model trained on a comprehensive sign language dataset. Preliminary results indicate a high accuracy rate in recognizing sign language gestures, with an average recognition rate of over 90% across a diverse set of signs. While challenges such as expanding gesture recognition and refining translation algorithms remain, this technology offers a promising solution to break down communication barriers and enhance the quality of life for those who rely on sign language.

DATASET AND IMPROVED YOLOV7 FOR TEXT-BASED TRAFFIC SIGN DETECTION

Abstract. Traffic sign detection is an important part of autonomous driving technology, and it is also important to have a large-scale dataset applicable to Chinese traffic scenarios. The article proposes a text-based self-labelled traffic sign dataset which consists of 3153 images, of which 2903 images are used for training and 250 images are used for validation. And an improved YOLOv7 algorithm is provided that incorporates the BiFormer attention mechanism into the YOLOv7 network to enhance its ability to detect small objects. This approach has the advantage of improved accuracy but may increase runtime. To mitigate this problem, the improved YOLOv7 network undergoes model pruning to compress the model size and increase its speed. Experimental results show that the improved YOLOv7 network in this paper improves the average accuracy by 2.9% while maintaining almost the same speed as the original network. After testing, the model has a real-time effect and practical significance. In conclusion, the text-based self-annotated dataset and the improved YOLOv7 network proposed in this paper have important reference values for text-based traffic sign recognition in automatic driving assistance systems.

Automated sign language detection and classification using reptile search algorithm with hybrid deep learning

Sign language recognition (SLR) contains the capability to convert sign language gestures into spoken or written language. This technology is helpful for deaf persons or hard of hearing by providing them with a way to interact with people who do not know sign language. It is also be utilized for automatic captioning in live events and videos. There are distinct methods of SLR comprising deep learning (DL), computer vision (CV), and machine learning (ML). One general approach utilises cameras for capturing the signer's hand and body movements and processing the video data for recognizing the gestures. One of challenges with SLR comprises the variability in sign language through various cultures and individuals, the difficulty of certain signs, and require for realtime processing. This study introduces an Automated Sign Language Detection and Classification using Reptile Search Algorithm with Hybrid Deep Learning (SLDC-RSAHDL). The presented SLDC-RSAHDL technique detects and classifies different types of signs using DL and metaheuristic optimizers. In the SLDC-RSAHDL technique, MobileNet feature extractor is utilized to produce feature vectors, and its hyperparameters can be adjusted by manta ray foraging optimization (MRFO) technique. For sign language classification, the SLDC-RSAHDL technique applies HDL model, which incorporates the design of Convolutional Neural Network (CNN) and Long-Short Term Memory (LSTM). At last, the RSA was exploited for the optimal hyperparameter selection of the HDL model, which resulted in an improved detection rate. The experimental result analysis of the SLDC-RSAHDL technique on sign language dataset demonstrates the improved performance of the SLDC-RSAHDL system over other existing DL techniques.

Zero-Shot Traffic Sign Recognition Based on Midlevel Feature Matching

Traffic sign recognition is a complex and challenging yet popular problem that can assist drivers on the road and reduce traffic accidents. Most existing methods for traffic sign recognition use convolutional neural networks (CNNs) and can achieve high recognition accuracy. However, these methods first require a large number of carefully crafted traffic sign datasets for the training process. Moreover, since traffic signs differ in each country and there is a variety of traffic signs, these methods need to be fine-tuned when recognizing new traffic sign categories. To address these issues, we propose a traffic sign matching method for zero-shot recognition. Our proposed method can perform traffic sign recognition without training data by directly matching the similarity of target and template traffic sign images. Our method uses the midlevel features of CNNs to obtain robust feature representations of traffic signs without additional training or fine-tuning. We discovered that midlevel features improve the accuracy of zero-shot traffic sign recognition. The proposed method achieves promising recognition results on the German Traffic Sign Recognition Benchmark open dataset and a real-world dataset taken from Sapporo City, Japan.

Sign Language Dataset for Automatic Motion Generation.

Several sign language datasets are available in the literature. Most of them are designed for sign language recognition and translation. This paper presents a new sign language dataset for automatic motion generation. This dataset includes phonemes for each sign (specified in HamNoSys, a transcription system developed at the University of Hamburg, Hamburg, Germany) and the corresponding motion information. The motion information includes sign videos and the sequence of extracted landmarks associated with relevant points of the skeleton (including face, arms, hands, and fingers). The dataset includes signs from three different subjects in three different positions, performing 754 signs including the entire alphabet, numbers from 0 to 100, numbers for hour specification, months, and weekdays, and the most frequent signs used in Spanish Sign Language (LSE). In total, there are 6786 videos and their corresponding phonemes (HamNoSys annotations). From each video, a sequence of landmarks was extracted using MediaPipe. The dataset allows training an automatic system for motion generation from sign language phonemes. This paper also presents preliminary results in motion generation from sign phonemes obtaining a Dynamic Time Warping distance per frame of 0.37.

SDViT: Stacking of Distilled Vision Transformers for Hand Gesture Recognition

Hand gesture recognition (HGR) is a rapidly evolving field with the potential to revolutionize human–computer interactions by enabling machines to interpret and understand human gestures for intuitive communication and control. However, HGR faces challenges such as the high similarity of hand gestures, real-time performance, and model generalization. To address these challenges, this paper proposes the stacking of distilled vision transformers, referred to as SDViT, for hand gesture recognition. An initially pretrained vision transformer (ViT) featuring a self-attention mechanism is introduced to effectively capture intricate connections among image patches, thereby enhancing its capability to handle the challenge of high similarity between hand gestures. Subsequently, knowledge distillation is proposed to compress the ViT model and improve model generalization. Multiple distilled ViTs are then stacked to achieve higher predictive performance and reduce overfitting. The proposed SDViT model achieves a promising performance on three benchmark datasets for hand gesture recognition: the American Sign Language (ASL) dataset, the ASL with digits dataset, and the National University of Singapore (NUS) hand gesture dataset. The accuracies achieved on these datasets are 100.00%, 99.60%, and 100.00%, respectively.

Alleviating data insufficiency for Chinese sign language recognition

Continuous Chinese sign language recognition (CCSLR) methods have shown their strong ability to learn excellent model architectures from datasets. However, due to data insufficiency, it is difficult to complete the CCSLR task. In this work, we focus on a simple but important solution to alleviate data insufficiency: how to refine the model architecture of a CCSLR network to improve the robustness of feature processing by using some better-quality non-Chinese sign language datasets. To this end, a simple empirical study was first conducted to verify the feasibility of knowledge transfer in the CCSLR task. Surprisingly, just by pre-training of our recognition model on a foreign sign language dataset, we can refine the model architecture and improve its robustness significantly. To make it more practical, the key issue of how to fine-tune the existing feature processing models for effective guidance should be carefully investigated. Then, we propose a novel scheme for fine-tuning of pre-trained models named FTP, which updates the spatial feature extractor initialized by a pre-trained backbone and freezes the temporal feature extractor implemented by a better shareable transformer encoder. Compared with the baseline method, our FTP method can achieve significant performance improvement on the public dataset USTC-CCSL.

MIPA-ResGCN: a multi-input part attention enhanced residual graph convolutional framework for sign language recognition

Sign language (SL) is used as primary mode of communication by individuals who experience deafness and speech disorders. However, SL creates an inordinate communication barrier as most people are not acquainted with it. To solve this problem, many technological solutions using wearable devices, video, and depth cameras have been put forth. The ubiquitous nature of cameras in contemporary devices has resulted in the emergence of sign language recognition (SLR) using video sequence as a viable and unobtrusive substitute. Nonetheless, the utilization of SLR methods based on visual features, commonly known as appearance-based methods, presents notable computational complexities. In response to these challenges, this study introduces an accurate and computationally efficient pose-based approach for SLR. Our proposed approach comprises three key stages: pose extraction, handcrafted feature generation, and feature space mapping and recognition. Initially, an efficient off-the-shelf pose extraction algorithm is employed to extract pose information of various body parts of a subject captured in a video. Then, a multi-input stream has been generated using handcrafted features, i.e., joints, bone lengths, and bone angles. Finally, an efficient and lightweight residual graph convolutional network (ResGCN) along with a novel part attention mechanism, is proposed to encode body's spatial and temporal information in a compact feature space and recognize the signs performed. In addition to enabling effective learning during model training and offering cutting-edge accuracy, the proposed model significantly reduces computational complexity. Our proposed method is assessed on five challenging SL datasets, WLASL-100, WLASL-300, WLASL-1000, LSA-64, and MINDS-Libras, achieving state-of-the-art (SOTA) accuracies of 83.33 %, 72.90 %, 64.92 %, 100± 0 %, and 96.70± 1.07 %, respectively. Compared to previous approaches, we achieve superior performance while incurring a lower computational cost.

Constraints on Optimising Encoder-Only Transformers for Modelling Sign Language with Human Pose Estimation Keypoint Data.

Supervised deep learning models can be optimised by applying regularisation techniques to reduce overfitting, which can prove difficult when fine tuning the associated hyperparameters. Not all hyperparameters are equal, and understanding the effect each hyperparameter and regularisation technique has on the performance of a given model is of paramount importance in research. We present the first comprehensive, large-scale ablation study for an encoder-only transformer to model sign language using the improved Word-level American Sign Language dataset (WLASL-alt) and human pose estimation keypoint data, with a view to put constraints on the potential to optimise the task. We measure the impact a range of model parameter regularisation and data augmentation techniques have on sign classification accuracy. We demonstrate that within the quoted uncertainties, other than ℓ2 parameter regularisation, none of the regularisation techniques we employ have an appreciable positive impact on performance, which we find to be in contradiction to results reported by other similar, albeit smaller scale, studies. We also demonstrate that the model architecture is bounded by the small dataset size for this task over finding an appropriate set of model parameter regularisation and common or basic dataset augmentation techniques. Furthermore, using the base model configuration, we report a new maximum top-1 classification accuracy of 84% on 100 signs, thereby improving on the previous benchmark result for this model architecture and dataset.

Machine Learning-Based Hand Sign Recognition

Machine Learning-Based Hand Sign Recognition is a project aimed at developing a robust and efficient system for recognizing hand signs and gestures using state-of-the-art machine learning techniques. Hand sign recognition holds significant potential in a wide range of applications, including communication assistance for individuals with hearing impairments, human-computer interaction, and automation in various industries. This project leverages deep learning models, particularly convolutional neural networks (CNNs) and recurrent neural networks (RNNs), to analyze and classify hand signs and gestures. The system is trained on a diverse dataset of hand signs, encompassing different gestures, poses, and lighting conditions, to ensure its adaptability and reliability in real-world scenarios. The project's success is contingent on the fine-tuning of machine learning models, thorough testing, and the incorporation of efficient computer vision techniques. Through rigorous evaluation and validation, we aim to achieve high accuracy and real-time performance. As a result, Machine Learning-Based Hand Sign Recognition has the potential to improve accessibility and convenience for those with hearing impairments, enhance human-computer interaction, and contribute to advancements in automation and robotics across multiple domains

Exploring MediaPipe optimization strategies for real-time sign language recognition

The present study meticulously investigates optimization strategies for real-time sign language recognition (SLR) employing the MediaPipe framework. We introduce an innovative multi-modal methodology, amalgamating four distinct Long Short-Term Memory (LSTM) models dedicated to processing skeletal coordinates ascertained from the MediaPipe framework. Rigorous evaluations were executed on esteemed sign language datasets. Empirical findings underscore that the multi-modal approach significantly elevates the accuracy of the SLR model while preserving its real-time capabilities. In comparative analyses with prevalent MediaPipe-based models, our multi-modal strategy consistently manifested superior performance metrics. A distinguishing characteristic of this approach is its inherent adaptability, facilitating modifications within the LSTM layers, rendering it apt for a myriad of challenges and data typologies. Integrating the MediaPipe framework with real-time SLR markedly amplifies recognition precision, signifying a pivotal advancement in the discipline.

Deep Transfer Learning for Sign Language Image Classification: A Bisindo Dataset Study

This study aims to identify and categorize the BISINDO sign language dataset, primarily consisting of image data. Deep learning techniques are used, with three pre-trained models: ResNet50 for training, MobileNetV4 for validation, and InceptionV3 for testing. The primary objective is to evaluate and compare the performance of each model based on the loss function derived during training. The training success rate provides a rough idea of the ResNet50 model's understanding of the BISINDO dataset, while MobileNetV4 measures validation loss to understand the model's generalization abilities. The InceptionV3-evaluated test loss serves as the ultimate litmus test for the model's performance, evaluating its ability to classify unobserved sign language images. The results of these exhaustive experiments will determine the most effective model and achieve the highest performance in sign language recognition using the BISINDO dataset.

BP-SRM: A directly training algorithm for spiking neural network constructed by spike response model

Spiking neural networks (SNNs) have attracted widespread attention due to their unique bio-interpretability and low-power properties, but the non-differentiability of discrete spike sequences fired by spiking neurons brings difficulties to the learning of SNNs. Recently, surrogate gradient (SG) and back-propagation through time (BPTT) have provided an excellent idea for training SNNs constructed by leaky integrate-and-fire (LIF) neuron models. The LIF neuron model has been widely used in previous SNNs due to its simplicity and low computational cost, however, this also limits its simulation of biological neuron dynamics, reducing the biological interpretability of SNNs. In this paper, we generalize SG and BPTT to SNNs constructed by spike response model (SRM) and propose the BP-SRM algorithm. Specifically, we address why BPTT succeeded in LIF neurons-based SNNs but failed in SRM neurons-based SNNs in previous researches. Then we establish an iterative form of the SRM neuron model by selecting different state variables. Based on iterable SRM, we get the spatiotemporal dependencies between the state variables of SRM neurons-based SNNs, which allows us to derive the gradient and update weights in SNN by BP-SRM. Then we design temporal channel normalization for BP-SRM and verify the performance of the SNNs on static image dataset, dynamic image dataset and engineering dataset, including Fashion-MNIST, Weather Dataset, N-MNIST, American Sign Language Dataset and bearing fault diagnosis dataset. The experiment results indicate that BP-SRM achieves the state-of-the-art performance of SNNs.