Sign Language Recognition System Research Articles

Enhancing communication: Deep learning for Arabic sign language translation

Abstract This study explores the field of sign language recognition through machine learning, focusing on the development and comparative evaluation of various algorithms designed to interpret sign language. With the prevalence of hearing impairment affecting millions globally, efficient sign language recognition systems are increasingly critical for enhancing communication for the deaf and hard-of-hearing community. We review several studies, showcasing algorithms with accuracies ranging from 63.5 to 99.6%. Building on these works, we introduce a novel algorithm that has been rigorously tested and has demonstrated a perfect accuracy of 99.7%. Our proposed algorithm utilizes a sophisticated convolutional neural network architecture that outperforms existing models. This work details the methodology of the proposed system, which includes preprocessing, feature extraction, and a multi-layered CNN approach. The remarkable performance of our algorithm sets a new benchmark in the field and suggests significant potential for real-world application in assistive technologies. We conclude by discussing the impact of these findings and propose directions for future research to further improve the accessibility and effectiveness of sign language recognition systems.

TRAFFIC SIGN RECOGNITION IN CHALLENGING WEATHER CONDITIONS USING CONVOLUTIONAL NEURAL NETWORKS

The ability of autonomous driving systems to recognize traffic signs in a variety of environmental conditions is crucial to their reliability. This study uses convolutional neural networks (CNNs) to provide a novel method for improving the accuracy of traffic sign recognition systems under challenging weather situations. The research focuses on developing a CNN-based model that is trained using the augmented German Traffic Sign Recognition Benchmark (GTSRB) dataset, which contains over 50,000 labeled images of road signs across 43 categories. To attempt to overcome the limitations of traditional CNN models under adverse environmental conditions, this work presents adaptive feature extraction layers that are especially made to reduce visibility problems brought on by rain, fog, and snow. By taking a comprehensive approach, the model uses advanced data augmentation methods to simulate different weather scenarios, greatly increasing the diversity of the training dataset. Through an analysis of theoretical and practical aspects, the study demonstrates how CNNs enhance the accuracy and efficiency of road sign detection systems in a different weather condition. Research evaluates the model's effectiveness using metrics such as precision, recall, and F1-score, proving its capability to reduce false positives and accurately detect relevant road sign instances. Additionally, the paper highlights the importance of careful dataset preparation and augmentation, model optimization, and training enhancements to improve the performance of road sign detection systems. The results of this research offer benefits for intelligent transport systems, autonomous driving, and road safety, pointing towards further advancements in precise and dependable road sign recognition technology.

Breaking the silence: empowering the mute-deaf community through automatic sign language decoding.

The objective of this study is to develop a system for automatic sign language recognition to improve the quality of life for the mute-deaf community in Egypt. The system aims to bridge the communication gap by identifying and converting right-hand gestures into audible sounds or displayed text. To achieve the objectives, a convolutional neural network (CNN) model is employed. The model is trained to recognize right-hand gestures captured by an affordable web camera. A dataset was created with the help of six volunteers for training, testing, and validation purposes. The proposed system achieved an impressive average accuracy of 99.65 % in recognizing right-hand gestures, with high precision value of 95.11 %. The system effectively addressed the issue of gesture similarity between certain alphabets by successfully distinguishing between their respective gestures. The proposed system offers a promising solution for automatic sign language recognition, benefiting the mute-deaf community in Egypt. By accurately identifying and converting right-hand gestures, the system facilitates communication and interaction with the wider world. This technology has the potential to greatly enhance the quality oflife for individuals who are unable to speak or hear, promoting inclusivity and accessibility.

Fast and Lightweight Vision-Language Model for Adversarial Traffic Sign Detection

Several attacks have been proposed against autonomous vehicles and their subsystems that are powered by machine learning (ML). Road sign recognition models are especially heavily tested under various adversarial ML attack settings, and they have proven to be vulnerable. Despite the increasing research on adversarial ML attacks against road sign recognition models, there is little to no focus on defending against these attacks. In this paper, we propose the first defense method specifically designed for autonomous vehicles to detect adversarial ML attacks targeting road sign recognition models, which is called ViLAS (Vision-Language Model for Adversarial Traffic Sign Detection). The proposed defense method is based on a custom, fast, lightweight, and salable vision-language model (VLM) and is compatible with any existing traffic sign recognition system. Thanks to the orthogonal information coming from the class label text data through the language model, ViLAS leverages image context in addition to visual data for highly effective attack detection performance. In our extensive experiments, we show that our method consistently detects various attacks against different target models with high true positive rates while satisfying very low false positive rates. When tested against four state-of-the-art attacks targeting four popular action recognition models, our proposed detector achieves an average AUC of 0.94. This result achieves a 25.3% improvement over a state-of-the-art defense method proposed for generic image attack detection, which attains an average AUC of 0.75. We also show that our custom VLM is more suitable for an autonomous vehicle compared to the popular off-the-shelf VLM and CLIP in terms of speed (4.4 vs. 9.3 milliseconds), space complexity (0.36 vs. 1.6 GB), and performance (0.94 vs. 0.43 average AUC).

Traffic Sign Recognition using HIICNN Stack Ensemble Method

Abstract: This project introduces a novel deep convolutional neural network (CNN) model aimed at enhancing traffic sign recognition for autonomous vehicle technology and road safety. Employing a stacking ensemble approach, we combine multiple CNNs with diverse improvement techniques, including preprocessing for adverse weather conditions and shooting errors, and data augmentation to handle class imbalance. Through adjustments in learning rates during model training, we mitigate overfitting. Our ensemble model achieves an exceptional 99.4 percent test accuracy on the German Traffic Sign Recognition Benchmark (GTSRB) dataset, surpassing prior studies. We utilize gradient-weighted Class Activation Mapping (Grad-CAM) for model explain- ability and an evidential deep learning approach to quantify classification uncertainty. Our framework underscores the efficacy of combining preprocessing, ensemble learning, and transfer learning for superior perfor- mance and reliability in traffic sign recognition systems, contributing significantly to autonomous driving and road safety.

Traffic Sign Recognition Using Deep Learning

Abstract: Traffic sign recognition is a crucial component in advanced driver assistance systems and autonomous vehicles, enhancing road safety and overall transportation efficiency. Deep learning, specifically convolutional neural networks (CNNs), has emerged as a powerful tool for image based recognition tasks. The proposed deep learning-based traffic sign recognition system exhibits promising results, providing a foundation for the development of intelligent transportation systems. The trained model demonstrates remarkable accuracy in recognizing a wide range of traffic signs under different scenarios, including challenging lighting conditions and occlusions. The system's real-time performance is evaluated on both simulated and actual road scenarios, showcasing its feasibility for deployment in practical driving environments. The integration of such technology into vehicles holds great potential for enhancing road safety, reducing accidents, and contributing to the evolution of autonomous driving technology.

Multi-culture Sign Language Detection and Recognition Using Fine-tuned Convolutional Neural Network

The speech and hearing-impaired community use sign language as the primary means of communication. It is quite challenging for the general population to interpret or learn sign language completely. A sign language recognition system must be designed and developed to address this communication barrier. Most current sign language recognition systems rely on wearable sensors, keeping the recognition system unaffordable for most individuals. Moreover, the existing vision-based sign recognition frameworks do not consider all of the spatial and temporal information required for accurate recognition. A novel vision-based hybrid deep neural net methodology is proposed in this project for recognizing American Sign Language (ASL) and custom sign gestures. The proposed framework aims to establish a single framework for tracking and extracting multi-semantic properties, such as non-manual components and manual co-articulations. Furthermore, spatial feature extraction from the sign gestures is deployed using a Hybrid Deep Neural Network (HDNN) with atrous convolutions. The temporal and sequential feature extraction is carried out by employing attention-based HDNN. In addition, the distinguished abstract feature extraction is done using modified autoencoders. The discriminative feature extraction for differentiating the sign gestures from unwanted transition gestures is done by leveraging the hybrid attention module. The experimentation of the proposed model has been carried out on the novel multi-signer ASL and custom sign language dataset. The proposed sign language recognition framework with hybrid neural nets, specifically using HDNN, yields better results than other state-of-the-art frameworks. Additionally, a detection module is incorporated using Flask web, allowing manual input of images/signs for real-time recognition.

Isolated sign language recognition through integrating pose data and motion history images

This article presents an innovative approach for the task of isolated sign language recognition (SLR); this approach centers on the integration of pose data with motion history images (MHIs) derived from these data. Our research combines spatial information obtained from body, hand, and face poses with the comprehensive details provided by three-channel MHI data concerning the temporal dynamics of the sign. Particularly, our developed finger pose-based MHI (FP-MHI) feature significantly enhances the recognition success, capturing the nuances of finger movements and gestures, unlike existing approaches in SLR. This feature improves the accuracy and reliability of SLR systems by more accurately capturing the fine details and richness of sign language. Additionally, we enhance the overall model accuracy by predicting missing pose data through linear interpolation. Our study, based on the randomized leaky rectified linear unit (RReLU) enhanced ResNet-18 model, successfully handles the interaction between manual and non-manual features through the fusion of extracted features and classification with a support vector machine (SVM). This innovative integration demonstrates competitive and superior results compared to current methodologies in the field of SLR across various datasets, including BosphorusSign22k-general, BosphorusSign22k, LSA64, and GSL, in our experiments.

Sign language recognition based on dual-path background erasure convolutional neural network

Sign language is an important way to provide expression information to people with hearing and speaking disabilities. Therefore, sign language recognition has always been a very important research topic. However, many sign language recognition systems currently require complex deep models and rely on expensive sensors, which limits the application scenarios of sign language recognition. To address this issue, based on computer vision, this study proposed a lightweight, dual-path background erasing deep convolutional neural network (DPCNN) model for sign language recognition. The DPCNN consists of two paths. One path is used to learn the overall features, while the other path learns the background features. The background features are gradually subtracted from the overall features to obtain an effective representation of hand features. Then, these features are flatten into a one-dimensional layer, and pass through a fully connected layer with an output unit of 128. Finally, use a fully connected layer with an output unit of 24 as the output layer. Based on the ASL Finger Spelling dataset, the total accuracy and Macro-F1 scores of the proposed method is 99.52% and 0.997, respectively. More importantly, the proposed method can be applied to small terminals, thereby improving the application scenarios of sign language recognition. Through experimental comparison, the dual path background erasure network model proposed in this paper has better generalization ability.

Sign Language Recognition Using Machine Learning

Communication is very important in human daily life and the most widely used type of communication is verbal communication. But there are people with hearing and speech impairment who cannot communicate verbally and the language which they use for communication is sign language. Many other languages, tools are being developed for inter-language translation from sign language to text. There has been a lot of research done in the field of American Sign Language but the work is limited in the case of Indian Sign Language. This is due to lack of standards and the variation in the language. The proposed system aims to recognize Indian Sign Language digit gestures and convert it into text. By using Machine Learning Techniques, sign language recognition leads to the development of a more accurate and robust system. As Deep learning techniques, ResNet100 and ensemble models continue to evolve, sign language recognition system plays a transformative role in bridging the communication gap between deaf and hearing individuals. It helps the user to recognize the sign language by using this proposed system.

MLMSign: Multi-lingual multi-modal illumination-invariant sign language recognition

Sign language (SL) serves as a visual communication tool bearing great significance for deaf people to interact with others and facilitate their daily life. Wide varieties of SLs and the lack of interpretation knowledge necessitate developing automated sign language recognition (SLR) systems to attenuate the communication gap between the deaf and hearing communities. Despite numerous advanced static SLR systems, they are not practical and favorable enough for real-life scenarios once assessed simultaneously from different critical aspects: accuracy in dealing with high intra- and slight inter-class variations, robustness, computational complexity, and generalization ability. To this end, we propose a novel multi-lingual multi-modal SLR system, namely MLMSign, by taking full strengths of hand-crafted features and deep learning models to enhance the performance and the robustness of the system against illumination changes while minimizing computational cost. The RGB sign images and 2D visualizations of their hand-crafted features, i.e., Histogram of Oriented Gradients (HOG) features and a∗ channel of L∗a∗b∗ color space, are employed as three input modalities to train a novel Convolutional Neural Network (CNN). The number of layers, filters, kernel size, learning rate, and optimization technique are carefully selected through an extensive parametric study to minimize the computational cost without compromising accuracy. The system’s performance and robustness are significantly enhanced by jointly deploying the models of these three modalities through ensemble learning. The impact of each modality is optimized based on their impact coefficient determined by grid search. In addition to the comprehensive quantitative assessment, the capabilities of our proposed model and the effectiveness of ensembling over three modalities are evaluated qualitatively using the Grad-CAM visualization model. Experimental results on the test data with additional illumination changes verify the high robustness of our system in dealing with overexposed and underexposed lighting conditions. Achieving a high accuracy (>99.33%) on six benchmark datasets (i.e., Massey, Static ASL, NUS II, TSL Fingerspelling, BdSL36v1, and PSL) demonstrates that our system notably outperforms the recent state-of-the-art approaches with a minimum number of parameters and high generalization ability over complex datasets. Its promising performance for four different sign languages makes it a feasible system for multi-lingual applications.

Sign Language Recognition using CNN

Abstract: The sign language Recognition System is a technology that bridges the gap between the deaf people and the hearing world. Sign language is a vital mode of expression for deaf people, yet effective communication remains a challenge. This project aims to develop a robust Sign Language Recognition System capable of accurately translating sign language gestures into text. The system utilizes a deep learning approach, specifically convolutional neural networks (CNNs) to analyze video input of sign language and then generate the corresponding output. The project involves data collection, pre-processing, segmentation, feature extraction, model training, and evaluation. The proposed SLR system has the potential to enhance. Communication and accessibility for deaf individuals, promoting inclusivity and improving their quality of life. The project represents a comprehensive exploration of both technical and ethical aspects in the realm of computer vision and deep learning applications.

Development of Dynamic Image Recognition System for Hand Sign Language into Audio and Visual Output using Artificial Neural Networks

Abstract: This system provides a simple solution to a complex problem faced by deaf and dumb community. This paper visualizes the development web application of a real-time image recognition system whose main aim is to convert or translate the hand signs used in sign language in textual or visual and audio form output by using artificial neural networks. This system used computer vision to see signs as an input for processing and makes accurate predictions by using deep learning methods in realtime. By using artificial neural networks (ANN), the system achieves better performance in predicting or recognizing hand signs. The system makes communication between people using sign language and people who don’t know sign language effective by providing output as an audio and texted / visual, which even helps people in both ways. The project aims only to provide the society a solution by helping people who use sign language to express themselves to make it more effective and give them an ability to speak by using the technologies.

All Country Sign Language Recognition System to Help Deaf and Mute People

With the use of contemporary technology, the "All Country Sign Language Recognition System to Help Deaf and Mute People" project seeks to provide a strong solution that will close communication barriers between the deaf and mute population and the general public. The project aims to use machine learning techniques and computer vision techniques to properly understand and convert sign language motions into generally used languages, with a concentration on Indian sign language. Deaf and mute people will benefit from increased accessibility and inclusion in a variety of spheres of everyday life, including social interactions, work, and education, by virtue of the implementation of this system. The initiative helps to create a more inclusive society in addition to addressing the urgent demand for efficient communication tools for the deaf and mute communities.In this people can interact using different country sign languages like ISL, ASL, BSL, etc.

Review Paper on SignSense : An AI Framework for Sign Language Recognition

In this project, we propose an ensemble learning-based system for Sign Language Recognition (SLR) integrated with an Explainable AI (XAI) component called SignExplainer. Our goal is to enhance transparency and trust in SLR systems by providing interpretable predictions. The ensemble learning architecture is designed to recognize sign gestures from images, and the SignExplainer module generates statistical values to evaluate prediction correctness. Performance evaluation on benchmark datasets like ASL and BSL demonstrates the effectiveness of our approach in interpreting predictions from various machine learning and deep learning models. Future work aims to extend this methodology to real-time applications and other Sign Languages, advancing accessibility and inclusivity for the hearing-impaired community

SignSense: AI Framework for Sign Language Recognition

Sign Language recognition is a pioneering framework designed to advance the field of Sign Language Recognition (SLR) through the innovative application of ensemble deep learning models. The primary goal of this research is to significantly improve the accuracy, resilience and interpretability of SLR systems. Leveraging the unique features of ResNet within an ensemble learning paradigm. The key component of InceptionResNetv2 architecture is its deep and effective feature extraction capabilities. The utilization of InceptionResNet model enhances the model ability to capture intricate details crucial for accurate sign language recognition. This framework is also to scale seamlessly, accommodating an expanding vocabulary of signs, diverse users and dynamic environmental conditions without compromising performance.

Empowering Nonverbal Communication: A Comprehensive Examination of Sign Language Recognition Systems and Implementation of Interpretation

This paper provides a analysis of previous research on sign language recognition systems and their impact on Providing meaningful interaction among non-verbal individuals. It introduces a real-time sign language interpreter that translates text into gestures and vice-versa, accommodating regional sign languages and promoting personal expression. In a manner distinct from previous systems, it emphasizes individualization and includes text chat features for seamless communication. By enabling users to create and share their own gestures, it Promotes cultural diversity and authentic self-expression, promoting inclusivity and empowerment within the non-verbal community. This innovative solution also alleviates the pressure on non-verbal communication users to perform gestures in front of a camera for accurate recognition, addressing issues with outdated sign languages and the lack of personal sign gestures. Key Word: Sign Language Recognition, Text-to-sign Translation, Sign Language Interpreter, Sign Language Evolution, Recognition Models, Recognition Techniques.

Occlusion robust sign language recognition system for indian sign language using CNN and pose features

Occlusion robust sign language recognition system for indian sign language using CNN and pose features

A Review on Sign Language Recognition and Learning System

This paper presents innovative approaches to various aspects of sign language recognition and learning, catering to the diverse needs within this field. It introduces a system utilizing 2D image sampling and concatenation, trained with convolutional neural networks, achieving accurate and robust sign recognition even with low-cost cameras. Another system, SignQuiz, offers a cost-effective web-based solution for learning finger-spelled signs in Indian Sign Language (ISL), outperforming traditional printed mediums. Additionally, a dynamic hand gesture recognition system employing deep learning architectures demonstrates superior performance over existing methods, addressing challenges such as hand segmentation and sequence modeling. Furthermore, an efficient deep convolutional neural network approach for hand gesture recognition, including transfer learning, achieves high recognition rates across various datasets. Finally, a robust model for static sign recognition in ISL, utilizing CNNs and evaluating on a diverse dataset, attains exceptional accuracy and outperforms previous works. These contributions collectively advance the field of sign language recognition and learning, offering solutions that are accurate, cost-effective, and efficient, thereby facilitating better communication and interaction for the hearing-impaired community. Key Words: CNN, Sign language recognition, Hand gesture recognition, Machine learning, Transfer Learning, SVM.

Sign Language Recognition System

Abstract: Sign language serves as a primary mode of communication for the Deaf and hard-of-hearing community. This paper presents a Sign Language Recognition System (SLRS) designed to facilitate seamless communication between individuals proficient in sign language and those who may not share this proficiency. The system employs a multi-faceted approach, integrating computer vision, machine learning, and signal processing techniques to accurately interpret and recognize sign language gestures. The methodology involves data collection from diverse signing styles, preprocessing to enhance data quality, feature extraction capturing key aspects of sign language expressions, and the selection and training of machine learning models. The system aims to represent sign language gestures effectively, providing a foundation for real-time recognition. Integration into hardware or software platforms ensures practical applications in various settings, including education, employment, and public spaces.