Sign Language Recognition System Research Articles

Traffic Sign Recognition using HIICNN Stack Ensemble Method

Abstract: This project introduces a novel deep convolutional neural network (CNN) model aimed at enhancing traffic sign recognition for autonomous vehicle technology and road safety. Employing a stacking ensemble approach, we combine multiple CNNs with diverse improvement techniques, including preprocessing for adverse weather conditions and shooting errors, and data augmentation to handle class imbalance. Through adjustments in learning rates during model training, we mitigate overfitting. Our ensemble model achieves an exceptional 99.4 percent test accuracy on the German Traffic Sign Recognition Benchmark (GTSRB) dataset, surpassing prior studies. We utilize gradient-weighted Class Activation Mapping (Grad-CAM) for model explain- ability and an evidential deep learning approach to quantify classification uncertainty. Our framework underscores the efficacy of combining preprocessing, ensemble learning, and transfer learning for superior perfor- mance and reliability in traffic sign recognition systems, contributing significantly to autonomous driving and road safety.

Traffic Sign Recognition Using Deep Learning

Abstract: Traffic sign recognition is a crucial component in advanced driver assistance systems and autonomous vehicles, enhancing road safety and overall transportation efficiency. Deep learning, specifically convolutional neural networks (CNNs), has emerged as a powerful tool for image based recognition tasks. The proposed deep learning-based traffic sign recognition system exhibits promising results, providing a foundation for the development of intelligent transportation systems. The trained model demonstrates remarkable accuracy in recognizing a wide range of traffic signs under different scenarios, including challenging lighting conditions and occlusions. The system's real-time performance is evaluated on both simulated and actual road scenarios, showcasing its feasibility for deployment in practical driving environments. The integration of such technology into vehicles holds great potential for enhancing road safety, reducing accidents, and contributing to the evolution of autonomous driving technology.

Multi-culture Sign Language Detection and Recognition Using Fine-tuned Convolutional Neural Network

The speech and hearing-impaired community use sign language as the primary means of communication. It is quite challenging for the general population to interpret or learn sign language completely. A sign language recognition system must be designed and developed to address this communication barrier. Most current sign language recognition systems rely on wearable sensors, keeping the recognition system unaffordable for most individuals. Moreover, the existing vision-based sign recognition frameworks do not consider all of the spatial and temporal information required for accurate recognition. A novel vision-based hybrid deep neural net methodology is proposed in this project for recognizing American Sign Language (ASL) and custom sign gestures. The proposed framework aims to establish a single framework for tracking and extracting multi-semantic properties, such as non-manual components and manual co-articulations. Furthermore, spatial feature extraction from the sign gestures is deployed using a Hybrid Deep Neural Network (HDNN) with atrous convolutions. The temporal and sequential feature extraction is carried out by employing attention-based HDNN. In addition, the distinguished abstract feature extraction is done using modified autoencoders. The discriminative feature extraction for differentiating the sign gestures from unwanted transition gestures is done by leveraging the hybrid attention module. The experimentation of the proposed model has been carried out on the novel multi-signer ASL and custom sign language dataset. The proposed sign language recognition framework with hybrid neural nets, specifically using HDNN, yields better results than other state-of-the-art frameworks. Additionally, a detection module is incorporated using Flask web, allowing manual input of images/signs for real-time recognition.

Isolated sign language recognition through integrating pose data and motion history images

This article presents an innovative approach for the task of isolated sign language recognition (SLR); this approach centers on the integration of pose data with motion history images (MHIs) derived from these data. Our research combines spatial information obtained from body, hand, and face poses with the comprehensive details provided by three-channel MHI data concerning the temporal dynamics of the sign. Particularly, our developed finger pose-based MHI (FP-MHI) feature significantly enhances the recognition success, capturing the nuances of finger movements and gestures, unlike existing approaches in SLR. This feature improves the accuracy and reliability of SLR systems by more accurately capturing the fine details and richness of sign language. Additionally, we enhance the overall model accuracy by predicting missing pose data through linear interpolation. Our study, based on the randomized leaky rectified linear unit (RReLU) enhanced ResNet-18 model, successfully handles the interaction between manual and non-manual features through the fusion of extracted features and classification with a support vector machine (SVM). This innovative integration demonstrates competitive and superior results compared to current methodologies in the field of SLR across various datasets, including BosphorusSign22k-general, BosphorusSign22k, LSA64, and GSL, in our experiments.

Sign language recognition based on dual-path background erasure convolutional neural network

Sign language is an important way to provide expression information to people with hearing and speaking disabilities. Therefore, sign language recognition has always been a very important research topic. However, many sign language recognition systems currently require complex deep models and rely on expensive sensors, which limits the application scenarios of sign language recognition. To address this issue, based on computer vision, this study proposed a lightweight, dual-path background erasing deep convolutional neural network (DPCNN) model for sign language recognition. The DPCNN consists of two paths. One path is used to learn the overall features, while the other path learns the background features. The background features are gradually subtracted from the overall features to obtain an effective representation of hand features. Then, these features are flatten into a one-dimensional layer, and pass through a fully connected layer with an output unit of 128. Finally, use a fully connected layer with an output unit of 24 as the output layer. Based on the ASL Finger Spelling dataset, the total accuracy and Macro-F1 scores of the proposed method is 99.52% and 0.997, respectively. More importantly, the proposed method can be applied to small terminals, thereby improving the application scenarios of sign language recognition. Through experimental comparison, the dual path background erasure network model proposed in this paper has better generalization ability.

Sign Language Recognition Using Machine Learning

Communication is very important in human daily life and the most widely used type of communication is verbal communication. But there are people with hearing and speech impairment who cannot communicate verbally and the language which they use for communication is sign language. Many other languages, tools are being developed for inter-language translation from sign language to text. There has been a lot of research done in the field of American Sign Language but the work is limited in the case of Indian Sign Language. This is due to lack of standards and the variation in the language. The proposed system aims to recognize Indian Sign Language digit gestures and convert it into text. By using Machine Learning Techniques, sign language recognition leads to the development of a more accurate and robust system. As Deep learning techniques, ResNet100 and ensemble models continue to evolve, sign language recognition system plays a transformative role in bridging the communication gap between deaf and hearing individuals. It helps the user to recognize the sign language by using this proposed system.

MLMSign: Multi-lingual multi-modal illumination-invariant sign language recognition

Sign language (SL) serves as a visual communication tool bearing great significance for deaf people to interact with others and facilitate their daily life. Wide varieties of SLs and the lack of interpretation knowledge necessitate developing automated sign language recognition (SLR) systems to attenuate the communication gap between the deaf and hearing communities. Despite numerous advanced static SLR systems, they are not practical and favorable enough for real-life scenarios once assessed simultaneously from different critical aspects: accuracy in dealing with high intra- and slight inter-class variations, robustness, computational complexity, and generalization ability. To this end, we propose a novel multi-lingual multi-modal SLR system, namely MLMSign, by taking full strengths of hand-crafted features and deep learning models to enhance the performance and the robustness of the system against illumination changes while minimizing computational cost. The RGB sign images and 2D visualizations of their hand-crafted features, i.e., Histogram of Oriented Gradients (HOG) features and a∗ channel of L∗a∗b∗ color space, are employed as three input modalities to train a novel Convolutional Neural Network (CNN). The number of layers, filters, kernel size, learning rate, and optimization technique are carefully selected through an extensive parametric study to minimize the computational cost without compromising accuracy. The system’s performance and robustness are significantly enhanced by jointly deploying the models of these three modalities through ensemble learning. The impact of each modality is optimized based on their impact coefficient determined by grid search. In addition to the comprehensive quantitative assessment, the capabilities of our proposed model and the effectiveness of ensembling over three modalities are evaluated qualitatively using the Grad-CAM visualization model. Experimental results on the test data with additional illumination changes verify the high robustness of our system in dealing with overexposed and underexposed lighting conditions. Achieving a high accuracy (>99.33%) on six benchmark datasets (i.e., Massey, Static ASL, NUS II, TSL Fingerspelling, BdSL36v1, and PSL) demonstrates that our system notably outperforms the recent state-of-the-art approaches with a minimum number of parameters and high generalization ability over complex datasets. Its promising performance for four different sign languages makes it a feasible system for multi-lingual applications.

Sign Language Recognition using CNN

Abstract: The sign language Recognition System is a technology that bridges the gap between the deaf people and the hearing world. Sign language is a vital mode of expression for deaf people, yet effective communication remains a challenge. This project aims to develop a robust Sign Language Recognition System capable of accurately translating sign language gestures into text. The system utilizes a deep learning approach, specifically convolutional neural networks (CNNs) to analyze video input of sign language and then generate the corresponding output. The project involves data collection, pre-processing, segmentation, feature extraction, model training, and evaluation. The proposed SLR system has the potential to enhance. Communication and accessibility for deaf individuals, promoting inclusivity and improving their quality of life. The project represents a comprehensive exploration of both technical and ethical aspects in the realm of computer vision and deep learning applications.

Development of Dynamic Image Recognition System for Hand Sign Language into Audio and Visual Output using Artificial Neural Networks

Abstract: This system provides a simple solution to a complex problem faced by deaf and dumb community. This paper visualizes the development web application of a real-time image recognition system whose main aim is to convert or translate the hand signs used in sign language in textual or visual and audio form output by using artificial neural networks. This system used computer vision to see signs as an input for processing and makes accurate predictions by using deep learning methods in realtime. By using artificial neural networks (ANN), the system achieves better performance in predicting or recognizing hand signs. The system makes communication between people using sign language and people who don’t know sign language effective by providing output as an audio and texted / visual, which even helps people in both ways. The project aims only to provide the society a solution by helping people who use sign language to express themselves to make it more effective and give them an ability to speak by using the technologies.

All Country Sign Language Recognition System to Help Deaf and Mute People

With the use of contemporary technology, the "All Country Sign Language Recognition System to Help Deaf and Mute People" project seeks to provide a strong solution that will close communication barriers between the deaf and mute population and the general public. The project aims to use machine learning techniques and computer vision techniques to properly understand and convert sign language motions into generally used languages, with a concentration on Indian sign language. Deaf and mute people will benefit from increased accessibility and inclusion in a variety of spheres of everyday life, including social interactions, work, and education, by virtue of the implementation of this system. The initiative helps to create a more inclusive society in addition to addressing the urgent demand for efficient communication tools for the deaf and mute communities.In this people can interact using different country sign languages like ISL, ASL, BSL, etc.

Review Paper on SignSense : An AI Framework for Sign Language Recognition

In this project, we propose an ensemble learning-based system for Sign Language Recognition (SLR) integrated with an Explainable AI (XAI) component called SignExplainer. Our goal is to enhance transparency and trust in SLR systems by providing interpretable predictions. The ensemble learning architecture is designed to recognize sign gestures from images, and the SignExplainer module generates statistical values to evaluate prediction correctness. Performance evaluation on benchmark datasets like ASL and BSL demonstrates the effectiveness of our approach in interpreting predictions from various machine learning and deep learning models. Future work aims to extend this methodology to real-time applications and other Sign Languages, advancing accessibility and inclusivity for the hearing-impaired community

SignSense: AI Framework for Sign Language Recognition

Sign Language recognition is a pioneering framework designed to advance the field of Sign Language Recognition (SLR) through the innovative application of ensemble deep learning models. The primary goal of this research is to significantly improve the accuracy, resilience and interpretability of SLR systems. Leveraging the unique features of ResNet within an ensemble learning paradigm. The key component of InceptionResNetv2 architecture is its deep and effective feature extraction capabilities. The utilization of InceptionResNet model enhances the model ability to capture intricate details crucial for accurate sign language recognition. This framework is also to scale seamlessly, accommodating an expanding vocabulary of signs, diverse users and dynamic environmental conditions without compromising performance.

Empowering Nonverbal Communication: A Comprehensive Examination of Sign Language Recognition Systems and Implementation of Interpretation

This paper provides a analysis of previous research on sign language recognition systems and their impact on Providing meaningful interaction among non-verbal individuals. It introduces a real-time sign language interpreter that translates text into gestures and vice-versa, accommodating regional sign languages and promoting personal expression. In a manner distinct from previous systems, it emphasizes individualization and includes text chat features for seamless communication. By enabling users to create and share their own gestures, it Promotes cultural diversity and authentic self-expression, promoting inclusivity and empowerment within the non-verbal community. This innovative solution also alleviates the pressure on non-verbal communication users to perform gestures in front of a camera for accurate recognition, addressing issues with outdated sign languages and the lack of personal sign gestures. Key Word: Sign Language Recognition, Text-to-sign Translation, Sign Language Interpreter, Sign Language Evolution, Recognition Models, Recognition Techniques.

Occlusion robust sign language recognition system for indian sign language using CNN and pose features

Occlusion robust sign language recognition system for indian sign language using CNN and pose features

A Review on Sign Language Recognition and Learning System

This paper presents innovative approaches to various aspects of sign language recognition and learning, catering to the diverse needs within this field. It introduces a system utilizing 2D image sampling and concatenation, trained with convolutional neural networks, achieving accurate and robust sign recognition even with low-cost cameras. Another system, SignQuiz, offers a cost-effective web-based solution for learning finger-spelled signs in Indian Sign Language (ISL), outperforming traditional printed mediums. Additionally, a dynamic hand gesture recognition system employing deep learning architectures demonstrates superior performance over existing methods, addressing challenges such as hand segmentation and sequence modeling. Furthermore, an efficient deep convolutional neural network approach for hand gesture recognition, including transfer learning, achieves high recognition rates across various datasets. Finally, a robust model for static sign recognition in ISL, utilizing CNNs and evaluating on a diverse dataset, attains exceptional accuracy and outperforms previous works. These contributions collectively advance the field of sign language recognition and learning, offering solutions that are accurate, cost-effective, and efficient, thereby facilitating better communication and interaction for the hearing-impaired community. Key Words: CNN, Sign language recognition, Hand gesture recognition, Machine learning, Transfer Learning, SVM.

Sign Language Recognition System

Abstract: Sign language serves as a primary mode of communication for the Deaf and hard-of-hearing community. This paper presents a Sign Language Recognition System (SLRS) designed to facilitate seamless communication between individuals proficient in sign language and those who may not share this proficiency. The system employs a multi-faceted approach, integrating computer vision, machine learning, and signal processing techniques to accurately interpret and recognize sign language gestures. The methodology involves data collection from diverse signing styles, preprocessing to enhance data quality, feature extraction capturing key aspects of sign language expressions, and the selection and training of machine learning models. The system aims to represent sign language gestures effectively, providing a foundation for real-time recognition. Integration into hardware or software platforms ensures practical applications in various settings, including education, employment, and public spaces.

Empowering Communication with CNN Sign Language Recognition

Abstract: The SignSense project presents an innovative solution aimed at breaking the communication barriers faced by the hearing-impaired community. Leveraging advanced technologies, including Convolutional Neural Networks (CNNs) and Javabased algorithms, SignSense pioneers a Sign Language Recognition System. This system empowers individuals with hearing impairments to express themselves naturally, ensuring accurate translation of sign language gestures into text or voice. The project's motivation stems from fostering inclusivity and understanding between hearing-impaired individuals and the broader society.

Enhanced Sign Language Communication through Deep Learning-based Gesture and Action Recognition and Correction

Sign language is a crucial form of communication for the hearing impaired, presenting unique challenges for technological interpretation. This project focuses on developing a Sign Language Detection Using Action Recognition (SLDAR) system to address these challenges. Leveraging Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and multimodal fusion techniques, our system aims to accurately detect and recognize sign language gestures in real-time. The project involves collecting and annotating a diverse dataset comprising video, audio, and text data. We then design and train a deep learning model that combines features from these modalities to improve accuracy and robustness.The project involves collecting and annotating a diverse dataset, designing a deep learning model for multimodal fusion, and implementing a user-friendly interface for real-time interaction. The system's performance is evaluated using metrics such as accuracy, precision, and recall. Our goal is to enhance accessibility and communication for the hearing impaired through the development of this advanced sign language recognition system. Keywords:Acoustic Features, Linguistic Context, Mean Opinion Score (MOS), Natural Language Processing, NLP Algorithms, Signal-to-Noise Ratio (SNR), Speech Enhancement, Speech Quality, Telecommunication Systems, Voice Assistants.

Enhancing Signer-Independent Recognition of Isolated Sign Language through Advanced Deep Learning Techniques and Feature Fusion

Sign Language Recognition (SLR) systems are crucial bridges facilitating communication between deaf or hard-of-hearing individuals and the hearing world. Existing SLR technologies, while advancing, often grapple with challenges such as accurately capturing the dynamic and complex nature of sign language, which includes both manual and non-manual elements like facial expressions and body movements. These systems sometimes fall short in environments with different backgrounds or lighting conditions, hindering their practical applicability and robustness. This study introduces an innovative approach to isolated sign language word recognition using a novel deep learning model that combines the strengths of both residual three-dimensional (R3D) and temporally separated (R(2+1)D) convolutional blocks. The R3(2+1)D-SLR network model demonstrates a superior ability to capture the intricate spatial and temporal features crucial for accurate sign recognition. Our system combines data from the signer’s body, hands, and face, extracted using the R3(2+1)D-SLR model, and employs a Support Vector Machine (SVM) for classification. It demonstrates remarkable improvements in accuracy and robustness across various backgrounds by utilizing pose data over RGB data. With this pose-based approach, our proposed system achieved 94.52% and 98.53% test accuracy in signer-independent evaluations on the BosphorusSign22k-general and LSA64 datasets.

An Expert System for Indian Sign Language Recognition Using Spatial Attention–based Feature and Temporal Feature

Sign Language (SL) is the only means of communication for the hearing-impaired people. Normal people have difficulty understanding SL, resulting in a communication barrier between hearing impaired people and hearing community. However, the Sign Language Recognition System (SLRS) has helped to bridge the communication gap. Many SLRs are proposed for recognizing SL; however, a limited number of works are reported for Indian Sign Language (ISL). Most of the existing SLRS focus on global features other than the Region of Interest (ROI). Focusing more on the hand region and extracting local features from the ROI improves system accuracy. The attention mechanism is a widely used technique for emphasizing the ROI. However, only a few SLRS used the attention method. They employed the Convolution Block Attention Module and temporal attention but Spatial Attention (SA) is not utilized in previous SLRS. Therefore, a novel SA based SLRS named Spatial Attention-based Sign Language Recognition Module (SASLRM) is proposed to recognize ISL words for emergency situations. SASLRM recognizes ISL words by combining convolution features from a pretrained VGG-19 model and attention features from a SA module. The proposed model accomplished an average accuracy of 95.627% on the ISL dataset. The proposed SASLRM is further validated on LSA64, WLASL, and Cambridge Hand Gesture Recognition datasets where, the proposed model reached an accuracy of 97.84%, 98.86%, and 98.22%, respectively. The results indicate the effectiveness of the proposed SLRS in comparison with the existing SLRS.