Sign Recognition Research Articles

Interactive learning of natural sign language with radar

AbstractOver the past decade, there have been great advancements in radio frequency sensor technology for human–computer interaction applications, such as gesture recognition, and human activity recognition more broadly. While there is a significant amount of study on these topics, in most cases, experimental data are acquired in controlled settings by directing participants what motion to articulate. However, especially for communicative motions, such as sign language, such directed data sets do not accurately capture natural, in situ articulations. This results in a difference in the distribution of directed American Sign Language (ASL) versus natural ASL, which severely degrades natural sign language recognition in real‐world scenarios. To overcome these challenges and acquire more representative data for training deep models, the authors develop an interactive gaming environment, ChessSIGN, which records video and radar data of participants as they play the game without any external direction. The authors investigate various ways of generating synthetic samples from directed ASL data, but show that ultimately such data does not offer much improvement over just initialising using imagery from ImageNet. In contrast, an interactive learning paradigm is proposed by the authors in which model training is shown to improve as more and more natural ASL samples are acquired and augmented via synthetic samples generated from a physics‐aware generative adversarial network. The authors show that the proposed approach enables the recognition of natural ASL in a real‐world setting, achieving an accuracy of 69% for 29 ASL signs—a 60% improvement over conventional training with directed ASL data.

Sign Language Detection Using CNN and LSTM Based Model

In the context of our current work, we have taken a major step in examining and contrasting the effectiveness of several machine learning models for sign language recognition. Convolutional Neural Network (CNN) model building was the focus of our minor project, which resulted in the successful publication of a research paper [6]. Our method was validated by this model, which showed significant accuracy in recognizing sign language. We wanted to build a Long Short-Term Memory (LSTM) model for the same reason as we moved on to our larger project. The aim was not only to construct an additional efficient model but also to conduct a comparative analysis between the LSTM and the CNN model that was previously constructed. This comparison included a thorough assessment of both models based on a number of variables, including accuracy, loss, speed, and others. We can now confidently state that we have created an LSTM model for sign language recognition that is fully operational. In addition, we have compared our CNN model with our LSTM model, assessing both models according to training accuracy, training loss, validation accuracy, validation loss, and training duration. This development opens the door for more research in this area and represents an important turning point in our study. Our research will be strengthened by the solid foundation our findings provide. Keywords: Convolutional Neural Network, Long Short-Term Memory, Sign Language Detection, Training, Validation

Efficient YOLO-Based Deep Learning Model for Arabic Sign Language Recognition

Verbal communication is the dominant form of self-expression and interpersonal communication. Speech is a considerable obstacle for individuals with disabilities, including those who are deaf, hard of hearing, mute, and nonverbal. Sign language is a complex system of gestures and visual signs facilitating individual communication. With the help of artificial intelligence, the hearing and the deaf can communicate more easily. Automatic detection and recognition of sign language is a complex and challenging task in computer vision and machine learning. This paper proposes a novel technique using deep learning to recognize the Arabic Sign Language (ArSL) accurately. The proposed method relies on advanced attention mechanisms and convolutional neural network architecture integrated with a robust You Only Look Once (YOLO) object detection model that improves the detection and recognition rate of the proposed technique. In our proposed method, we integrate the self-attention block, channel attention module, spatial attention module, and cross-convolution module into feature processing for accurate detection. The recognition accuracy of our method is significantly improved, with a higher detection rate of 99%. The methodology outperformed conventional methods, achieving a precision rate of 0.9 and a mean average precision (mAP) of 0.9909 at an intersection over union (IoU) of 0.5. From IoU thresholds of 0.5 to 0.95, the mAP continuously remains high, indicating its effectiveness in accurately identifying signs at different precision levels. The results show the model’s robustness in accurately detecting and classifying complex multiple ArSL signs. The results show the robustness and efficacy of the proposed model.

MLMSign: Multi-lingual multi-modal illumination-invariant sign language recognition

Sign language (SL) serves as a visual communication tool bearing great significance for deaf people to interact with others and facilitate their daily life. Wide varieties of SLs and the lack of interpretation knowledge necessitate developing automated sign language recognition (SLR) systems to attenuate the communication gap between the deaf and hearing communities. Despite numerous advanced static SLR systems, they are not practical and favorable enough for real-life scenarios once assessed simultaneously from different critical aspects: accuracy in dealing with high intra- and slight inter-class variations, robustness, computational complexity, and generalization ability. To this end, we propose a novel multi-lingual multi-modal SLR system, namely MLMSign, by taking full strengths of hand-crafted features and deep learning models to enhance the performance and the robustness of the system against illumination changes while minimizing computational cost. The RGB sign images and 2D visualizations of their hand-crafted features, i.e., Histogram of Oriented Gradients (HOG) features and a∗ channel of L∗a∗b∗ color space, are employed as three input modalities to train a novel Convolutional Neural Network (CNN). The number of layers, filters, kernel size, learning rate, and optimization technique are carefully selected through an extensive parametric study to minimize the computational cost without compromising accuracy. The system’s performance and robustness are significantly enhanced by jointly deploying the models of these three modalities through ensemble learning. The impact of each modality is optimized based on their impact coefficient determined by grid search. In addition to the comprehensive quantitative assessment, the capabilities of our proposed model and the effectiveness of ensembling over three modalities are evaluated qualitatively using the Grad-CAM visualization model. Experimental results on the test data with additional illumination changes verify the high robustness of our system in dealing with overexposed and underexposed lighting conditions. Achieving a high accuracy (>99.33%) on six benchmark datasets (i.e., Massey, Static ASL, NUS II, TSL Fingerspelling, BdSL36v1, and PSL) demonstrates that our system notably outperforms the recent state-of-the-art approaches with a minimum number of parameters and high generalization ability over complex datasets. Its promising performance for four different sign languages makes it a feasible system for multi-lingual applications.

Sign Recognition and Voice Conversion Device for Dumb

This project proposes a comprehensive hardware setup for home automation and environmental monitoring. The system includes components such as a Bluetooth module for remote control, a voice module (APR9600) for audio feedback, an LCD for display purposes, relay modules for controlling electrical devices, bulbs with holders for lighting, a CPU fan for temperature regulation, speakers for audio output, and a 5V power supply for operation. The core functionality of the system revolves around the integration of flux sensors for environmental monitoring. When flux sensors detect specific conditions, predefined audio signals are played through the voice module. These conditions correspond to six distinct signs, facilitating real-time alerts and notifications. Moreover, the system offers remote control capabilities through Bluetooth connectivity, enabling users to manage lighting and fan operations conveniently. Additionally, temperature monitoring is implemented using a DHT11 sensor, triggering the activation of the CPU fan when temperature thresholds are exceeded. Furthermore, an LCD display enhances user interaction by providing visual feedback and status updates. The project utilizes Arduino UNO as the central processing unit and incorporates 12V 1A adapters for powering specific components. Connectors and power cables are utilized for seamless integration and electrical connectivity. Overall, this project presents an integrated solution for home automation and environmental monitoring, offering flexibility, convenience, and enhanced user experience.

TB-Net: Intra- and inter-video correlation learning for continuous sign language recognition

Visual feature extraction is the key to continuous sign language recognition (CSLR). However, current CSLR methods based on single-branch networks only rely on intra-video correlation learning to facilitate visual feature optimization. Obviously, this is not conducive to obtaining more robust visual feature representations. Hence, we pioneered a novel CSLR method via intra- and inter-video correlation learning with a two-branch network, named TB-Net. TB-Net explicitly establishes intra-video correlation between glosses and the most relevant video clips at each branch and then introduces inter-video correlation to enhance visual feature extraction at the branch confluence. Specifically, we introduce a contrastive learning-based inter-video correlation module (IEM), which co-optimizes visual features from both branches by designing inter- and intra-video losses to enhance their generalization. In addition, we propose an intra-video correlation module (IAM) based on a gloss-guided attention feature generator to adaptively build mappings between glosses and video clips, which in turn contributes to the acquisition of preliminary gloss-guided visual features within a single video. Extensive experiments on four public CSLR benchmarks show the superior performance of our method.

Advancements in Vision-Based Sign Language Recognition: A Comprehensive Review

As nearly 70 million individuals suffer from disabling hearing loss, sign language serves as an important means of communication. Unfortunately, the lack of proficiency in sign language among the general population hinders meaningful interactions with those who rely on it. This paper presents an extensive analysis of the cutting-edge methodologies in sign language translation, with the ultimate goal of facilitating effective communication between sign language users and the broader community. In addition to reviewing state-of-the-art approaches, this work also investigates into the challenges and limitations faced by gesture recognition research. Overall, it is expected that the study may provide readers and researchers with a guide for future research and creation in the field of sign language recognition.

Highly Flexible and Superelastic Graphene Nanofibrous Aerogels for Intelligent Sign Language.

Highly flexible and superelastic aerogels at large deformation have become urgent mechanical demands in practical uses, but both properties are usually exclusive. Here a trans-scale porosity design is proposed in graphene nanofibrous aerogels (GNFAs) to break the trade-off between high flexibility and superelasticity. The resulting GNFAs can completely recover after 1000 fatigue cycles at 60% folding strain, and notably maintain excellent structural integrity after 10000 cycles at 90% compressive strain, outperforming most of the reported aerogels. The mechanical robustness is demonstrated to be derived from the trans-scale porous structure, which is composed of hyperbolic micropores and porous nanofibers to enable the large elastic deformation capability. It is further revealed that flexible and superelastic GNFAs exhibit high sensitivity and ultrastability as an electrical sensors to detect tension and flexion deformation. As proof, The GNFA sensor is implemented onto a human finger and achieves the intelligent recognition of sign language with high accuracy by multi-layer artificial neural network. This study proposes a highly flexible and elastic graphene aerogel for wearable human-machine interfaces in sensor technology.

A Hybrid Image Augmentation Technique for User- and Environment-Independent Hand Gesture Recognition Based on Deep Learning

This research stems from the increasing use of hand gestures in various applications, such as sign language recognition to electronic device control. The focus is the importance of accuracy and robustness in recognizing hand gestures to avoid misinterpretation and instruction errors. However, many experiments on hand gesture recognition are conducted in limited laboratory environments, which do not fully reflect the everyday use of hand gestures. Therefore, the importance of an ideal background in hand gesture recognition, involving only the signer without any distracting background, is highlighted. In the real world, the use of hand gestures involves various unique environmental conditions, including differences in background colors, varying lighting conditions, and different hand gesture positions. However, the datasets available to train hand gesture recognition models often lack sufficient variability, thereby hindering the development of accurate and adaptable systems. This research aims to develop a robust hand gesture recognition model capable of operating effectively in diverse real-world environments. By leveraging deep learning-based image augmentation techniques, the study seeks to enhance the accuracy of hand gesture recognition by simulating various environmental conditions. Through data duplication and augmentation methods, including background, geometric, and lighting adjustments, the diversity of the primary dataset is expanded to improve the effectiveness of model training. It is important to note that the utilization of the green screen technique, combined with geometric and lighting augmentation, significantly contributes to the model’s ability to recognize hand gestures accurately. The research results show a significant improvement in accuracy, especially with implementing the proposed green screen technique, underscoring its effectiveness in adapting to various environmental contexts. Additionally, the study emphasizes the importance of adjusting augmentation techniques to the dataset’s characteristics for optimal performance. These findings provide valuable insights into the practical application of hand gesture recognition technology and pave the way for further research in tailoring techniques to datasets with varying complexities and environmental variations.

Real-Time Navigation Roads: Lightweight and Efficient Convolutional Neural Network (LE-CNN) for Arabic Traffic Sign Recognition in Intelligent Transportation Systems (ITS)

Smart cities are now embracing the new frontier of urban living, with advanced technology being used to enhance the quality of life for residents. Many of these cities have developed transportation systems that improve efficiency and sustainability, as well as quality. Integrating cutting-edge transportation technology and data-driven solutions improves safety, reduces environmental impact, optimizes traffic flow during peak hours, and reduces congestion. Intelligent transportation systems consist of many systems, one of which is traffic sign detection. This type of system utilizes many advanced techniques and technologies, such as machine learning and computer vision techniques. A variety of traffic signs, such as yield signs, stop signs, speed limits, and pedestrian crossings, are among those that the traffic sign detection system is trained to recognize and interpret. Ensuring accurate and robust traffic sign recognition is paramount for the safe deployment of self-driving cars in diverse and challenging environments like the Arab world. However, existing methods often face many challenges, such as variability in the appearance of signs, real-time processing, occlusions that can block signs, low-quality images, and others. This paper introduces an advanced Lightweight and Efficient Convolutional Neural Network (LE-CNN) architecture specifically designed for accurate and real-time Arabic traffic sign classification. The proposed LE-CNN architecture leverages the efficacy of depth-wise separable convolutions and channel pruning to achieve significant performance improvements in both speed and accuracy compared to existing models. An extensive evaluation of the LE-CNN on the Arabic traffic sign dataset that was carried out demonstrates an impressive accuracy of 96.5% while maintaining superior performance with a remarkably low inference time of 1.65 s, crucial for real-time applications in self-driving cars. It achieves high accuracy with low false positive and false negative rates, demonstrating its potential for real-world applications like autonomous driving and advanced driver-assistance systems.

Securing AI Models Against Backdoor Attacks: A Novel Approach Using Image Steganography

<p>Artificial Intelligence (AI) has become ubiquitous, transforming numerous domains including traffic sign recognition, defect detection, and healthcare. However, this widespread adoption has brought about significant cybersecurity challenges, particularly in the form of backdoor attacks, which manipulate training datasets to compromise model integrity. While the integration of AI has proven beneficial, there is a lack of comprehensive strategies to protect AI models from these covert attacks, necessitating innovative approaches for securing AI systems. In this study, we demonstrate a novel methodology that integrates image steganography with deep learning techniques, aiming to obscure backdoor triggers and enhance the resilience of AI models against these attacks. We employ a diverse set of AI models and conduct extensive evaluations in a traffic sign recognition scenario, specifically targeting the STOP sign. The results reveal that shallow models are challenged in learning trigger information and are sensitive to trigger settings, while deeper models achieve an impressive 98.03% attack success rate. The image steganography technique used requires minimal data adjustments, making the triggers more challenging to detect than with traditional methods. Our findings underscore the stealth and severity of backdoor attacks, emphasizing the need for advanced security measures in AI and contributing to the broader understanding and development of robust protections against such attacks.</p> <p> </p>

Enhancing Arabic Sign Language Recognition using Deep Learning

The present time, Sign language is very important for people who suffer from hearing loss or who cannot speak. Normal humans tend to disregard the significance of signal language, which is a mere supply of communique to mute and deaf societies. So, this study proposes a developed model for sign Language Recognition for Arabic using the Deep learning Convolutional Neural Network (CNN) algorithm. Then set the algorithm by developing programming on Open-CV, using Python language. The dataset contains 54049 snapshots of Arabic signal language alphabets. The 32 folders were created, and each one of them included 1500 images incorporating hand gestures in at-variance environments. The data set was divided into a training section with a percentage of 70 %, a section for testing with a percentage of 20 %, and a section for validation with a percentage of 10 %. The results show that the suggested model achieved an accuracy rate of 94.8%, and it has proven its effectiveness and success, especially after being tried and tested by several users and obtaining their comments and feedback.

Sign Language Recognition using CNN

Abstract: The sign language Recognition System is a technology that bridges the gap between the deaf people and the hearing world. Sign language is a vital mode of expression for deaf people, yet effective communication remains a challenge. This project aims to develop a robust Sign Language Recognition System capable of accurately translating sign language gestures into text. The system utilizes a deep learning approach, specifically convolutional neural networks (CNNs) to analyze video input of sign language and then generate the corresponding output. The project involves data collection, pre-processing, segmentation, feature extraction, model training, and evaluation. The proposed SLR system has the potential to enhance. Communication and accessibility for deaf individuals, promoting inclusivity and improving their quality of life. The project represents a comprehensive exploration of both technical and ethical aspects in the realm of computer vision and deep learning applications.

A unified approach for continuous sign language recognition and translation

A unified approach for continuous sign language recognition and translation

All Country Sign Language Recognition System to Help Deaf and Mute People

With the use of contemporary technology, the "All Country Sign Language Recognition System to Help Deaf and Mute People" project seeks to provide a strong solution that will close communication barriers between the deaf and mute population and the general public. The project aims to use machine learning techniques and computer vision techniques to properly understand and convert sign language motions into generally used languages, with a concentration on Indian sign language. Deaf and mute people will benefit from increased accessibility and inclusion in a variety of spheres of everyday life, including social interactions, work, and education, by virtue of the implementation of this system. The initiative helps to create a more inclusive society in addition to addressing the urgent demand for efficient communication tools for the deaf and mute communities.In this people can interact using different country sign languages like ISL, ASL, BSL, etc.

Research on Road Sign Recognition Method for Autonomous Vehicles Based on Convolutional Neural Networks

With the rapid development of autonomous driving technology, road sign recognition has become an important link to ensure the safety of autonomous vehicles. This paper proposes a road sign recognition method for autonomous vehicles based on Convolutional Neural Networks (CNN), aiming to improve the accuracy and efficiency of road sign recognition by using CNN. Firstly, the background and significance of this issue were introduced, including the development of autonomous vehicles and their demand for road sign recognition technology. The focus was on the application of convolutional neural networks in road sign recognition. This article provides a detailed introduction to the dataset used, the designed convolutional neural network model, and data preprocessing methods. In order to further improve accuracy, methods such as data augmentation and image enhancement were adopted. The research results of this paper indicate that the road sign recognition method for autonomous vehicles based on convolutional neural networks has high accuracy and efficiency, providing important support for the practical application of autonomous driving technology.

Распознавание дорожных знаков российского образца с использованием нейронных сетей

The problem of creating a model for recognizing objects in images and possible ways to solve it is considered using the example of working with Russian road signs according to ISS R 52290-2004. The analysis of methods for constructing predictive models of image recognition and existing solutions in the public domain is carried out. A convolutional neural network is used as the basic model. A road sign recognition model based on the YOLOv7 transfer network has been developed as a result of training on a dataset from the Russian RTSD road sign image database. The metrics for evaluating the quality of the created model are analyzed and described. The created model meets the quality requirements for objective metrics, allows you to make forecasts taking into account specific situations in different weather conditions and at different times of the day for 146 different predefined classes. The characteristic of the class is the number of the sign according to ISS R 52290-2004. The model has a prediction accuracy of 0.847 with a prediction completeness of 0.811. The average average prediction accuracy of the model is 0.884 when tested on 493 images from the test sample. The test sample does not overlap with the training sample, which consists of 1,842 images. The developed model is published in the public domain both for use for scientific purposes and for further further education. This provides an opportunity for researchers in this field to familiarize themselves with a practical example of the implementation of the model, to supplement or improve it if necessary. The method described in this paper will allow researchers in various subject areas to find a solution that allows them to overcome resource constraints when creating a high-performance and high-quality predictive recognition model.

Sign Language Detection using ML Technologies

Sign language recognition is a leap forward for helping deaf-mute people. In sign language, every sign has a meaning assigned to it, so that it becomes easy to understand and interpret by the people. The main primary objective of our project is to bridge the gap between the deaf and dumb people and ordinary people without the need of an intermediary translator. Gesture recognition is a computer based visual technique used to detect and identify the object in an image or video. An application of this gesture recognition technique involves translating the sign language into the American language which can be further understood and interpreted by normal people. Many researchers have proposed their systems for the implementation of the ASL system. This report is a review of some studies related to the same topic. We addressed different approaches that used Convolutional Neural Networks (CNNs), K- Nearest Neighbours algorithm (KNN), Edge detection algorithm, Deep Neural Network (DNN), fuzzy clustering machine learning algorithm, Kernelized correlation filters (KCF) algorithm along with their results and drawbacks. Furthermore, we report on research gaps while summarizing these studies. From the above research papers, we got an average accuracy of 90.24%. To overcome the drawbacks, we propose a system which collects images for deep learning using webcam and OpenCV, with the help of TensorFlow Object Detection and Python that allows you to translate sign language in real time

Multiscale temporal network for continuous sign language recognition

Multiscale temporal network for continuous sign language recognition

Progressive Pulmonary Fibrosis and Interstitial Lung Abnormalities: AJR Expert Panel Narrative Review.

Progressive pulmonary fibrosis (PPF) and interstitial lung abnormalities (ILA) are relatively new concepts in interstitial lung disease (ILD) imaging and clinical management. Recognition of signs of PPF, as well as identification and classification of ILA, are important tasks during chest high-resolution CT interpretation, to optimize management of patients with ILD and those at risk of developing ILD. However, following professional society guidance, the role of imaging surveillance remains unclear in stable patients with ILD, asymptomatic patients with ILA who are at risk of progression, and asymptomatic patients at risk of developing ILD without imaging abnormalities. In this AJR Expert Panel Narrative Review, we summarize the current knowledge regarding PPF and ILA and describe the range of clinical practice with respect to imaging patients with ILD, those with ILA, and those at risk of developing ILD. In addition, we offer suggestions to help guide surveillance imaging in areas with an absence of published guidelines, where such decisions are currently driven primarily by local pulmonologists' preference.