Sign Recognition Research Articles

Quantifying inconsistencies in the Hamburg Sign Language Notation System

The advent of machine learning (ML) has significantly advanced the recognition and translation of sign languages, bridging communication gaps for hearing-impaired communities. At the heart of these technologies is data labeling, crucial for training ML algorithms on a huge amount of consistently labeled data to achieve models that generalize well. The adoption of language-agnostic annotations is essential to connect different sign languages, as single-language databases often provide limited lexicon examples, insufficient for training robust ML algorithms. This study critically examines the Hamburg Sign Language Notation System (HamNoSys), which describes the signer’s initial position and body movements, in contrary to the meanings of glosses. Despite HamNoSys’s utility in standardizing transcriptions across various sign languages, our investigation uncovers inconsistencies within HamNoSys that may negatively impact the development of accurate and reliable ML models. By analyzing HamNoSys labels across five sign languages, we identified a lack of standardized annotation procedures and the complexities within HamNoSys that introduce biases and errors. Our findings underscore the urgent need for unified, standardized data annotation guidelines to enhance the accuracy and efficiency of sign language recognition technologies. This research highlights the importance of addressing annotation challenges and advocates for a comprehensive, diversified database to improve the generalization of ML models.

GRTR: Gradient Rebalanced Traffic Sign Recognition for Autonomous Vehicles

Traffic sign recognition is a crucial aspect of autonomous vehicle research, and deep learning techniques have significantly contributed to its progress. Nevertheless, the distribution of traffic sign information in natural complex road conditions is long-tailed, and traffic sign identification in complex road conditions has become a significant barrier to autonomous vehicle applications. The imbalanced distribution of information on the dataset migrates to the feature space during training, resulting in imbalanced classifier prediction. In this paper, we propose the gradient rebalanced traffic sign recognition (GRTR) method to address this problem for the first time. GRTR first evaluates the prediction and classification bias of the classifier using the fitted deviation between the model’s output probability and the ground-truth distributions. Then, GRTR dynamically adjusts the correction and compensation factors following the classifier’s prediction and classification biases. GRTR rebalances the positive and negative sample gradients for each category based on the synergistic effect of the correction and compensation factors to prevent the transfer of distribution imbalance and to significantly enhance the performance of the traffic sign classifier under difficult road conditions. Experimental results demonstrate that our GRTR achieves state-of-the-art performance on long-tailed traffic sign and multilabel datasets. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Most traffic sign recognition algorithms are still designed based on the assumption of a balanced distribution of traffic signs in the dataset. Real-world autonomous vehicles require traffic sign recognition on datasets with severely imbalanced distributions. This paper proposes a general approach to solving the long-tailed traffic sign recognition problem.

TRAFFIC SIGN RECOGNITION IN CHALLENGING WEATHER CONDITIONS USING CONVOLUTIONAL NEURAL NETWORKS

The ability of autonomous driving systems to recognize traffic signs in a variety of environmental conditions is crucial to their reliability. This study uses convolutional neural networks (CNNs) to provide a novel method for improving the accuracy of traffic sign recognition systems under challenging weather situations. The research focuses on developing a CNN-based model that is trained using the augmented German Traffic Sign Recognition Benchmark (GTSRB) dataset, which contains over 50,000 labeled images of road signs across 43 categories. To attempt to overcome the limitations of traditional CNN models under adverse environmental conditions, this work presents adaptive feature extraction layers that are especially made to reduce visibility problems brought on by rain, fog, and snow. By taking a comprehensive approach, the model uses advanced data augmentation methods to simulate different weather scenarios, greatly increasing the diversity of the training dataset. Through an analysis of theoretical and practical aspects, the study demonstrates how CNNs enhance the accuracy and efficiency of road sign detection systems in a different weather condition. Research evaluates the model's effectiveness using metrics such as precision, recall, and F1-score, proving its capability to reduce false positives and accurately detect relevant road sign instances. Additionally, the paper highlights the importance of careful dataset preparation and augmentation, model optimization, and training enhancements to improve the performance of road sign detection systems. The results of this research offer benefits for intelligent transport systems, autonomous driving, and road safety, pointing towards further advancements in precise and dependable road sign recognition technology.

Sign Language Recognition

Abstract: Communication through signs has consistently been a significant way for communication among hearing and speech impaired humans, generally called deaf and dumb. It is the only mode of communicating for such individuals to pass on their messages to other human beings, and hence other humans need to comprehend their language. In this project, a sign language detection or recognition web framework is proposed with the help of image processing. This application would help in recognizing Sign Language. The dataset used is the Sign Language dataset. This application could be used in schools or any place, which would make the communication process easier between the impaired and non-impaired people. The proposed method can be used for the ease of recognition of sign language. The method used is Deep Learning for image recognition and the data is trained using Random Forest. Using this method, we would recognize the gesture and predict which sign is shown

Sign Language Recognition (SLR)

Abstract— Sign Language is mainly used by deaf (hard hearing) and dumb people to exchange information between their own community and with other people. It is a language where people use their hand gestures to communicate as they can’t speak or hear. Sign Language Recognition (SLR) deals with recognizing the hand gestures acquisition and continues till text or speech is generated for corresponding hand gestures. Here hand gestures for sign language can be classified as static and dynamic. However, static hand gesture recognition is simpler than dynamic hand gesture recognition, but both recognition is important to the human community. We can use Deep Learning Computer Vision to recognize the hand gestures by building Deep Neural Network architectures (Convolution Neural Network Architectures. Index Terms – HTML, CSS, Python, Pip, Mediapip, opencv, NumPy

SignBridge: Bridging Communication Gaps with a Video Calling App for Inclusive Conversations – Integrating Sign-Language Recognition for the Deaf and Mute

SignBridge: Bridging Communication Gaps with a Video Calling App for Inclusive Conversations – Integrating Sign-Language Recognition for the Deaf and Mute

Perturbation Augmentation for Adversarial Training with Diverse Attacks

Adversarial Training (AT) aims to alleviate the vulnerability of deep neural networks to adversarial perturbations. However, the AT techniques struggle to maintain the performance on natural samples while improving the deep model’s robustness. The absence of perturbation diversity in generated during the adversarial training degrades the generalizability of the robust models, causing overfitting to particular perturbations and a decrease in natural performance. This study proposes an adversarial training framework that augments adversarial directions from a single-step attack to address the trade-off between robustness and generalization. Inspired by feature scattering adversarial training, the proposed framework computes a principal adversarial direction with a single-step attack that finds a perturbation disrupting the inter-sample relationships in the mini-batch during adversarial training. The principal direction obtained at each iteration is augmented by sampling new adversarial directions within a region spanning 45 degrees from the principal adversarial direction. The proposed adversarial training approach does not require extra backpropagation steps in adversarial direction augmentation. Therefore, generalization of the robust model is improved without posing an additional burden on the feature scattering adversarial training. Experiments on CIFAR-10, CIFAR-100, SVHN, Tiny-ImageNet, and The German Traffic Sign Recognition Benchmark consistently improve the accuracy on adversarial with an almost pristine natural performance.

Tinysign: sign language recognition in low resolution settings

Tinysign: sign language recognition in low resolution settings

Unveiling Interpretability: Analyzing Transfer Learning in Deep Learning Models for Traffic Sign Recognition

Unveiling Interpretability: Analyzing Transfer Learning in Deep Learning Models for Traffic Sign Recognition

Silent no more: a comprehensive review of artificial intelligence, deep learning, and machine learning in facilitating deaf and mute communication

People who often communicate via sign language are essential to our society and significantly contribute. They struggle with communication mostly because other people, who often do not understand sign language, cannot interact with them. It is necessary to develop a dependable system for automatic sign language recognition. This paper aims to provide a comprehensive review of the advancements in artificial intelligence (AI), deep learning (DL), and machine learning (ML) technologies that have been used to facilitate communication for individuals who are deaf and mute (D–M). This study explores various applications of these technologies, including sign language interpretation, speech recognition, and text-to-speech synthesis. By examining the current state of research and development in AI, ML, and DL for the D–M field, the survey sheds light on the potential and challenges faced in utilizing AI, deep learning, and ML to bridge the communication gap for the D–M community. The findings of this survey will contribute to a greater understanding of the potential impact of these technologies in improving access to communication for individuals who are D–M, thereby aiding in the development of more inclusive and accessible solutions.

Design of a solution for the development of an application that will interpret the recognition of Mexican sign language through Convolutional Networks

Design of a solution for the development of an application that will interpret the recognition of Mexican sign language through Convolutional Networks

A rare case of invasive mucormycosis with thrombosis of cerebral vasculature in diabetic ketoacidosis

Mucormycosis is a rare fungal infection. It is an aggressive angio invasive infection hence if left untreated is always fatal. This necrotising infection commonly involves the nose, paranasal sinuses, orbits and brain. Uncontrolled diabetes, immunosuppression, transplant recipients, COVID-19 infection are the commonest risk factors. Most common form is rhino orbital cerebral involvement. Antifungal agents along with surgical debridement is the mainstay of treatment. We are reporting such a case of invasive mucormycosis in a young diabetic male to stress the importance of clinical examination and early recognition of clinical signs of this rare invasive infection and aggressive management.

MambaTSR: You only need 90k parameters for traffic sign recognition

Traffic Sign Recognition (TSR) has made significant progress in recent years, and both convolution neural network (CNN)-based and transformer-based models have been widely explored. In addition, combining CNN and transformer can effectively utilize both local and global information for judgment. However, this approach is still affected by the secondary complexity of transformer and cannot maximize the performance. Recently, a state space model (SSM)-based architecture called Mamba has been proposed, which excels in long-range modelling while maintaining linear complexity. When we directly use the Mamba architecture for TSR, it performs poorly because the local features cannot be fully utilized, which are crucial for recognizing traffic sign details. In this paper, we explore the potential of this SSM-based model in TSR from both efficiency and effectiveness perspectives, and we customize a MambaTSR architecture with ∼90k parameters and ∼1.4 ms processing time. Specifically, we use patch embedding and a four-stage encoder at the macro level, while at the micro level we employ three-stream adaptive mining embedding (TAME) to obtain local information and four efficient visual state space (EVSS) blocks to explore global associations. Experiments on German, China, and India datasets show that our method achieves optimal performance and reduces parameters by ∼89 % and processing time by ∼58 % compared to the state-of-the-art method. The code can be accessed at https://github.com/1024AILab/MambaTSR.

Hand Gesture Recognition System Based on Indian Sign Language Using SVM and CNN

Hand gestures (HGs) have been extensively acknowledged as a communication standard for the deaf and dumb school system in the Indian sign language (ISL). We suggested an automated Indian sign language recognition (ISLR) method for English in this study. Initially, the hand region is segmented using the Grasshopper optimization algorithm (GOA) based on the skin color model method. The segmentation’s efficacy is further assessed using three techniques: GOA-based skin color detection algorithm (SCDA), particle swarm optimization-based SCDA (PSO-SCDA), and artificial bee colony-based SCDA (ABC-SCDA). Finally, a database for the retrieved motions representing distinct English alphabets is created. For gesture recognition, the system is trained, and a template-based matching approach is used. The classification technique employs the support vector machine (SVM) and the convolution neural network (CNN). Our suggested recognition method achieved the highest accuracy of 97.85% with GOA-SCDA, compared to 89.29% and 93.96% with PSO-SCDA and ABC-SCDA, respectively. Furthermore, CNN outperformed SVM in classification, with a high accuracy of 99.2% and precision of 81.8%.

Video-Based Sign Language Recognition via ResNet and LSTM Network.

Sign language recognition technology can help people with hearing impairments to communicate with non-hearing-impaired people. At present, with the rapid development of society, deep learning also provides certain technical support for sign language recognition work. In sign language recognition tasks, traditional convolutional neural networks used to extract spatio-temporal features from sign language videos suffer from insufficient feature extraction, resulting in low recognition rates. Nevertheless, a large number of video-based sign language datasets require a significant amount of computing resources for training while ensuring the generalization of the network, which poses a challenge for recognition. In this paper, we present a video-based sign language recognition method based on Residual Network (ResNet) and Long Short-Term Memory (LSTM). As the number of network layers increases, the ResNet network can effectively solve the granularity explosion problem and obtain better time series features. We use the ResNet convolutional network as the backbone model. LSTM utilizes the concept of gates to control unit states and update the output feature values of sequences. ResNet extracts the sign language features. Then, the learned feature space is used as the input of the LSTM network to obtain long sequence features. It can effectively extract the spatio-temporal features in sign language videos and improve the recognition rate of sign language actions. An extensive experimental evaluation demonstrates the effectiveness and superior performance of the proposed method, with an accuracy of 85.26%, F1-score of 84.98%, and precision of 87.77% on Argentine Sign Language (LSA64).

I saw the “female prostate”

This article delves into the diagnostic implications of the female prostate sign, a distinctive radiological sign observed in magnetic resonance imaging of female patients with substantial urethral diverticula. We discuss the association of this sign with urethral diverticula, emphasizing its mimetic resemblance to prostatic hypertrophy observed in older males. Through a comprehensive review of clinical presentations, diagnostic imaging advancements, and treatment modalities, our article underscores the significance of magnetic resonance imaging as a superior diagnostic tool. Our findings support the enhanced recognition and understanding of the female prostate sign among healthcare professionals, facilitating accurate diagnoses and informed management of urethral diverticula.

Empowering Communication: A Deep Learning Framework for Arabic Sign Language Recognition with an Attention Mechanism

This article emphasises the urgent need for appropriate communication tools for communities of people who are deaf or hard-of-hearing, with a specific emphasis on Arabic Sign Language (ArSL). In this study, we use long short-term memory (LSTM) models in conjunction with MediaPipe to reduce the barriers to effective communication and social integration for deaf communities. The model design incorporates LSTM units and an attention mechanism to handle the input sequences of extracted keypoints from recorded gestures. The attention layer selectively directs its focus toward relevant segments of the input sequence, whereas the LSTM layer handles temporal relationships and encodes the sequential data. A comprehensive dataset comprised of fifty frequently used words and numbers in ArSL was collected for developing the recognition model. This dataset comprises many instances of gestures recorded by five volunteers. The results of the experiment support the effectiveness of the proposed approach, as the model achieved accuracies of more than 85% (individual volunteers) and 83% (combined data). The high level of precision emphasises the potential of artificial intelligence-powered translation software to improve effective communication for people with hearing impairments and to enable them to interact with the larger community more easily.

A lightweight network architecture for traffic sign recognition based on enhanced LeNet-5 network.

As an important part of the unmanned driving system, the detection and recognition of traffic sign need to have the characteristics of excellent recognition accuracy, fast execution speed and easy deployment. Researchers have applied the techniques of machine learning, deep learning and image processing to traffic sign recognition successfully. Considering the hardware conditions of the terminal equipment in the unmanned driving system, in this research work, the goal was to achieve a convolutional neural network (CNN) architecture that is lightweight and easily implemented for an embedded application and with excellent recognition accuracy and execution speed. As a classical CNN architecture, LeNet-5 network model was chosen to be improved, including image preprocessing, improving spatial pool convolutional neural network, optimizing neurons, optimizing activation function, etc. The test experiment of the improved network architecture was carried out on German Traffic Sign Recognition Benchmark (GTSRB) database. The experimental results show that the improved network architecture can obtain higher recognition accuracy in a short interference time, and the algorithm loss is significantly reduced with the progress of training. At the same time, compared with other lightweight network models, this network architecture gives a good recognition result, with a recognition accuracy of 97.53%. The network structure is simple, the algorithm complexity is low, and it is suitable for all kinds of terminal equipment, which can have a wider application in unmanned driving system.

A neural-network based web application on real-time recognition of Pakistani sign language

Communication between deaf and non-deaf individuals is affected by a language barrier, as the hearing-impaired rely on sign language to communicate. This two-way process requires fluency in language to be effective. In Pakistan, there are approximately 10 million hearing-impaired citizens at present, and a lack of assistive technology hinders this process. This led us to the objective of our study: to develop a web application that allows real-time recognition of Pakistani Sign Language (PSL) letters. The application backend was developed using Python, and the frontend was developed with Cascading Style Sheets (CSS) and Bootstrap libraries. The dataset consisted of static and dynamic gestures of PSL and the study employed deep learning techniques using the Mediapipe library to model two recognition systems. The offline testing accuracy was 2% and 4% higher than previous significant works on static and dynamic PSL recognition respectively. Real-time testing of both models achieved accurate prediction of all 39 letters.

User Interface Design Analysis of Android-Based Sign Language Recognition Learning Media at Special Schools

This research aims to conduct an in-depth analysis of the user interface (UI) design of Android-based sign language recognition learning media used in special schools. In the context of educational inclusion, sign language learning plays an important role in facilitating effective communication for individuals with hearing loss. The use of mobile technology, such as Android applications, has offered an attractive solution to enhance sign language learning by providing flexible and affordable access. However, the UI design of sign language learning apps does not always adequately consider the special needs of users, especially students with hearing impairment. Therefore, this research aims to analyze the UI design of existing Android-based sign language learning apps, focusing on accessibility, intuitiveness, and effectiveness in delivering learning content. This research uses the Research and Development (R&amp;D) method with the Waterfall model approach. This research method involves analyzing various aspects of UI design, including layout, navigation, use of symbols, and interactivity. Data was collected through direct observation of student use of the application and interviews with educators in special schools. Data analysis was conducted using qualitative and quantitative approaches to identify strengths and weaknesses in the app's UI design. The results showed that of the four instruments used in analyzing the UI design of this "AkuBisa" learning media application, using a Likert scale and a Guttman scale, the following results were obtained: needs analysis instrument with an average value of 3.45; media development storyboard instrument with an average value of 3.37; media display design instrument with an average value of 3.31; and Blackbox testing instrument with a percentage level of functionality reaching 100%. The results of testing and assessment of the four instruments show that the user interface design of android-based sign language recognition learning media is declared very valid in fulfilling the elements of accessibility, intuitiveness, and effectiveness as a learning media application and is very feasible to use as an alternative learning media for sign language recognition in special schools.