American Sign Language Gestures Research Articles

Transfer learning with YOLOV8 for real-time recognition system of American Sign Language Alphabet

Sign language serves as a sophisticated means of communication vital to individuals who are deaf or hard of hearing, relying on hand movements, facial expressions, and body language to convey nuanced meaning. American Sign Language (ASL) exemplifies this linguistic complexity with its distinct grammar and syntax. The advancement of real-time ASL gesture recognition has explored diverse methodologies, including motion sensors and computer vision techniques. This study specifically addresses the recognition of ASL alphabet gestures using computer vision through Mediapipe for hand movement tracking and YOLOv8 for training the deep learning model. The model achieved notable performance metrics: precision of 98%, recall rate of 98%, F1 score of 99%, mean Average Precision (mAP) of 98%, and mAP50-95 of 93%, underscoring its exceptional accuracy and sturdy capabilities.

DeafTech Vision: A Visual Computer's Approach to Accessible Communication through Deep Learning-Driven ASL Analysis

Sign language is commonly used by people with hearing and speech impairments, making it difficult for those without such disabilities to understand. However, sign language is not limited to communication within the deaf community alone. It has been officially recognized in numerous countries and is increasingly being offered as a second language option in educational institutions. In addition, sign language has shown its usefulness in various professional sectors, including interpreting, education, and healthcare, by facilitating communication between people with and without hearing impairments. Advanced technologies, such as computer vision and machine learning algorithms, are used to interpret and translate sign language into spoken or written forms. These technologies aim to promote inclusivity and provide equal opportunities for people with hearing impairments in different domains, such as education, employment, and social interactions. In this paper, we implement a DeafTech Vision (DTV-CNN) architecture based on the convolutional neural network to recognize American Sign Language (ASL) gestures using deep learning techniques. Our main objective is to develop a robust ASL sign classification model to enhance human-computer interaction and assist individuals with hearing impairments. Through extensive evaluation, our model consistently outperformed baseline methods in terms of precision. It achieved an outstanding accuracy rate of 99.87% on the ASL alphabet test dataset and 99.94% on the ASL digit dataset, significantly exceeding previous research, which reported an accuracy of 90.00%. We also illustrated the model's learning trends and convergence points using loss and error graphs. These results highlight the DTV-CNN's effectiveness and capability in distinguishing complex ASL gestures.

Sign Language to Text Conversion using CNN

Sign language is a communication strategy used by those who are unable to hear. So those people who know sign language can communicate with people who are deaf. But a majority of our people don’t know sign language therefore there comes a communication gap between the ones who know sign language and others who don’t know. This project's major purpose is to bridge this gap by developing a systemthat recognizesmultiple sign languages and translates them into text in real-time. We use machine learning technologies to construct this system especially, convolutional neural networks (cnns), which are used to recognize and translate American Sign Language (ASL) into text by capturing it using a webcam. The transformed text is then presented on the screen by which individuals can comprehend and communicate with those who use sign language. The system's performance is evaluated on a dataset of ASL gestures, attaining excellent accuracy and indicating its potential for practical usage in enhancing communication accessibility for the deaf and hard-of-hearing community.

A Comprehensive Study on Relative Distances of Hand Landmarks Approach for American Sign Language Gesture

A Comprehensive Study on Relative Distances of Hand Landmarks Approach for American Sign Language Gesture

EchoGest: Soft Ultrasonic Waveguides based Sensing Skin for Subject-Independent Hand Gesture Recognition.

Gesture recognition is crucial for enhancing human-computer interaction and is particularly pivotal in rehabilitation contexts, aiding individuals recovering from physical impairments and significantly improving their mobility and interactive capabilities. However, current wearable hand gesture recognition approaches are often limited in detection performance, wearability, and generalization. We thus introduce EchoGest, a novel hand gesture recognition system based on soft, stretchable, transparent artificial skin with integrated ultrasonic waveguides. Our presented system is the first to use soft ultrasonic waveguides for hand gesture recognition. Ecoflex™ 00-31 and Ecoflex™ 00-45 Near Clear™ silicone elastomers were employed to fabricate the artificial skin and ultrasonic waveguides, while 0.1 mm diameter silver-plated copper wires connected the transducers in the waveguides to the electrical system. The wires are enclosed within an additional elastomer layer, achieving a sensing skin with a total thickness of around 500 μm. Ten participants wore the EchoGest system and performed static hand gestures from two gesture sets: 8 daily life gestures and 10 American Sign Language (ASL) digits 0-9. Leave-One-Subject-Out Cross-Validation analysis demonstrated accuracies of 91.13% for daily life gestures and 88.5% for ASL gestures. The EchoGest system has significant potential in rehabilitation, particularly for tracking and evaluating hand mobility, which could substantially reduce the workload of therapists in both clinical and home-based settings. Integrating this technology could revolutionize hand gesture recognition applications, from real-time sign language translation to innovative rehabilitation techniques.

Exploiting domain transformation and deep learning for hand gesture recognition using a low-cost dataglove

Hand gesture recognition is one of the most widely explored areas under the human–computer interaction domain. Although various modalities of hand gesture recognition have been explored in the last three decades, in recent years, due to the availability of hardware and deep learning algorithms, hand gesture recognition research has attained renewed momentum. In this paper, we evaluate the effectiveness of a low-cost dataglove for classifying hand gestures in the light of deep learning. We have developed a cost-effective dataglove using five flex sensors, an inertial measurement unit, and a powerful microcontroller for onboard processing and wireless connectivity. We have collected data from 25 subjects for 24 static and 16 dynamic American sign language gestures for validating our system. Moreover, we proposed a novel Spatial Projection Image-based technique for dynamic hand gesture recognition. We also explored a parallel-path neural network architecture for handling multimodal data more effectively. Our method produced an F1-score of 82.19% for static gestures and 97.35% for dynamic gestures from a leave-one-out-cross-validation approach. Overall, this study demonstrates the promising performance of a generalized hand gesture recognition technique in hand gesture recognition. The dataset used in this work has been made publicly available.

A reliable and efficient machine learning pipeline for american sign language gesture recognition using EMG sensors

A reliable and efficient machine learning pipeline for american sign language gesture recognition using EMG sensors

Sign Language Recognition and Converting into Text

Abstract: Sign language is a mode of communication that use a variety of hand movements and actions to convey a message. Deciphering these motions might be a pattern recognition challenge. People use a range of gestures and behaviours to communicate with one another. This study is a system for a human-computer interface that can identify american sign language gestures and produce textual output that reflects the meaning of the gesture. To identify and learn gestures, the proposed system would employ convolutional neural networks and long short term memory networks. This will help to break down the communication gap. Keywords: Data Set, Feature Extraction and Representation, Artificial Neural Networks, Convolutional Neural Networks, Tensor Flow, Keras, OpenCV.

Sensor Fusion of Motion-Based Sign Language Interpretation with Deep Learning.

Sign language was designed to allow hearing-impaired people to interact with others. Nonetheless, knowledge of sign language is uncommon in society, which leads to a communication barrier with the hearing-impaired community. Many studies of sign language recognition utilizing computer vision (CV) have been conducted worldwide to reduce such barriers. However, this approach is restricted by the visual angle and highly affected by environmental factors. In addition, CV usually involves the use of machine learning, which requires collaboration of a team of experts and utilization of high-cost hardware utilities; this increases the application cost in real-world situations. Thus, this study aims to design and implement a smart wearable American Sign Language (ASL) interpretation system using deep learning, which applies sensor fusion that “fuses” six inertial measurement units (IMUs). The IMUs are attached to all fingertips and the back of the hand to recognize sign language gestures; thus, the proposed method is not restricted by the field of view. The study reveals that this model achieves an average recognition rate of 99.81% for dynamic ASL gestures. Moreover, the proposed ASL recognition system can be further integrated with ICT and IoT technology to provide a feasible solution to assist hearing-impaired people in communicating with others and improve their quality of life.

A Wireless Multi-Channel Capacitive Sensor System for Efficient Glove-Based Gesture Recognition With AI at the Edge

This brief presents a wireless smart glove based on multi-channel capacitive pressure sensors that is able to recognize 10 American Sign Language gestures at the edge. In this system, 16 capacitive sensors are fabricated on a glove to capture the hand gestures. The sensor data is captured by a 16-channel CDMA-like capacitance-to-digital converter for training/inference at the edge device. Unlike the conventional approach where the capacitive information is recovered before further signal processing, our proposed system approach takes advantage of the capability of the machine learning (ML) algorithms and directly processes the code-modulated signals without demodulation. As a result, it reduces the input data throughput fed into the ML algorithms by $20\times $ . The on-site ML implementation significantly reduces decision-making latency and lowers the required data throughput for wireless transmission by at least $4\times $ . The highest testing classification accuracy of our system achieved is 99.7%, with a <0.1% difference from the conventional demodulated sensing scheme.

Sign Language Translation using Hand Gesture Detection

This Paper demonstrate a module on a sign to speech (voice) converter for auto Conversion of American sign language (ASL) to English Speech and text. It minimizes the gap of communication between speech impaired and other humans. It could be used to understand a speech impaired person’s thoughts or views which he communicates with others through its ASL gestures but failing to communicate due to a large communication gap between them. It also work as a translator for person who do not understand the sign language and allows the communication in the natural way of speaking. The proposed module is an interactive application module developed using Python and its Advanced Libraries. This module uses inbuilt camera of the system to get the images and perform analysis on those images to predict the meaning of that gesture and provide the output as text on screen and speech through speaker of the system that makes this module very much cost effective. This module recognizes one handed ASL gestures of alphabets (A-Z) with highly consistent, fairly high precision and accuracy.

DACTYL ALPHABET GESTURE RECOGNITION IN A VIDEO SEQUENCE USING MICROSOFT KINECT

Abstract. This paper presents an efficient framework for solving the problem of static gesture recognition based on data obtained from the web cameras and depth sensor Kinect (RGB-D - data). Each gesture given by a pair of images: color image and depth map. The database store gestures by it features description, genereated by frame for each gesture of the alphabet. Recognition algorithm takes as input a video sequence (a sequence of frames) for marking, put in correspondence with each frame sequence gesture from the database, or decide that there is no suitable gesture in the database. First, classification of the frame of the video sequence is done separately without interframe information. Then, a sequence of successful marked frames in equal gesture is grouped into a single static gesture. We propose a method combined segmentation of frame by depth map and RGB-image. The primary segmentation is based on the depth map. It gives information about the position and allows to get hands rough border. Then, based on the color image border is specified and performed analysis of the shape of the hand. Method of continuous skeleton is used to generate features. We propose a method of skeleton terminal branches, which gives the opportunity to determine the position of the fingers and wrist. Classification features for gesture is description of the position of the fingers relative to the wrist. The experiments were carried out with the developed algorithm on the example of the American Sign Language. American Sign Language gesture has several components, including the shape of the hand, its orientation in space and the type of movement. The accuracy of the proposed method is evaluated on the base of collected gestures consisting of 2700 frames.

Gesture Recognition Using Neural Networks Based on HW/SW Cosimulation Platform

Hardware/software (HW/SW) cosimulation integrates software simulation and hardware simulation simultaneously. Usually, HW/SW co-simulation platform is used to ease debugging and verification for very large-scale integration (VLSI) design. To accelerate the computation of the gesture recognition technique, an HW/SW implementation using field programmable gate array (FPGA) technology is presented in this paper. The major contributions of this work are: (1) a novel design of memory controller in the Verilog Hardware Description Language (Verilog HDL) to reduce memory consumption and load on the processor. (2) The testing part of the neural network algorithm is being hardwired to improve the speed and performance. The American Sign Language gesture recognition is chosen to verify the performance of the approach. Several experiments were carried out on four databases of the gestures (alphabet signs A to Z). (3) The major benefit of this design is that it takes only few milliseconds to recognize the hand gesture which makes it computationally more efficient.

Distinguishing the processing of gestures from signs in deaf individuals: An fMRI study

Manual gestures occur on a continuum from co-speech gesticulations to conventionalized emblems to language signs. Our goal in the present study was to understand the neural bases of the processing of gestures along such a continuum. We studied four types of gestures, varying along linguistic and semantic dimensions: linguistic and meaningful American Sign Language (ASL), non-meaningful pseudo-ASL, meaningful emblematic, and nonlinguistic, non-meaningful made-up gestures. Pre-lingually deaf, native signers of ASL participated in the fMRI study and performed two tasks while viewing videos of the gestures: a visuo-spatial (identity) discrimination task and a category discrimination task. We found that the categorization task activated left ventral middle and inferior frontal gyrus, among other regions, to a greater extent compared to the visual discrimination task, supporting the idea of semantic-level processing of the gestures. The reverse contrast resulted in enhanced activity of bilateral intraparietal sulcus, supporting the idea of featural-level processing (analogous to phonological-level processing of speech sounds) of the gestures. Regardless of the task, we found that brain activation patterns for the nonlinguistic, non-meaningful gestures were the most different compared to the ASL gestures. The activation patterns for the emblems were most similar to those of the ASL gestures and those of the pseudo-ASL were most similar to the nonlinguistic, non-meaningful gestures. The fMRI results provide partial support for the conceptualization of different gestures as belonging to a continuum and the variance in the fMRI results was best explained by differences in the processing of gestures along the semantic dimension.

American Sign Language Recognition Using Multi-Dimensional Hidden Markov Models

An American Sign Language (ASL) recognition system developed based on multidimensional Hidden Markov Models (HMM) is presented in this paper. A Cyberglove(superscript TM) sensory glove and a Flock of Birds(superscript ®) motion tracker are used to extract the features of ASL gestures. The data obtained from the strain gages in the glove defines the hand shape while the data from the motion tracker describes the trajectory of hand movement. Our objective is to continuously recognize ASL gestures using these input devices in real time. With the features extracted from the sensory data, we specify multi-dimensional states for ASL signs in the HMM processor. The system gives an average of 95% correct recognition for the 26 alphabets and 36 basic handshapes in the ASL after it has been trained with 8 samples. New gestures can be accommodated in the system with an interactive, learning processor. The developed system forms a sound foundation for continuous recognition of ASL full signs.