Sign Language Gesture Recognition Research Articles

Combinatorial Bionic Hierarchical Flexible Strain Sensor for Sign Language Recognition with Machine Learning.

Flexible strain sensors have been widely researched in fields such as smart wearables, human health monitoring, and biomedical applications. However, achieving a wide sensing range and high sensitivity of flexible strain sensors simultaneously remains a challenge, limiting their further applications. To address these issues, a cross-scale combinatorial bionic hierarchical design featuring microscale morphology combined with a macroscale base to balance the sensing range and sensitivity is presented. Inspired by the combination of serpentine and butterfly wing structures, this study employs three-dimensional printing, prestretching, and mold transfer processes to construct a combinatorial bionic hierarchical flexible strain sensor (CBH-sensor) with serpentine-shaped inverted-V-groove/wrinkling-cracking structures. The CBH-sensor has a high wide sensing range of 150% and high sensitivity with a gauge factor of up to 2416.67. In addition, it demonstrates the application of the CBH-sensor array in sign language gesture recognition, successfully identifying nine different sign language gestures with an impressive accuracy of 100% with the assistance of machine learning. The CBH-sensor exhibits considerable promise for use in enabling unobstructed communication between individuals who use sign language and those who do not. Furthermore, it has wide-ranging possibilities for use in the field of gesture-driven interactions in human-computer interfaces.

Active convolutional neural networks sign language (ActiveCNN-SL) framework: a paradigm shift in deaf-mute communication

Real-time speech-to-text and text-to-speech technologies have significantly influenced the accessibility of communication for individuals who are deaf or mute. This research aims to assess the efficacy of these technologies in facilitating communication between deaf or mute individuals and those who are neither deaf nor mute. A mixed-method approach will incorporate qualitative and quantitative data collection and analysis techniques. The study will involve participants from deaf or mute and non-deaf or non-mute communities. The research will scrutinize the precision and efficiency of communication using these technologies and evaluate user experience and satisfaction. Furthermore, the study intends to pinpoint potential obstacles and limitations of these technologies and offer suggestions for enhancing their effectiveness in fostering inclusivity. The study proposes an active learning framework for sign language gesture recognition, termed Active Convolutional Neural Networks—Sign Language (ActiveCNN-SL). ActiveCNN-SL aims to minimize the labeled data required for training and augment the accuracy of sign language gesture recognition through iterative human feedback. This proposed framework holds the potential to enhance communication accessibility for deaf and mute individuals and encourage inclusivity across various environments. The proposed framework is trained using two primary datasets: (i) the Sign Language Gesture Images Dataset and (ii) the American Sign Language Letters (ASL)—v1. The framework employs Resnet50 and YoloV.8 to train the datasets. It has demonstrated high performance in terms of precision and accuracy. The ResNet model achieved a remarkable accuracy rate of 99.98% during training, and it also exhibited a validation accuracy of 100%, surpassing the baseline CNN and RNN models. The YOLOv8 model outperformed previous methods on the ASL alphabet dataset, achieving an overall mean average accuracy for all classes of 97.8%.

Flexible Conformally Bioadhesive MXene Hydrogel Electronics for Machine Learning-Facilitated Human-Interactive Sensing.

Wearable epidermic electronics assembled from conductive hydrogels are attracting various research attention for their seamless integration with human body for conformally real-time health monitoring, clinical diagnostics and medical treatment, and human-interactive sensing. Nevertheless, it remains a tremendous challenge to simultaneously achieve conformally bioadhesive epidermic electronics with remarkable self-adhesiveness, reliable ultraviolet (UV) protection ability, and admirable sensing performance for high-fidelity epidermal electrophysiological signals monitoring, along with timely photothermal therapeutic performances after medical diagnostic sensing, as well as efficient antibacterial activity and reliable hemostatic effect for potential medical therapy. Herein, a conformally bioadhesive hydrogel-based epidermic sensor, featuring superior self-adhesiveness and excellent UV-protection performance, is developed by dexterously assembling conducting MXene nanosheets network with biological hydrogel polymer network for conformally stably attaching onto human skin for high-quality recording of various epidermal electrophysiological signals with high signal-to-noise ratios (SNR) and low interfacial impedance for intelligent medical diagnosis and smart human-machine interface. Moreover, a smart sign language gesture recognition platform based on collected electromyogram (EMG) signals is designed for hassle-free communication with hearing-impaired people with the help of advanced machine learning algorithms. Meanwhile, the bioadhesive MXene hydrogel possesses reliable antibacterial capability, excellent biocompatibility, and effective hemostasis properties for promising bacterial-infected wound bleeding.

Frequency and Spatial Domain-Based Approaches for Recognition of Indian Sign Language Gestures

Objectives: The objective of this paper is to introduce and demonstrate an innovative approach for the recognition of Indian sign language gestures, with a focus on bridging communication gap between the deaf and hearing communities. The goal is to contribute to the development of effective tools and technologies that facilitate seamless communication between individuals using sign language and the people with no knowledge about sign language. Methods: The methodology consists of three key steps. First, data pre-processing involves resizing and contours extraction. Next, feature extraction employs Fourier descriptors for frequency domain analysis and gray-level-co-occurrence matrix for spatial domain analysis. Finally, various machine learning models including SVM, Random Forest, Logistic Regression, K-Nearest Neighbor and Naive Bayes are trained on a standard dataset. Findings: In our controlled experimental setup, we applied a diverse set of machine learning classifiers to evaluate the proposed approach for gesture recognition. Among the classifiers tested, K-Nearest Neighbors demonstrated the highest accuracy, achieving 99.82%. To validate the robustness of our approach, we employed k-fold cross-validation with 5 folds. Novelty: This study presents an innovative method for sign language recognition by employing a dual-domain fusion strategy that prominently emphasizes the frequency domain. Through the integration of Fourier descriptors, the research conducts a detailed frequency domain analysis to characterize the contour shapes of sign language gestures. The synergy with gray-level co-occurrence matrix texture features in the spatial domain analysis, contributes to the creation of a comprehensive feature vector. The proposed approach ensures a thorough exploration of gesture features, there by advancing the precision and efficacy of sign language recognition. Keywords: Indian Sign Language (ISL), Sign Language Recognition (SLR), Frequency domain, Spatial domain, Fourier descriptors, Gray level co­occurrence matrix (GLCM), K­ Fold

Enhancing Accessibility with LSTM-Based Sign Language Detection

Sign language serves as a vital means of communication for the deaf and hard of hearing community. However, identifying sign language poses a significant challenge due to its complexity and the lack of a standardized global framework. Recent advances in machine learning, particularly Long Short-Term Memory (LSTM) algorithms, offer promise in the field of sign language gesture recognition. This research introduces an innovative method that leverages LSTM, a type of recurrent neural network designed for processing sequential input. Our goal is to create a highly accurate system capable of anticipating and reproducing sign language motions with precision. LSTM's unique capabilities enhance the recognition of complex gestures by capturing the temporal relationships and fine details inherent in sign language. The results of this study demonstrate that LSTM-based approaches outperform existing state-of-the-art techniques, highlighting the effectiveness of LSTM in sign language recognition and their potential to facilitate communication between the deaf and hearing communities.

The development of a wearable device for Sign Language Gesture Recognition (SLGR) by using Conceptual Design Approach (CDA)

In recent years, there has been a significant focus on researching and developing Sign Language Gesture Recognition (SLGR) for people with hearing and speaking impairments. This is especially important in Indonesia, where there are approximately two million people with these disabilities. However, current research and developments on SLGR devices are specifically designed to understand only particular sign language systems, such as Korean Standard Sign Language (KSDSL) and American Sign Language (ASL), each of which has its unique gestures and models. As a result, no device has been developed to recognize the gestures of the Indonesian sign language system, known as Sistem Isyarat Bahasa Indonesia (SIBI). Therefore, this study aims to develop an SLGR device that can recognize and translate SIBI gestures into output images via text and speech. The development of the SLGR device in this study is conducted by using the Conceptual Design Approach (CDA) methodology. Where in this case, previous research on SLGR devices is first observed as a benchmark for comparison. Furthermore, the benchmark is used as the basis for the function, sub-function, and specification of the proposed design. Based on these stages of benchmarking and concept development, the study concluded that the final design of the proposed SLGR device is constructed by using 5 flex and Gyroscope sensors connected wirelessly to the Raspberry microcontroller. The device is equipped with a voice system and LCR RPi as the output system for translation. Based on the combination of these sensors, the device is able to identify any particular gestures that correspond to words and phrases in SIBI and translate them into speech via the designated speaker and text display on the LCD screen. To fully understand the performance of the device, experimental tests are conducted by analyzing the input of 26 alphabets in the SIBI system. As a result, the device demonstrated an average of 92% accuracy to convert sign language into voice and text, which demonstrates the usefulness of the proposed device

Sign Language Gesture Recognition and Classification Based on Event Camera with Spiking Neural Networks

Sign language recognition has been utilized in human–machine interactions, improving the lives of people with speech impairments or who rely on nonverbal instructions. Thanks to its higher temporal resolution, less visual redundancy information and lower energy consumption, the use of an event camera with a new dynamic vision sensor (DVS) shows promise with regard to sign language recognition with robot perception and intelligent control. Although previous work has focused on event camera-based, simple gesture datasets, such as DVS128Gesture, event camera gesture datasets inspired by sign language are critical, which poses a great impediment to the development of event camera-based sign language recognition. An effective method to extract spatio-temporal features from event data is significantly desired. Firstly, the event-based sign language gesture datasets are proposed and the data have two sources: traditional sign language videos to event stream (DVS_Sign_v2e) and DAVIS346 (DVS_Sign). In the present dataset, data are divided into five classification, verbs, quantifiers, position, things and people, adapting to actual scenarios where robots provide instruction or assistance. Sign language classification is demonstrated in spike neuron networks with a spatio-temporal back-propagation training method, leading to the best recognition accuracy of 77%. This work paves the way for the combination of event camera-based sign language gesture recognition and robotic perception for the future intelligent systems.

미디어파이프와 장단기기억을 이용한 수화 동작인식 앱 개발

미디어파이프와 장단기기억을 이용한 수화 동작인식 앱 개발

ShuffleNetv2-YOLOv3: a real-time recognition method of static sign language based on a lightweight network

To better meet the communication needs of hearing impaired people and the public, it is of great significance to recognize sign language more quickly and accurately on embedded platforms and mobile terminals. YOLOv3, raised by Joseph Redmon and Ali Farhadi in 2018, achieved a great improvement in detection speed with considerable accuracy by optimizing Yolo. However, YOLOv3 is still too bloated to use on mobile terminals. A static sign language recognition method based on the ShuffleNetv2-YOLOv3 lightweight model was proposed. The ShuffleNetv2-YOLOv3 lightweight model makes the network lightweight by using ShuffleNetv2 as the backbone network of YOLOv3. The lightweight network improved the recognition speed steeply. Combing with the CIoU loss function, the ShuffleNetv2-YOLOv3 keeps the recognition accuracy while improving the recognition speed. Recognition effectiveness of the self-made sign language images and public database by the ShuffleNetv2-YOLOv3 lightweight model was evaluated by F1 score and mAP value. The performance of the ShuffleNetv2-YOLOv3 model was compared with that of the YOLOv3-tiny, SSD, Faster-RCNN, and YOLOv4-tiny model, respectively. The experimental results show that the proposed ShuffleNetv2-YOLOv3 model achieved a good balance between the accuracy and speed of the gesture detection under the premise of model lightweight. The F1 score and mAP value of the ShuffleNetv2-YOLOv3 model were 99.1% and 98.4%, respectively. The gesture detection speed on the GPU reaches 54 frames per second, which is better than other models. The mobile terminal application of the proposed lightweight model was also evaluated. The minimal inference speed of single frame images on the CPU and GPU is 0.14 and 0.025 s per image, respectively. It is only 1/6.5 and 1/8.5 of the running speed of the original YOLOv3 model. The ShuffleNetv2-YOLOv3 lightweight model is conducive to quick, real time, and similar static sign language gesture recognition, laying a good foundation for real-time gesture recognition in the embedded platforms and mobile terminals.

Machine learning approaches to intelligent sign language recognition and classification

Sign language is a great visual communication technique for those who have auditory or speech impairments. The deaf and dumb people have long relied on sign language recognition (SLR) to communicate and integrate into society. This research uses Indian Sign Language to identify elementary sign-language gestures in images/videos and compares machine language methods. Images are pre-processed and feature-extracted to improve the performance of deployed models. The goal is to create a system that uses an efficient classifier to deliver reliable hand sign-language gesture recognition. For the recognition of the Indian Sign Language dataset for the Sign Language Translation and Recognition ISL-CSLTR database, the accuracy and precision of classification methods are analysed and compared. When compared to the decision tree and KNN models, the Random Forest model had a greater accuracy of 84% and 83% precision. We also got 77% Recall and a 0.7 F Score according to this study. The tool used for evaluating our work in python.

Cross-Scene Sign Language Gesture Recognition Based on Frequency-Modulated Continuous Wave Radar

This paper uses millimeter-wave radar to recognize gestures in four different scene domains. The four scene domains are the experimental environment, the experimental location, the experimental direction, and the experimental personnel. The experiments are carried out in four scene domains, using part of the data of a scene domain as the training set for training. The remaining data is used as a validation set to validate the training results. Furthermore, the gesture recognition results of known scenes can be extended to unknown stages after obtaining the original gesture data in different scene domains. Then, three kinds of hand gesture features independent of the scene domain are extracted: range-time spectrum, range-doppler spectrum, and range-angle spectrum. Then, they are fused to represent a complete and comprehensive gesture action. Then, the gesture is trained and recognized using the three-dimensional convolutional neural network (CNN) model. Experimental results show that the three-dimensional CNN can fuse different gesture feature sets. The average recognition rate of the fused gesture features in the same scene domain is 87%, and the average recognition rate in the unknown scene domain is 83.1%, which verifies the feasibility of gesture recognition across scene domains.

A reliable and efficient machine learning pipeline for american sign language gesture recognition using EMG sensors

A reliable and efficient machine learning pipeline for american sign language gesture recognition using EMG sensors

Utilizing motion and spatial features for sign language gesture recognition using cascaded CNN and LSTM models

Sign language is a language produced by body parts gestures and facial expressions. The aim of an automatic sign language recognition system is to assign meaning to each sign gesture. Recently, several computer vision systems have been proposed for sign language recognition using a variety of recognition techniques, sign languages, and gesture modalities. However, one of the challenging problems involves image preprocessing, segmentation, extraction and tracking of relevant static and dynamic features related to manual and nonmanual gestures from different images in sequence. In this paper, we studied the efficiency, scalability, and computation time of three cascaded architectures of convolutional neural network (CNN) and long short-term memory (LSTM) for the recognition of dynamic sign language gestures. The spatial features of dynamic signs are captured using CNN and fed into a multilayer stacked LSTM for temporal information learning. To track the motion in video frames, the absolute temporal differences between consecutive frames are computed and fed into the recognition system. Several experiments have been conducted on three benchmarking datasets of two sign languages to evaluate the proposed models. We also compared the proposed models with other techniques. The attained results show that our models capture better spatio-temporal features pertaining to the recognition of various sign language gestures and consistently outperform other techniques with over 99% accuracy.

Air-CSL: Chinese Sign Language Recognition Based on the Commercial WiFi Devices

Artificial intelligence and Internet of Things (IoT) devices are experiencing explosive growth. Currently, the commonly used gesture recognition methods are difficult to deploy and expensive, so this paper uses the Channel State Information (CSI) for Chinese sign language recognition. Aiming at the problems of current gesture recognition methods, such as strong personnel dependence, high computational resource consumption, and low robustness, we proposed a Chinese sign language gesture recognition method named Air-CSL. In this method, the Local Outlier Factor (LOF) removal algorithm and the Discrete Wavelet Transform (DWT) are used to reduce the noise in the data, and the subcarriers that best represent the gesture data are selected by principal component analysis. After denoising, mathematical statistics were extracted from the gesture waveform as the eigenvalues, and the features were fused by the Deep Restricted Boltzmann Machine (DBM). Finally, the result of gesture classification and recognition is obtained by the Gated Recurrent Unit (GRU). In this way, the prediction model realizes as well as the classification of sign language gestures. The results show that the proposed method can effectively recognize Chinese sign language gestures of different people in different environments and has good robustness.

Sign Language Gesture Recognition with Convolutional-Type Features on Ensemble Classifiers and Hybrid Artificial Neural Network

The proposed research deals with constructing a sign gesture recognition system to enable improved interaction between sign and non-sign users. With respect to this goal, five types of features are utilized—hand coordinates, convolutional features, convolutional features with finger angles, convolutional features on hand edges and convolutional features on binary robust invariant scalable keypoints—and trained on ensemble classifiers to accurately predict the label of the sign image provided as input. In addition, a hybrid artificial neural network is also fabricated that takes two of the aforementioned features, namely convolutional features and convolutional features on hand edges to precisely locate the hand region of the sign gesture under consideration in an attempt for classification. Experiments are also performed with convolutional neural networks on those benchmark datasets which are not accurately classified by the previous two methods. Overall, the proposed methodologies are able to handle a diverse variety of images that include labyrinthine backgrounds, user-specific distinctions, minuscule discrepancies between classes and image alterations. As a result, they are able to produce accuracies comparable with state-of-the-art literature.

Deep Leaning Based Static Indian-Gujarati Sign Language Gesture Recognition

Deep Leaning Based Static Indian-Gujarati Sign Language Gesture Recognition

Breaking the Silence: Convolutional Neural Networks for Sign Language Recognition in the Deaf Community

The deaf community faces communication barriers that hinder their interaction with the hearing world. This work aims to bridge the gap by enabling accurate recognition of Arabic sign language gestures. The proposed Convolutional Neural Networks (CNNs) architecture is designed to effectively capture the spatial features inherent in sign language gestures, thereby enhancing recognition accuracy. A distinctive aspect of our work involves the integration of a CNN architecture with a residual design, which effectively captures intricate spatial features inherent in sign language gestures, thereby enhancing recognition precision. The study leverages the ArSL2018 dataset, a comprehensive collection of grayscale sign language images with diverse lighting conditions and backgrounds. Our custom-built CNN model is trained on this dataset, utilizing a specialized learning rate scheduler for improved convergence. The experimental results showcase promising performance, demonstrating the potential of CNNs in sign language recognition. Furthermore, we present visualizations of the model's predictions using t-SNE, revealing the clustering patterns of different sign language gestures.

Superresolution Reconstruction in Automatic Thai Sign Language Feature Extraction Using Adaptive Triangulation Interpolation

Superresolution is an image processing technique for improving image quality and enhancing low-resolution images. This paper presents a novel interpolation method for increasing superresolution reconstruction effectiveness using a triangulation interpolation algorithm for automatic Thai sign language feature extraction. This approach uses three neighboring pixels for estimation. The experiment compared the superresolution reconstruction performance using the triangulation interpolation algorithm and the nearest-neighbor, SRCNN, bilinear, bicubic, GPR, and NEDI methods. The super-resolution reconstruction using the triangulation-improved interpolation technique provided the best PSNR measurements of image quality between the original and superresolution-reconstructed images. The PSNR value of the sign language image was 40.608, improving performance by 13.15%. The Thai sign language gesture recognition using 2D convolutional neural networks showed that the designed model increased the gesture recognition effectiveness with an accuracy of 0.95 and loss of 0.14. Thus, this study provides state-of-the-art superresolution reconstruction for automatic Thai sign language gesture recognition.

ArSL-CNN a convolutional neural network for Arabic sign language gesture recognition

<p class="IJASEITAbtract">Sign language (SL) is a visual language means of communication for people who are Deaf or have hearing impairments. In Arabic-speaking countries, there are many Arabic sign languages (ArSL) and these use the same alphabets. This study proposes ArSL-CNN, a deep learning model that is based on a convolutional neural network (CNN) for translating Arabic SL (ArSL). Experiments were performed using a large ArSL dataset (ArSL2018) that contains 54049 images of 32 sign language gestures, collected from forty participants. The results of the first experiments with the ArSL-CNN model returned a train and test accuracy of 98.80% and 96.59%, respectively. The results also revealed the impact of imbalanced data on model accuracy. For the second set of experiments, various re-sampling methods were applied to the dataset. Results revealed that applying the synthetic minority oversampling technique (SMOTE) improved the overall test accuracy from 96.59% to 97.29%, yielding a statistically signicant improvement in test accuracy (p=0.016, α<0=05). The proposed ArSL-CNN model can be trained on a variety of Arabic sign languages and reduce the communication barriers encountered by Deaf communities in Arabic-speaking countries.</p>

ArSL-CNN: A convolutional neural network for arabic sign language gesture recognition

ArSL-CNN: A convolutional neural network for arabic sign language gesture recognition