American Sign Language Dataset Research Articles

Multi-culture Sign Language Detection and Recognition Using Fine-tuned Convolutional Neural Network

The speech and hearing-impaired community use sign language as the primary means of communication. It is quite challenging for the general population to interpret or learn sign language completely. A sign language recognition system must be designed and developed to address this communication barrier. Most current sign language recognition systems rely on wearable sensors, keeping the recognition system unaffordable for most individuals. Moreover, the existing vision-based sign recognition frameworks do not consider all of the spatial and temporal information required for accurate recognition. A novel vision-based hybrid deep neural net methodology is proposed in this project for recognizing American Sign Language (ASL) and custom sign gestures. The proposed framework aims to establish a single framework for tracking and extracting multi-semantic properties, such as non-manual components and manual co-articulations. Furthermore, spatial feature extraction from the sign gestures is deployed using a Hybrid Deep Neural Network (HDNN) with atrous convolutions. The temporal and sequential feature extraction is carried out by employing attention-based HDNN. In addition, the distinguished abstract feature extraction is done using modified autoencoders. The discriminative feature extraction for differentiating the sign gestures from unwanted transition gestures is done by leveraging the hybrid attention module. The experimentation of the proposed model has been carried out on the novel multi-signer ASL and custom sign language dataset. The proposed sign language recognition framework with hybrid neural nets, specifically using HDNN, yields better results than other state-of-the-art frameworks. Additionally, a detection module is incorporated using Flask web, allowing manual input of images/signs for real-time recognition.

American Sign Language Recognition and Translation Using Perception Neuron Wearable Inertial Motion Capture System.

Sign language is designed as a natural communication method to convey messages among the deaf community. In the study of sign language recognition through wearable sensors, the data sources are limited, and the data acquisition process is complex. This research aims to collect an American sign language dataset with a wearable inertial motion capture system and realize the recognition and end-to-end translation of sign language sentences with deep learning models. In this work, a dataset consisting of 300 commonly used sentences is gathered from 3 volunteers. In the design of the recognition network, the model mainly consists of three layers: convolutional neural network, bi-directional long short-term memory, and connectionist temporal classification. The model achieves accuracy rates of 99.07% in word-level evaluation and 97.34% in sentence-level evaluation. In the design of the translation network, the encoder-decoder structured model is mainly based on long short-term memory with global attention. The word error rate of end-to-end translation is 16.63%. The proposed method has the potential to recognize more sign language sentences with reliable inertial data from the device.

Machine Learning Based Customized Solution for Deafness and Mute People

Individuals with hearing or speech impairments often possess a higher degree of familiarity with sign language due to its widespread usage. Consequently, a noticeable communication divide exists between those with such impairments and the general populace. A pivotal solution in bridging this gap involves employing human sign language interpreters. Regrettably, the scarcity of sign language interpreters worldwide in comparison to the number of individuals with hearing or speech impairments has resulted in certain individuals being unable to bear the cost of a human interpreter each time they engage in conversation. To address this issue, it is imperative to automate communication in a manner that liberates the deaf community from dependence on human translators. This article centers on the development of an application capable of real-time conversion between American Sign Language and text, along with supplementary functionalities aimed at dismantling the communication barriers faced by individuals with hearing or speech challenges when interacting with the broader public. the primary objective of this application is to accurately perceive and interpret the user's sign language. The initial stage involves training the system to recognize and interpret signs utilizing algorithms for object recognition and motion tracking. For this purpose, a convolutional neural network model was trained employing a meticulously curated American Sign Language dataset. Subsequently, the recognized signs are translated into English, forming grammatically sound words. Incorporated within the system are a language translator and a virtual sign keyboard, augmenting its capabilities. Moreover, a text-to-text transformer, structured on an encoder-decoder framework, is harnessed to identify grammatical inaccuracies and generate coherent phrases. For a comprehensive understanding of the process, detailed elaboration will follow in subsequent sections.

ASL champ!: a virtual reality game with deep-learning driven sign recognition

We developed an American Sign Language (ASL) learning platform in a Virtual Reality (VR) environment to facilitate immersive interaction and real-time feedback for ASL learners. We describe the first game to use an interactive teaching style in which users learn from a fluent signing avatar and the first implementation of ASL sign recognition using deep learning within the VR environment. Advanced motion-capture technology powers an expressive ASL teaching avatar within an immersive three-dimensional environment. The teacher demonstrates an ASL sign for an object, prompting the user to copy the sign. Upon the user’s signing, a third-party plugin executes the sign recognition process alongside a deep learning model. Depending on the accuracy of a user’s sign production, the avatar repeats the sign or introduces a new one. We gathered a 3D VR ASL dataset from fifteen diverse participants to power the sign recognition model. The proposed deep learning model’s training, validation, and test accuracy are 90.12%, 89.37%, and 86.66%, respectively. The functional prototype can teach sign language vocabulary and be successfully adapted as an interactive ASL learning platform in VR.

Multi-Modal Multi-Channel American Sign Language Recognition

In this paper, we propose a machine learning-based multi-stream framework to recognize American Sign Language (ASL) manual signs and nonmanual gestures (face and head movements) in real time from RGB-D videos. Our approach is based on 3D Convolutional Neural Networks (3D CNNs) by fusing the multi-modal features including hand gestures, facial expressions, and body poses from multiple channels (RGB, Depth, Motion, and Skeleton joints). To learn the overall temporal dynamics in a video, a proxy video is generated by selecting a subset of frames for each video which are then used to train the proposed 3D CNN model. We collected a new ASL dataset, ASL-100-RGBD, which contains 42 RGB-D videos captured by a Microsoft Kinect V2 camera. Each video consists of 100 ASL manual signs, along with RGB channel, Depth maps, Skeleton joints, Face features, and HD face. The dataset is fully annotated for each semantic region (i.e. the time duration of each sign that the human signer performs). Our proposed method achieves 92.88% accuracy for recognizing 100 ASL sign glosses in our newly collected ASL-100-RGBD dataset. The effectiveness of our framework for recognizing hand gestures from RGB-D videos is further demonstrated on a large-scale dataset, ChaLearn IsoGD, achieving the state-of-the-art results.

Fusion of Attention-Based Convolution Neural Network and HOG Features for Static Sign Language Recognition

The deaf and hearing-impaired community expresses their emotions, communicates with society, and enhances the interaction between humans and computers using sign language gestures. This work presents a strategy for efficient feature extraction that uses a combination of two different methods that are the convolutional block attention module (CBAM)-based convolutional neural network (CNN) and standard handcrafted histogram of oriented gradients (HOG) feature descriptor. The proposed framework aims to enhance accuracy by extracting meaningful features and resolving issues like rotation, similar hand orientation, etc. The HOG feature extraction technique provides a compact feature representation that signifies meaningful information about sign gestures. The CBAM attention module is incorporated into the structure of CNN to enhance feature learning using spatial and channel attention mechanisms. Then, the final feature vector is formed by concatenating these features. This feature vector is provided to the classification layers to predict static sign gestures. The proposed approach is validated on two publicly available static Massey American Sign Language (ASL) and Indian Sign Language (ISL) databases. The model’s performance is evaluated using precision, recall, F1-score, and accuracy. Our proposed methodology achieved 99.22% and 99.79% accuracy for the ASL and ISL datasets. The acquired results signify the efficiency of the feature fusion and attention mechanism. Our network performed better in accuracy compared to the earlier studies.

Constraints on Optimising Encoder-Only Transformers for Modelling Sign Language with Human Pose Estimation Keypoint Data.

Supervised deep learning models can be optimised by applying regularisation techniques to reduce overfitting, which can prove difficult when fine tuning the associated hyperparameters. Not all hyperparameters are equal, and understanding the effect each hyperparameter and regularisation technique has on the performance of a given model is of paramount importance in research. We present the first comprehensive, large-scale ablation study for an encoder-only transformer to model sign language using the improved Word-level American Sign Language dataset (WLASL-alt) and human pose estimation keypoint data, with a view to put constraints on the potential to optimise the task. We measure the impact a range of model parameter regularisation and data augmentation techniques have on sign classification accuracy. We demonstrate that within the quoted uncertainties, other than ℓ2 parameter regularisation, none of the regularisation techniques we employ have an appreciable positive impact on performance, which we find to be in contradiction to results reported by other similar, albeit smaller scale, studies. We also demonstrate that the model architecture is bounded by the small dataset size for this task over finding an appropriate set of model parameter regularisation and common or basic dataset augmentation techniques. Furthermore, using the base model configuration, we report a new maximum top-1 classification accuracy of 84% on 100 signs, thereby improving on the previous benchmark result for this model architecture and dataset.

BP-SRM: A directly training algorithm for spiking neural network constructed by spike response model

Spiking neural networks (SNNs) have attracted widespread attention due to their unique bio-interpretability and low-power properties, but the non-differentiability of discrete spike sequences fired by spiking neurons brings difficulties to the learning of SNNs. Recently, surrogate gradient (SG) and back-propagation through time (BPTT) have provided an excellent idea for training SNNs constructed by leaky integrate-and-fire (LIF) neuron models. The LIF neuron model has been widely used in previous SNNs due to its simplicity and low computational cost, however, this also limits its simulation of biological neuron dynamics, reducing the biological interpretability of SNNs. In this paper, we generalize SG and BPTT to SNNs constructed by spike response model (SRM) and propose the BP-SRM algorithm. Specifically, we address why BPTT succeeded in LIF neurons-based SNNs but failed in SRM neurons-based SNNs in previous researches. Then we establish an iterative form of the SRM neuron model by selecting different state variables. Based on iterable SRM, we get the spatiotemporal dependencies between the state variables of SRM neurons-based SNNs, which allows us to derive the gradient and update weights in SNN by BP-SRM. Then we design temporal channel normalization for BP-SRM and verify the performance of the SNNs on static image dataset, dynamic image dataset and engineering dataset, including Fashion-MNIST, Weather Dataset, N-MNIST, American Sign Language Dataset and bearing fault diagnosis dataset. The experiment results indicate that BP-SRM achieves the state-of-the-art performance of SNNs.

Dataset classification: An efficient feature extraction approach for grammatical facial expression recognition

In this paper, an efficient features extraction using validated statistical approaches is proposed, along with a robust Grammatical Facial Expressions (GFEs) classifier in facial expression recognition systems. Accordingly, a new dataset was collected from 70 participants (33 males and 37 females) ranging in age from 18 to 46. The total number of video clips collected was 765.The features extracted in this study consist of 17 features associated with three categories of non-manual features: facial expression, head movement, and eye-gaze. Automatic recognition of nine classes of grammatical facial expressions in two languages (Arabic and Persian) is performed using a linear Support Vector Machine (SVM) classifier.The proposed system was also validated by testing it on the American Sign Language (ASL) dataset. In comparison to previous works on the ASL dataset, the results showed a higher accuracy rate of 95%.

Modelling Sign Language with Encoder-Only Transformers and Human Pose Estimation Keypoint Data

We present a study on modelling American Sign Language (ASL) with encoder-only transformers and human pose estimation keypoint data. Using an enhanced version of the publicly available Word-level ASL (WLASL) dataset, and a novel normalisation technique based on signer body size, we show the impact model architecture has on accurately classifying sets of 10, 50, 100, and 300 isolated, dynamic signs using two-dimensional keypoint coordinates only. We demonstrate the importance of running and reporting results from repeated experiments to describe and evaluate model performance. We include descriptions of the algorithms used to normalise the data and generate the train, validation, and test data splits. We report top-1, top-5, and top-10 accuracy results, evaluated with two separate model checkpoint metrics based on validation accuracy and loss. We find models with fewer than 100k learnable parameters can achieve high accuracy on reduced vocabulary datasets, paving the way for lightweight consumer hardware to perform tasks that are traditionally resource-intensive, requiring expensive, high-end equipment. We achieve top-1, top-5, and top-10 accuracies of 97%, 100%, and 100%, respectively, on a vocabulary size of 10 signs; 87%, 97%, and 98% on 50 signs; 83%, 96%, and 97% on 100 signs; and 71%, 90%, and 94% on 300 signs, thereby setting a new benchmark for this task.

An Approach for Gesture Recognition Based on a Lightweight Convolutional Neural Network

Gesture recognition, which plays an important role to understand meaningful movements of human bodies, is one of the most effective approaches for humans to interact. Sign language is a fundamental and innate means of communication for hearing-impaired individuals. Though significant progress has been made, the state-of-the-art gesture recognition methods yield week performance for conditions with dynamic gestures in videos. Thus, robust gesture recognition remains a challenging issue because of many barriers of gesture-irrelevant factors. The key to robust gesture recognition is to learn effective and concise spatiotemporal information. Inspired by the great promise of the convolutional neural network (CNN) and its breakthroughs, we introduce an approach for identifying static alphabet gestures in the American Sign Language (ASL). The proposed CNN-based approach has been developed to classify letters of the alphabet from A to Z. It composes of three phases: a preprocessing phase for extracted of the region of interest, a feature extraction, and a classification phase. The performance of proposed gesture recognition approach is evaluated on the common ASL dataset and it achieves 94.83% accuracy, which is good enough to develop a strong translator from gesture-based ASL to spoken language as it is capable to handle a variety of 24 hand gestures.

Two-Stream Mixed Convolutional Neural Network for American Sign Language Recognition.

The Convolutional Neural Network (CNN) has demonstrated excellent performance in image recognition and has brought new opportunities for sign language recognition. However, the features undergo many nonlinear transformations while performing the convolutional operation and the traditional CNN models are insufficient in dealing with the correlation between images. In American Sign Language (ASL) recognition, J and Z with moving gestures bring recognition challenges. This paper proposes a novel Two-Stream Mixed (TSM) method with feature extraction and fusion operation to improve the correlation of feature expression between two time-consecutive images for the dynamic gestures. The proposed TSM-CNN system is composed of preprocessing, the TSM block, and CNN classifiers. Two consecutive images in the dynamic gesture are used as inputs of streams, and resizing, transformation, and augmentation are carried out in the preprocessing stage. The fusion feature map obtained by addition and concatenation in the TSM block is used as inputs of the classifiers. Finally, a classifier classifies images. The TSM-CNN model with the highest performance scores depending on three concatenation methods is selected as the definitive recognition model for ASL recognition. We design 4 CNN models with TSM: TSM-LeNet, TSM-AlexNet, TSM-ResNet18, and TSM-ResNet50. The experimental results show that the CNN models with the TSM are better than models without TSM. The TSM-ResNet50 has the best accuracy of 97.57% for MNIST and ASL datasets and is able to be applied to a RGB image sensing system for hearing-impaired people.

Hypertuned Deep Convolutional Neural Network for Sign Language Recognition.

Sign language plays a pivotal role in the lives of impaired people having speaking and hearing disabilities. They can convey messages using hand gesture movements. American Sign Language (ASL) recognition is challenging due to the increasing intra-class similarity and high complexity. This paper used a deep convolutional neural network for ASL alphabet recognition to overcome ASL recognition challenges. This paper presents an ASL recognition approach using a deep convolutional neural network. The performance of the DeepCNN model improves with the amount of given data; for this purpose, we applied the data augmentation technique to expand the size of training data from existing data artificially. According to the experiments, the proposed DeepCNN model provides consistent results for the ASL dataset. Experiments prove that the DeepCNN gives a better accuracy gain of 19.84%, 8.37%, 16.31%, 17.17%, 5.86%, and 3.26% as compared to various state-of-the-art approaches.

Sign Language Recognition using Deep Learning

Abstract: Millions of people with speech and hearing impairments communicate with sign languages every day. For hearingimpaired people, gesture recognition is a natural way of communicating, much like voice recognition is for most people. In this study, we look at the issue of translating/converting sign language to text and propose a better solution based on machine learning techniques. We want to establish a system that hearing-impaired people may utilise in their everyday lives to promote communication and collaboration between hearing-impaired people and people who aren't trained in American Sign Language (ASL). To develop a deep learning model for the ASL dataset, we'll use a technique called Transfer Learning in combination with Data Augmentation. Keywords: Sign language, machine leaning, Transfer learning, ASL, Inception v3

Gesture Detection using Tensor flow lite Efficient Net Model for Communication and E learning Module for Mute and Deaf

Human communication plays a vital role; without communicating, day-to-day tasks seem difficult to complete. And the world has an almost 5% population that struggles with hearing or speaking disability, which contributes to 430 million people worldwide, and this will grow up to 900million just in the next 25 to 30 years. With the increasing noise pollution, hearing capacity degrades, leading to various hearing problems. The WHO statistics show that 32million kids are acoustically impaired. With disabilities, there are multiple issues these people face, such as lack of learning facilities, job opportunities, communication platforms, etc. These people need a cooperative environment to express, learn at their pace and level of understanding. This paper focuses on developing an application that bridges the gap between these acoustically disabled people and people unknown to their way of communication. The proposed research is an edge device application provides features like a gesture to text, speech to text, e-learning platform, and Alert mechanism. This paper majorly focuses on developing a friendly all in one platform for mute and deaf community for communication, learning and emergency alerts. The research was conducted with two approaches the traditional CNN and Tensorflow lite EfficientNet model to train the ASL (American Sign Language) dataset for the communication platform, where we obtained accuracy of 98.91% and 98.82% respectively. To overcome the computational barriers of traditional CNN approach, Tensorf low lite Efficient Net model was brought into the picture. The proposed methodology would help build a platform for the deaf and mute community to express themselves better and gain wider exposure to the world.

Analisis Perbandingan Algoritma Klasifikasi MLP dan CNN pada Dataset American Sign Language

People who have hearing loss (deafness) or speech impairment (hearing impairment) usually use sign language to communicate. One of the most basic and flexible sign languages ​​is the Alphabet Sign Language to spell out the words you want to pronounce. Sign language uses hand, finger, and face movements to speak the user's thoughts. However, for alphabetical sign language, facial expressions are not used but only gestures or symbols formed using fingers and hands. In fact, there are still many people who don't understand the meaning of sign language. The use of image classification can help people more easily learn and translate sign language. Image classification accuracy is the main problem in this case. This research conducted a comparison of image classification algorithms, namely Convolutional Neural Network (CNN) and Multilayer Perceptron (MLP) to recognize American Sign Language (ASL) except the letters "J" and "Z" because movement is required for both. This is done to see the effect of the convolution and pooling stages on CNN on the resulting accuracy value and F1 Score in the ASL dataset. Based on the comparison, the use of CNN which begins with Gaussian Low Pass Filtering preprocessing gets the best accuracy of 96.93% and F1 Score 96.97%

Classification of Finger Spelling American Sign Language Using Convolutional Neural Network

Sign language is a combination of complex hand movements, body postures, and facial expressions. However, only a limited number of people can understand and use it. A computer aid sign language recognition with finger spelling style utilizing a convolutional neural network (CNN) is proposed to reduce the burden. We compared two CNN architectures such as Resnet 50, and DenseNet 121 to classify the American sign language dataset. Several data splitting proportions were also tested. From the experimental result, it is shown that the Resnet 50 architecture with 80:20 data splitting for training and testing indicates the best performance with an accuracy of 0.999913, sensitivity 0.998966, precision 0.998958, specificity 0.999955, F1-score 0.999913, and error 0.0000898.

Deep convolutional neural network for hand sign language recognition using model E

An image processing system that based computer vision has received many attentions from science and technology expert. Research on image processing is needed in the development of human-computer interactions such as hand recognition or gesture recognition for people with hearing impairments and deaf people. In this research we try to collect the hand gesture data and used a simple deep neural network architecture that we called model E to recognize the actual hand gestured. The dataset that we used is collected from kaggle.com and in the form of ASL (American Sign Language) datasets. We doing accuracy comparison with another existing model such as AlexNet to see how robust our model. We find that by adjusting kernel size and number of epoch for each model also give a different result. After comparing with AlexNet model we find that our model E is perform better with 96.82% accuracy.

Dynamic Hand Gesture Recognition Based on a Leap Motion Controller and Two-Layer Bidirectional Recurrent Neural Network.

Dynamic hand gesture recognition is one of the most significant tools for human–computer interaction. In order to improve the accuracy of the dynamic hand gesture recognition, in this paper, a two-layer Bidirectional Recurrent Neural Network for the recognition of dynamic hand gestures from a Leap Motion Controller (LMC) is proposed. In addition, based on LMC, an efficient way to capture the dynamic hand gestures is identified. Dynamic hand gestures are represented by sets of feature vectors from the LMC. The proposed system has been tested on the American Sign Language (ASL) datasets with 360 samples and 480 samples, and the Handicraft-Gesture dataset, respectively. On the ASL dataset with 360 samples, the system achieves accuracies of 100% and 96.3% on the training and testing sets. On the ASL dataset with 480 samples, the system achieves accuracies of 100% and 95.2%. On the Handicraft-Gesture dataset, the system achieves accuracies of 100% and 96.7%. In addition, 5-fold, 10-fold, and Leave-One-Out cross-validation are performed on these datasets. The accuracies are 93.33%, 94.1%, and 98.33% (360 samples), 93.75%, 93.5%, and 98.13% (480 samples), and 88.66%, 90%, and 92% on ASL and Handicraft-Gesture datasets, respectively. The developed system demonstrates similar or better performance compared to other approaches in the literature.

Hand Gesture Recognition and Voice Conversion for Speech Impaired

Speech impairment may be infirmities that affect a human’s capability to reach out others via conversation and listening. Humankind in such community, suffering from the above said disability, adopts sign language as an alternate mode of conversation. Even though speech and hearing impaired are using sign language everywhere, the non-sign language speakers face challenges in communicating with speech impaired. Modern approaches in deep learning and computer vision have resulted in a tremendous improvement in the areas of motion and hand sign identification. The main motive presented work is to develop vision-based applications that provide sign language transliteration to speech enabling communication among verbal and non-verbal communicators. The proposed framework uses a methodology of collecting video sequences followed by extracting temporal and spatial features. Recognition of spatial features is compassed using “Convolutional Neural Networks” and to train on temporal features, “Recurrent Neural Networks” were utilized. The databases employed are the American Sign Language datasets.