Hand Sign Recognition Research Articles

Innovative Hand Sign to Text-and-Speech Conversion System

In present scenario, it is difficult for people who are unable to speak due to various reasons. It may not be a permanent condition, as muteness and deafness can be caused or manifest due to several different phenomena, such as physiological injury, illness, medical side effects, psychological trauma, developmental disorders, or neurological disorders. People who are suffering from this typically use the Sign Language to communicate with other people. However, it is not possible for everyone to be acquainted with communication using sign language. This project focuses on developing a solution which can be used by regular people to understand the people having mutism. Hand Sign Recognition Systems have evolved significantly with advancements in machine learning and deep learning techniques. These systems are designed to recognize gestures or signs, making them useful in various applications such as sign language interpretation, human-computer interaction, and virtual reality. This paper gives an overview of the current state of hand sign recognition systems, highlighting the techniques used, challenges, and possible future directions. The focus of the paper is on deep learning-based methods and their impact on the accuracy and usability of hand sign recognition in real-time applications [1]

A Canvas of Air and Signs: Integrating Voice Activated Hand Sign Recognition and Air Canvas for Hearing Impaired and Non-Verbal People

Moving hands in air and watching your screen writes for you and at the same time making different Signs with hands and the system displays it as well as speak what Sign you are making.This seems to be very futuristic approach towards the realm of Image Processing and Gesture Recognition . In this paper, we present a very interesting and a novel approach towards an interactive learning platform where one can draw the content on screen while moving their hand in air and can also use hand sign language to communicate with an ease with the Hearing Impaired and Dumb Community. Our system combines both technologies to create a smooth and engaging experience for users. It can be used in interactive art setups or virtual reality setups. Air canvas enables users to draw and manipulate digital content in mid-air with object tracking using Computer Vision and Mediapipe framework, while hand gesture recognition allows for real-time interpretation of Hand Signs to perform actions or commands within the system. This Model not only recognizes the sign but also speaks it loud using pyttsx3 a text-to-speech conversion Library, ensuring a good communication between a normal human and people with Non-Verbal and Hearing Impaired disability. To enhance the performance of the model We validate the model with a real dataset trained by us. This training was essential for refining the accuracy and efficiency of the model.

An improved custom convolutional neural network based hand sign recognition using machine learning algorithm

AbstractThe biggest challenge the deaf and dumb group faces is that individuals around them do not understand sign language, which they use to communicate with one another. Written communication is slower than face‐to‐face contact, despite the fact that it can be used. Many sign languages have been developed around the world because they are more effective in emergency situations than text‐based communication. India in‐spite of having the large deaf population of almost 18 million and having only around 250 trained/untrained; skilled interpreters. The proposed system can utilize a custom convolution neural networks (CCNNs) model to identify hand motions in order to resolve this issue. This system uses a filter to process the hand before sending it through a classifier to identify the type of hand movements. CCNN strategy employs two levels of algorithm to predict and evaluate symbols that are increasingly similar to one another in order to get as close to precisely recognizing the symbol presented as possible. Convolutional neural networks (CNNs) are able to precisely identify a variety of gestures after being trained on large datasets of hand sign photographs. As a result of their frequent usage of many layers of filters and pooling to extract relevant information from the input images, these networks can recognize hand signs with an accuracy rate of 99.95%, which is much greater than previously built models like SIGNGRAPH, SVM, KNN, CNN + Bi‐LSTM, 3D‐CNN and 2D CNN network and 1D CNN skeleton network. The simulation result shows that a suggested CCNN‐based learning approach is useful for hand sign detection and future usage research when compared with existing machine learning models.

Hand Sign Language Recognition and Conversion into Text Using Convolutional Neural Network

The current research looks into using convolutional neural networks, also known as CNNs, to develop a hand sign understanding and conversion system. The study addresses the urgent need for a precise and effective way to close the communication gap that exists between the hearing and the deaf communities. The study examines how well CNNs can identify and translate hand sign gestures into text, giving sign language users a real-time means of communication. We use three steps in our methodology: gathering datasets, training models, and thorough assessment. The outcomes show encouraging hand sign recognition accuracy, creating new opportunities for inclusive communication. Sign language is one of the oldest and most natural forms of language for communication, but since most people do not know sign language and interpreters are very difficult to come by, we have come up with a real-time method using neural networks for fingerspelling-based American sign language. In our method, the hand is first passed through a filter and after the filter is applied the hand is passed through a classifier which predicts the class of the hand gestures. Our method provides 95.7 % accuracy for the 26 letters of the alphabet. Keywords-Sign Language, ASL, Hearing disability, Convolutional Neural Network (CNN), Computer Vision, Machine Learning, Gesture recognition, Sign language recognition, Hue Saturation Value algorithm.

Hand Sign Recognition for AI Security Involves using Computer Vision and Machine Learning Techniques

Hand Sign Recognition for AI Security Involves using Computer Vision and Machine Learning Techniques

Machine Learning-Based Hand Sign Recognition

Machine Learning-Based Hand Sign Recognition is a project aimed at developing a robust and efficient system for recognizing hand signs and gestures using state-of-the-art machine learning techniques. Hand sign recognition holds significant potential in a wide range of applications, including communication assistance for individuals with hearing impairments, human-computer interaction, and automation in various industries. This project leverages deep learning models, particularly convolutional neural networks (CNNs) and recurrent neural networks (RNNs), to analyze and classify hand signs and gestures. The system is trained on a diverse dataset of hand signs, encompassing different gestures, poses, and lighting conditions, to ensure its adaptability and reliability in real-world scenarios. The project's success is contingent on the fine-tuning of machine learning models, thorough testing, and the incorporation of efficient computer vision techniques. Through rigorous evaluation and validation, we aim to achieve high accuracy and real-time performance. As a result, Machine Learning-Based Hand Sign Recognition has the potential to improve accessibility and convenience for those with hearing impairments, enhance human-computer interaction, and contribute to advancements in automation and robotics across multiple domains

REAL-TIME HAND GESTURE DETECTION AND RECOGNITION USING CONVOLUTIONAL ARTIFICIAL NEURAL NETWORKS

Real-time hand gesture detection is an interesting topic in pattern recognition and computer vision. In this study, we propose the use of a Convolutional Neural Network (CNN) to detect and recognize hands in real-time. Our goal is to develop a system that can accurately identify and interpret user gestures in real-time. The proposed approach involves two main stages, namely hand gesture recognition and gesture recognition. For stage detection, we use the CNN architecture to recognize hands in the video. We train the CNN model using a dataset containing various hand gestures. Once a hand is detected, we extract the relevant hand region and proceed to the gesture recognition stage. The gesture recognition stage involves training and testing CNN models for different hand signal recognition. We use a hand gesture dataset that contains a variety of common hand signals. The experimental results show that the proposed system can detect and recognize hand movements in real-time with satisfactory accuracy. Although there are still some challenges that need to be overcome, this research provides a solid foundation for further development in real-time hand gesture recognition.

K-Tournament Grasshopper Extreme Learner for FMG-Based Gesture Recognition.

The recognition of hand signs is essential for several applications. Due to the variation of possible signals and the complexity of sensor-based systems for hand gesture recognition, a new artificial neural network algorithm providing high accuracy with a reduced architecture and automatic feature selection is needed. In this paper, a novel classification method based on an extreme learning machine (ELM), supported by an improved grasshopper optimization algorithm (GOA) as a core for a weight-pruning process, is proposed. The k-tournament grasshopper optimization algorithm was implemented to select and prune the ELM weights resulting in the proposed k-tournament grasshopper extreme learner (KTGEL) classifier. Myographic methods, such as force myography (FMG), deliver interesting signals that can build the basis for hand sign recognition. FMG was investigated to limit the number of sensors at suitable positions and provide adequate signal processing algorithms for perspective implementation in wearable embedded systems. Based on the proposed KTGEL, the number of sensors and the effect of the number of subjects was investigated in the first stage. It was shown that by increasing the number of subjects participating in the data collection, eight was the minimal number of sensors needed to result in acceptable sign recognition performance. Moreover, implemented with 3000 hidden nodes, after the feature selection wrapper, the ELM had both a microaverage precision and a microaverage sensitivity of 97% for the recognition of a set of gestures, including a middle ambiguity level. The KTGEL reduced the hidden nodes to only 1000, reaching the same total sensitivity with a reduced total precision of only 1% without needing an additional feature selection method.

Hand Sign Recognition for Medical Field Genesis

Abstract: Inability to talk is taken into account to be true incapacity. folks with this incapacity use totally different modes to speak with others, there are variety of strategies accessible for his or her communication one such common methodology of communication is language. Developing language application for speech impaired folks is vital, as they’ll be ready to communicate simply with even people who don’t perceive language. Our project aims at taking the fundamental step in bridging the communication gap between traditional folks, deaf and dumb folk’s victimization language. the most focus of this work is to form a vision-based system to spot language gestures in real-time. Not solely that, however our project conjointly aims to focus on the audience or the users with very little or no data regarding any language. The project can aim to be helpful for each the traditional yet because the person with speaking or hearing incapacity. Although, the scope of the project shall be on the far side the compatibility, still it's herewith tried to usher in the module or a locality of the appliance for the speech impaired person. Future scope of the project aims to develop the appliance to assist person with all ability. Keywords: Hand Sign Recognition, American-Sign-Language, Yolo, Object Detection, Gesture to Speech, CNN, Machine Learning.

Sign Language Recognition

There seems to be an issue between the communication of differently abled and others. That is why we have decided to make an application to mitigate this issue and reduce the dependencies on third parties like translators. We have studied different approaches taken and discussed them. Recognition of hand signs is a popular topic since the 90s and various findings have been uncovered in the following years. We have written about popular methods for this. For this application to work we have to weigh in feature extraction, object detection, and predictions using a machine learning model.

Traffic control hand signal recognition using convolution and recurrent neural networks

Abstract Gesture understanding is one of the most challenging problems in computer vision. Among them, traffic hand signal recognition requires the consideration of speed and the validity of the commanding signal. The lack of available datasets is also a serious problem. Most classifiers approach these problems using the skeletons of target actors in an image. Extracting the three-dimensional coordinates of skeletons is simplified when depth information accompanies the images. However, depth cameras cost significantly more than RGB cameras. Furthermore, the extraction of the skeleton needs to be performed in prior. Here, we show a hand signal detection algorithm without skeletons. Instead of skeletons, we use simple object detectors trained to acquire hand directions. The variance in the time length of gestures mixed with random pauses and noise is handled with a recurrent neural network (RNN). Furthermore, we have developed a flag sequence algorithm to assess the validity of the commanding signal. In whole, the computed hand directions are sent to the RNN, which identifies six types of hand signals given by traffic controllers with the ability to distinguish time variations and intermittent randomly appearing noises. We constructed a hand signal dataset composed of 100 thousand RGB images that is made publicly available. We achieved correct recognition of the hand signals with various backgrounds at 91% accuracy. A processing speed of 30 FPS in FHD video streams, which is a 52% improvement over the best among previous works, was achieved. Despite the extra burden of deciding the validity of the hand signals, this method surpasses methods that solely use RGB video streams. Our work is capable of performing with nonstationary viewpoints, such as those taken from moving vehicles. To accomplish this goal, we set a higher priority for the speed and validity assessment of the recognized commanding signals. The collected dataset is made publicly available through the Korean government portal under the URL “data.go.kr/data/15075814/fileData.do.”

Hand Sign Recognition System Based on EIT Imaging and Robust CNN Classification

Hand sign recognition is gaining importance in human-human and in human-machine communication and interaction. Electrical Impedance Tomography (EIT) is thereby very interesting as it provides information on impedance changes in depth of the Section of the arm, which infers muscle contractions. This paper introduces an EIT imaging system for hand sign recognition and monitoring having a low complexity and including an electronic interface with 8 electrodes placed on the forearm, a Gauss-Newton image reconstruction algorithm, a robust CNN based hand sign classification and a virtual hand model for visualization. A database has been collected for EIT measurements in pole mode taken by eight subjects performing the American sign language numbers from 0 to 9. The overall imaging system is validated using a water tank system, where conductive objects can be changed in properties and positions. The correspondence between the reconstructed images and the expected muscle behavior for the hand signs is investigated. A robust Convolutional Neural Network (CNN) classification algorithm was implemented and optimized by implementing an Adam optimizer and conducting a dedicated study to avoid overfitting. The results obtained by CNN are compared to the results by a Support Vector Machine (SVM), and a Softmax classifier. They show a classification accuracy of 95.94%, 75.61%, and 62.9% respectively. In term of subject dependency, the system using the CNN model shows a higher performance, as the accuracy decreases only by 0.72% while increasing the number of subjects from one to eight. Finally, for visualization, a 3D virtual hand model is designed and controlled based on detected hand signs.

Real-time isolated hand sign language recognition using deep networks and SVD

One of the challenges in computer vision models, especially sign language, is real-time recognition. In this work, we present a simple yet low-complex and efficient model, comprising single shot detector, 2D convolutional neural network, singular value decomposition (SVD), and long short term memory, to real-time isolated hand sign language recognition (IHSLR) from RGB video. We employ the SVD method as an efficient, compact, and discriminative feature extractor from the estimated 3D hand keypoints coordinators. Despite the previous works that employ the estimated 3D hand keypoints coordinates as raw features, we propose a novel and revolutionary way to apply the SVD to the estimated 3D hand keypoints coordinates to get more discriminative features. SVD method is also applied to the geometric relations between the consecutive segments of each finger in each hand and also the angles between these sections. We perform a detailed analysis of recognition time and accuracy. One of our contributions is that this is the first time that the SVD method is applied to the hand pose parameters. Results on four datasets, RKS-PERSIANSIGN ( $$99.5 \pm 0.04$$ ), First-Person ( $$91 \pm 0.06$$ ), ASVID ( $$93 \pm 0.05$$ ), and isoGD ( $$86.1 \pm 0.04$$ ), confirm the efficiency of our method in both accuracy ( $$mean + std$$ ) and time recognition. Furthermore, our model outperforms or gets competitive results with the state-of-the-art alternatives in IHSLR and hand action recognition.

Vision-based hand signal recognition in construction: A feasibility study

In construction fields, it is common for workers to rely on hand signals to communicate and express thoughts due to their simple but effective nature. However, the meaning of these hand signals was not always captured precisely. As a result, construction errors and even accidents were produced. This paper presented a feasibility study on investigating whether the hand signals could be captured and interpreted automatically with computer vision technologies. It starts with the literature review of existing hand gesture recognition methods for sign language understanding, human-computer interaction, etc. It is then followed by creating a dataset containing 11 classes of hand signals in construction. The performance of two state-of-the-art 3D convolutional neural networks is measured and compared. The results indicated that a high classification accuracy (93.3%) and a short inference time (0.17 s/gesture) could be achieved, illustrating the feasibility of using computer vision to automate hand signal recognition in construction.

Hand Sign Recognition Using Deep Learning based on Machine Learning

Hand Sign Recognition Using Deep Learning based on Machine Learning

Deep convolutional neural network for hand sign language recognition using model E

An image processing system that based computer vision has received many attentions from science and technology expert. Research on image processing is needed in the development of human-computer interactions such as hand recognition or gesture recognition for people with hearing impairments and deaf people. In this research we try to collect the hand gesture data and used a simple deep neural network architecture that we called model E to recognize the actual hand gestured. The dataset that we used is collected from kaggle.com and in the form of ASL (American Sign Language) datasets. We doing accuracy comparison with another existing model such as AlexNet to see how robust our model. We find that by adjusting kernel size and number of epoch for each model also give a different result. After comparing with AlexNet model we find that our model E is perform better with 96.82% accuracy.

Hand pose aware multimodal isolated sign language recognition

Isolated hand sign language recognition from video is a challenging research area in computer vision. Some of the most important challenges in this area include dealing with hand occlusion, fast hand movement, illumination changes, or background complexity. While most of the state-of-the-art results in the field have been achieved using deep learning-based models, the previous challenges are not completely solved. In this paper, we propose a hand pose aware model for isolated hand sign language recognition using deep learning approaches from two input modalities, RGB and depth videos. Four spatial feature types: pixel-level, flow, deep hand, and hand pose features, fused from both visual modalities, are input to LSTM for temporal sign recognition. While we use Optical Flow (OF) for flow information in RGB video inputs, Scene Flow (SF) is used for depth video inputs. By including hand pose features, we show a consistent performance improvement of the sign language recognition model. To the best of our knowledge, this is the first time that this discriminant spatiotemporal features, benefiting from the hand pose estimation features and multi-modal inputs, are fused for isolated hand sign language recognition. We perform a step-by-step analysis of the impact in terms of recognition performance of the hand pose features, different combinations of the spatial features, and different recurrent models, especially LSTM and GRU. Results on four public datasets confirm that the proposed model outperforms the current state-of-the-art models on Montalbano II, MSR Daily Activity 3D, and CAD-60 datasets with a relative accuracy improvement of 1.64%, 6.5%, and 7.6%. Furthermore, our model obtains a competitive results on isoGD dataset with only 0.22% margin lower than the current state-of-the-art model.

Video-based isolated hand sign language recognition using a deep cascaded model

In this paper, we propose an efficient cascaded model for sign language recognition taking benefit from spatio-temporal hand-based information using deep learning approaches, especially Single Shot Detector (SSD), Convolutional Neural Network (CNN), and Long Short Term Memory (LSTM), from videos. Our simple yet efficient and accurate model includes two main parts: hand detection and sign recognition. Three types of spatial features, including hand features, Extra Spatial Hand Relation (ESHR) features, and Hand Pose (HP) features, have been fused in the model to feed to LSTM for temporal features extraction. We train SSD model for hand detection using some videos collected from five online sign dictionaries. Our model is evaluated on our proposed dataset (Rastgoo et al., Expert Syst Appl 150: 113336, 2020), including 10’000 sign videos for 100 Persian sign using 10 contributors in 10 different backgrounds, and isoGD dataset. Using the 5-fold cross-validation method, our model outperforms state-of-the-art alternatives in sign language recognition

Hand sign language recognition using multi-view hand skeleton

Hand sign language recognition from video is a challenging research area in computer vision, which performance is affected by hand occlusion, fast hand movement, illumination changes, or background complexity, just to mention a few. In recent years, deep learning approaches have achieved state-of-the-art results in the field, though previous challenges are not completely solved. In this work, we propose a novel deep learning-based pipeline architecture for efficient automatic hand sign language recognition using Single Shot Detector (SSD), 2D Convolutional Neural Network (2DCNN), 3D Convolutional Neural Network (3DCNN), and Long Short-Term Memory (LSTM) from RGB input videos. We use a CNN-based model which estimates the 3D hand keypoints from 2D input frames. After that, we connect these estimated keypoints to build the hand skeleton by using midpoint algorithm. In order to obtain a more discriminative representation of hands, we project 3D hand skeleton into three views surface images. We further employ the heatmap image of detected keypoints as input for refinement in a stacked fashion. We apply 3DCNNs on the stacked features of hand, including pixel level, multi-view hand skeleton, and heatmap features, to extract discriminant local spatio-temporal features from these stacked inputs. The outputs of the 3DCNNs are fused and fed to a LSTM to model long-term dynamics of hand sign gestures. Analyzing 2DCNN vs. 3DCNN using different number of stacked inputs into the network, we demonstrate that 3DCNN better capture spatio-temporal dynamics of hands. To the best of our knowledge, this is the first time that this multi-modal and multi-view set of hand skeleton features are applied for hand sign language recognition. Furthermore, we present a new large-scale hand sign language dataset, namely RKS-PERSIANSIGN, including 10′000 RGB videos of 100 Persian sign words. Evaluation results of the proposed model on three datasets, NYU, First-Person, and RKS-PERSIANSIGN, indicate that our model outperforms state-of-the-art models in hand sign language recognition, hand pose estimation, and hand action recognition.

Hand sign recognition from depth images with multi-scale density features for deaf mute persons

Among many of the fastest growing research fields, sign language recognition is one of the top. Deaf and dumb community uses sign language to express their ideas or views. Sign Language is a methodical coded language where meanings are assigned to every gestures. Many techniques have been developed with the advancement of science and technology to minimize the problem for speech and hearing disabled. The mode of such communication is part of human computer interaction. Hand gesture plays an important role here. The interaction with computer through gesture removes the use of conventional input devices like mouse and keyboards. To create a strong interface between user and computer, recognition of gesture is important. In this paper, a hand gesture recognition method based on multiscale density features is proposed. Depth images of numerals of American Sign Language are considered in this work and recognition rate of 98.20% is obtained, which is comparable with related state-of-the art methods.