Sign Language Recognition System Research Articles

Traffic sign classification using CNN and detection using faster-RCNN and YOLOV4

Autonomous driving cars are becoming popular everywhere and the need for a robust traffic sign recognition system that ensures safety by recognizing traffic signs accurately and fast is increasing. In this paper, we build a CNN that can classify 43 different traffic signs from the German Traffic Sign Recognition benchmark dataset. The dataset is made up of 39,186 images for training and 12,630 for testing. Our CNN for classification is light and reached an accuracy of 99.20% with only 0.8 M parameters. It is tested also under severe conditions to prove its generalization ability. We also used Faster R–CNN and YOLOv4 networks to implement a recognition system for traffic signs. The German Traffic Sign Detection benchmark dataset was used. Faster R–CNN obtained a mean average precision (mAP) of 43.26% at 6 Frames Per Second (FPS), which is not suitable for real-time application. YOLOv4 achieved an mAP of 59.88% at 35 FPS, which is the preferred model for real-time traffic sign detection. These mAPs are obtained using Intersect Over Union of 50%. A comparative analysis is also presented between these models.

Static and Dynamic Isolated Indian and Russian Sign Language Recognition with Spatial and Temporal Feature Detection Using Hybrid Neural Network

The Sign Language Recognition system intends to recognize the Sign language used by the hearing and vocally impaired populace. The interpretation of isolated sign language from static and dynamic gestures is a difficult study field in machine vision. Managing quick hand movement, facial expression, illumination variations, signer variation, and background complexity are amongst the most serious challenges in this arena. While deep learning-based models have been used to accomplish the entirety of the field's state-of-the-art outcomes, the previous issues have not been fully addressed. To overcome these issues, we propose a Hybrid Neural Network Architecture for the recognition of Isolated Indian and Russian Sign Language. In the case of static gesture recognition, the proposed framework deals with the 3D Convolution Net with an atrous convolution mechanism for spatial feature extraction. For dynamic gesture recognition, the proposed framework is an integration of semantic spatial multi-cue feature detection, extraction, and Temporal-Sequential feature extraction. The semantic spatial multi-cue feature detection and extraction module help in the generation of feature maps for Full-frame, pose, face, and hand. For face and hand detection, GradCam and Camshift algorithm have been used. The temporal and sequential module consists of a modified auto-encoder with a GELU activation function for abstract high-level feature extraction and a hybrid attention layer. The hybrid attention layer is an integration of segmentation and spatial attention mechanism. The proposed work also involves creating a novel multi-signer, single, and double-handed Isolated Sign representation dataset for Indian and Russian Sign Language. The experimentation was done on the novel dataset created. The accuracy obtained for Static Isolated Sign Recognition was 99.76%, and the accuracy obtained for Dynamic Isolated Sign Recognition was 99.85%. We have also compared the performance of our proposed work with other baseline models with benchmark datasets, and our proposed work proved to have better performance in terms of Accuracy metrics.

A Sign Language Recognition System Applied to Deaf-Mute Medical Consultation

It is an objective reality that deaf-mute people have difficulty seeking medical treatment. Due to the lack of sign language interpreters, most hospitals in China currently do not have the ability to interpret sign language. Normal medical treatment is a luxury for deaf people. In this paper, we propose a sign language recognition system: Heart-Speaker. Heart-Speaker is applied to a deaf-mute consultation scenario. The system provides a low-cost solution for the difficult problem of treating deaf-mute patients. The doctor only needs to point the Heart-Speaker at the deaf patient and the system automatically captures the sign language movements and translates the sign language semantics. When a doctor issues a diagnosis or asks a patient a question, the system displays the corresponding sign language video and subtitles to meet the needs of two-way communication between doctors and patients. The system uses the MobileNet-YOLOv3 model to recognize sign language. It meets the needs of running on embedded terminals and provides favorable recognition accuracy. We performed experiments to verify the accuracy of the measurements. The experimental results show that the accuracy rate of Heart-Speaker in recognizing sign language can reach 90.77%.

Transfer learning for Azerbaijani Sign Language Recognition

The goal of sign language technologies is to develop a bridging solution for the communication gap between the hearing-impaired community and the rest of society. Real-time Sign Language Recognition (SLR) is a state-of-the-art subject that promises to facilitate communication between the hearing-impaired community and others. Our research uses transfer learning to provide vision-based sign language recognition. We investigated recent works that use CNN-based methods and provided a literature review on deep learning systems for the sign language recognition (SLR) problem. This paper discusses the architecture of deep learning methods for SLR systems and explains a transfer learning application for fingerspelling sign classification. For the experiments, we used the Azerbaijani Sign Language Fingerspelling dataset and got 88.0% accuracy.

Real time static and dynamic sign language recognition </strong><strong>using deep learning

Sign language recognition systems are used for enabling communication between deaf-mute people and normal user. Spatial localization of the hands could be a challenging task when hands-only occupies 10% of the entire image. This is overcome by designing a real-time efficient system that is capable of performing the task of extraction, recognition, and classification within a single network with the use of a deep convolution network. The recognition is performed for static image dataset with a simple and complex background, dynamic video dataset. Static image dataset is trained and tested using a 2D deep-convolution neural network whereas dynamic video dataset is trained and tested using a 3D deep-convolution neural network. Spatial augmentation is done to increase the number of images of static dataset and key-frame extraction to extract the key-frames from the videos for dynamic dataset. To improve the system performance and accuracy Batch-Normalization layer is added to the convolution network. The accuracy is nearly 99% for dataset with a simple background, 92% for dataset with complex background, and 84% for the video dataset. By obtaining a good accuracy, the system is proved to be real-time efficient in recognizing and interpreting the sign language gestures.

Utilizing motion and spatial features for sign language gesture recognition using cascaded CNN and LSTM models

Sign language is a language produced by body parts gestures and facial expressions. The aim of an automatic sign language recognition system is to assign meaning to each sign gesture. Recently, several computer vision systems have been proposed for sign language recognition using a variety of recognition techniques, sign languages, and gesture modalities. However, one of the challenging problems involves image preprocessing, segmentation, extraction and tracking of relevant static and dynamic features related to manual and nonmanual gestures from different images in sequence. In this paper, we studied the efficiency, scalability, and computation time of three cascaded architectures of convolutional neural network (CNN) and long short-term memory (LSTM) for the recognition of dynamic sign language gestures. The spatial features of dynamic signs are captured using CNN and fed into a multilayer stacked LSTM for temporal information learning. To track the motion in video frames, the absolute temporal differences between consecutive frames are computed and fed into the recognition system. Several experiments have been conducted on three benchmarking datasets of two sign languages to evaluate the proposed models. We also compared the proposed models with other techniques. The attained results show that our models capture better spatio-temporal features pertaining to the recognition of various sign language gestures and consistently outperform other techniques with over 99% accuracy.

Sign language recognition by means of common spatial patterns: An analysis.

Currently there are around 466 million hard of hearing people and this amount is expected to grow in the coming years. Despite the efforts that have been made, there is a communication barrier between deaf and hard of hearing signers and non-signers in environments without an interpreter. Different approaches have been developed lately to try to deal with this issue. In this work, we present an Argentinian Sign Language (LSA) recognition system which uses hand landmarks extracted from videos of the LSA64 dataset in order to distinguish between different signs. Different features are extracted from the signals created with the hand landmarks values, which are first transformed by the Common Spatial Patterns (CSP) algorithm. CSP is a dimensionality reduction algorithm and it has been widely used for EEG systems. The features extracted from the transformed signals have been then used to feed different classifiers, such as Random Forest (RF), K-Nearest Neighbors (KNN) or Multilayer Perceptron (MLP). Several experiments have been performed from which promising results have been obtained, achieving accuracy values between 0.90 and 0.95 on a set of 42 signs.

합성곱 신경망을 이용한 지문자 수어 분류 모델 설계

In the post-COVID-19 era, remote education and discussion using video conferencing applications are spreading, but hearing-impaired people have difficulties in interactive. In this paper, we design a finger sign language recognition system using convolutional neural networks to solve the difficulties experienced by the hearing impaired. The hand area is detected from the image acquired by a camera using the Google MediaPipe Hands solution, and the detected hand area is classified by finger sign language classification model based on LeNet-5. In order to evaluate the performance of the finger sign language classification, the American finger sign language dataset (AFSL dataset) consisting of Alphabet sign language and numeric sign language is constructed from open dataset and the feasibility of real-time finger sign language recognition system is confirmed by implementing a prototype.

Automated Indian sign language recognition system by fusing deep and handcrafted feature

Automated Indian sign language recognition system by fusing deep and handcrafted feature

A Moroccan Sign Language Recognition Algorithm Using a Convolution Neural Network

Gesture recognition is an open phenomenon in computer vision, and one of the topics of current interest. Gesture recognition has many applications in the interpretation of sign language in deaf-mutes, one is in human-computer interaction, and the other is in immersive game technology. For this purpose, we have developed a model of image processing recognition of gestures, based on Artificial Neural Networks, starting from data collection, identification, tracking and classification of gestures, to the display of the obtained results. We propose an approach to contribute to the translation of sign language into voice/text format. In this paper, we present a Moroccan sign language recognition system using a Convolutional Neural Network (CNN). This system includes an important data set of more than 20 files. Each file contains 1,000 static images of each signal from several different angles that we collected with the camera. Different Sign Language models were evaluated and compared with the proposed CNN model. The proposed system achieved 99.33% and achieved the best performance with an accuracy rate of 98.7%.

A deep sign language recognition system for Indian sign language

A deep sign language recognition system for Indian sign language

Development of an American Sign Language Recognition System using Canny Edge and Histogram of Oriented Gradient

Sign language is used by people who have hearing and speaking difficulties, but not understood by many without these difficulties. Therefore, sign language recognition systems are developed to aid communication between hearing impaired people and others. This paper developed a static American Sign Language Recognition (ASLR) system using canny-edge and histogram of oriented gradient (HOG) for feature extraction with K-Nearest Neighbour (K-NN) as classifier. The sign language image datasets used consist of English alphabets from both Massey University and Kaggle, and numbers (0-9) from Massey University. Median filter was used to remove noise after images were converted to grayscale. Otsu algorithm was used for segmentation while edges in the images were preserved using canny edge detection technique with HOG parameters tuning to obtain feature vectors. The extracted features were used by K-NN for classification. An average recognition accuracy and computational testing time of 97.6% and 0.39s respectively were obtained based on experiments with the Massey University dataset. Similarly, an average recognition accuracy and computational testing time of 99.0% and 0.43s respectively were obtained based on experiments with the Kaggle dataset. The developed system successfully recognized static English alphabets and numbers and outperformed some existing systems.

Wi-Phrase: Deep Residual-Multihead Model for WiFi Sign Language Phrase Recognition

Sign language (SL) is used by hearing impaired and deaf people. The WiFi-based sign language recognition (SLR) technology has attracted much attention due to its contactless nature and wide applications. Most previous SLR works are designed to recognize a single isolated sign word in data samples. However, such assumption is not realistic in practical applications, such as, in daily communication, deaf people usually express their mind through a phrase (a.k.a. a group of words) rather than an isolated word. Compared with the previous works, the sign words in a phrase have variety of length, sequential patterns, and combinations in realistic communications between deaf people. Therefore, it is challenging in exploiting the WiFi signals to accurately capture the unique patterns of SL in the previous natural setting. In this article, we propose Wi-Phrase, a multigesture context-awareness SLR system. Wi-Phrase exploits WiFi signals to translate SL to the English phrase. To achieve this, Wi-Phrase employs principal component analysis (PCA) projection to filter out the noise and convert cleaned WiFi signals to spectrogram. Then, we propose a novel Residual-MultiHead model that exploit residual learn structure to obtain local patterns of phrases and adopt multihead block to capture the global context information of phrase. To prove the advanced nature of our model, we design a WiFi-based SL phrase data set of 40 categories for experiments. Our comprehensive evaluation shows that Wi-Phrase achieves an accurate phrase recognition accuracy of 95.03%. In future, we envision Wi-Phrase could be widely used as the phrase command control system for deaf people in IoT devices.

Artificial Intelligence Biosensing System on Hand Gesture Recognition for the Hearing Impaired

AI technologies have the potential to help deaf individuals communicate. Due to the complexity of sign fragmentation and the inadequacy of capturing hand gestures, the authors present a sign language recognition (SLR) system and wearable surface electromyography (sEMG) biosensing device based on a Deep SLR that converts sign language into printed message or speech, allowing people to better understand sign language and hand motions. On the forearms, two armbands containing a biosensor and multi-channel sEMG sensors are mounted to capture quite well arm and finger actions. Deep SLR was tested on an Android and iOS smartphone, and its usefulness was determined by comprehensive testing. Sign Speaker has a considerable limitation in terms of recognising two-handed signs with smartphone and smartwatch. To solve these issues, this research proposes a new real-time end-to-end SLR method. The average word error rate of continuous sentence recognition is 9.6%, and detecting signals and recognising a sentence with six sign words takes less than 0.9 s, demonstrating Deep SLR's recognition.

Real-time sign language recognition system

One of the more natural ways of interaction between the machine and human is offered by Human gestures which are useful for sign language recognition (SLR). Sign language is the speaking tongue of a segment of people who are known as Deaf people. they are the most people who benefit from the (SLR) by the Human-Computer Interaction (HCI).Most natural people cannot understand these kinds of languages, so Deaf people fail to communicate with them without this helper which achieves by the (HCI). This study introduced a proposed system that offered SLR in real-time for some gestures from the American sign language (ASL), by using one of the most suitable deep learning-based architectures that were called Convolutional neural networks (CNN) and choosing the Squeezenet module After comparing it with a traditional machine learning system that relied on extracting [HOG] features. Squeezenet presented the best result in accuracy reached to (100%) in the off-time testing and about (97.5%) in the real-time with a competition time of about 3.3sec for capturing the image, predicting it, and converting it to a text and spoken sentence.The system achieved this result without making any preprocessing for the image, which gave it simplicity and low computing time.

Automatic Traffic Sign Recognition System Using CNN

In recent times, self-driving vehicles have been widely adopted across different countries as they are equipped to drastically reduce the number of road accidents and congestion on the road thereby improving the traffic efficiency. To detect, identify, and label the traffic signs on the road in order to help the Advanced Driver Assistance Systems (ADAS) in these autonomous vehicles with navigation details, a Traffic Sign Recognition (TSR) System using a deep convolutional neural network model, Mask RCNN (Mask Regional Convolutional Neural Network), is proposed in this paper that aims to help the autonomous vehicles comprehend the road ahead and safely navigate to the desired destination. This paper presents the detection and labelling of Indian and European Signs and also the results of the system working efficiently under various challenging visibility conditions. The results obtained show that the Mask RCNN model has recorded higher performance compared to all the other CNN models that have been previously used for traffic sign recognition.

Detection and Recognition of Traffic Sign Boards using Random Forest Classifier

The traffic sign recognition system is a vital aspect of an intelligent transportation system since it provides information to drivers to help them drive more safely and effectively. This paper addresses some of these concerns, which will be accomplished in two steps. The first is the detection of traffic signs, which is divided into two stages. After a picture has been preprocessed to emphasize relevant information, signs are segmented based on color thresholding, shape-based detection. The second task is the recognition of traffic signs. There are two steps involved in this method. In this study, Histogram of Oriented Gradient is utilized as a feature extractor, and Random Forest Classifier is used in the recognition stage. The findings of the experiment show that utilizing Random Forest Classifier resulted in an accuracy score of 95.59 %, precision of 97.55 %, recall of 95.37% in the recognition process and 90.34 % accuracy in the detection process.

A vision-based deep learning approach for independent-users Arabic sign language interpretation

More than 5% of the people around the world are deaf and have severe difficulties in communicating with normal people according to the World Health Organization (WHO). They face a real challenge to express anything without an interpreter for their signs. Nowadays, there are a lot of studies related to Sign Language Recognition (SLR) that aims to reduce this gap between deaf and normal people as it can replace the need for an interpreter. However, there are a lot of challenges facing the sign recognition systems such as low accuracy, complicated gestures, high-level noise, and the ability to operate under variant circumstances with the ability to generalize or to be locked to such limitations. Hence, many researchers proposed different solutions to overcome these problems. Each language has its signs and it can be very challenging to cover all the languages’ signs. The current study objectives: (i) presenting a dataset of 20 Arabic words, and (ii) proposing a deep learning (DL) architecture by combining convolutional neural network (CNN) and recurrent neural network (RNN). The suggested architecture reported 98% accuracy on the presented dataset. It also reported 93.4% and 98.8% for the top-1 and top-5 accuracies on the UCF-101 dataset.

Sign annotation generation to alphabets via integrating visual data with somatosensory data from flexible strain sensor-based data glove

Sign language forms a significant way of communication for deaf-mute people. There are different types of sign languages of different countries in different regions. The sign language recognition system translates the gestures into the alphabet, which solves a barrier in comprehension of conversation. Visual techniques and somatosensory strain sensors are both useful modalities to detect sign gesture. In order to improve the recognition accuracy and wear resistance, this paper proposes a real-time alphabets conversion system, which combines the visual data with the data glove based on flexible strain sensor and the somatosensory data of the camera, supplemented by deep learning. In this learning structure, convolutional neural network is used to process the visual data, and then the sparse neural network for sensor data fusion and recognition is realized in the feature layer. A module is proposed to train the linking of hand motions primitives to alphabets (A–Z). The proposed framework is tested on a dataset of sign and descriptive alphabets. The proposed system would recognize the 26 letters with a recognition rate of up to 99.50% and a recognition time of less than 1 s, which helps to understand observations of sign as words and texts.

Yolo-Based Traffic Sign Recognition Algorithm.

With the rapid development of intelligent transportation, more and more vehicles are equipped with intelligent traffic sign recognition systems, which can reduce the potential safety hazards caused by human cognitive errors. Therefore, a more safe and reliable traffic sign recognition system is the demand of drivers, and it is also the research hotspot of current automobile manufacturers. However, the pictures taken by the actual driving car are inevitably distorted and blurred. In addition, there are external uncontrollable factors, such as the impact of bad weather, which make the research of traffic sign recognition system face many difficulties, and the practical application is far from mature. In order to solve the above challenges, this paper proposes a Yolo model for traffic sign recognition. Firstly, the traffic signs are roughly divided into several categories and then preprocessed according to the characteristics of various types of signs. The processed pictures are input into the optimized convolutional neural network to subdivide the categories to obtain the specific categories. Finally, the proposed recognition algorithm is tested with the data set based on the German traffic sign recognition standard and compared with other baseline algorithms. The results show that the algorithm greatly improves the running speed on the basis of ensuring a high classification accuracy and is more suitable for traffic sign recognition system.