Sign Language Recognition System Research Articles

Technological Aids for Deaf and Mute in The Modern World

Background: People suffering from hearing and speaking disabilities have a few ways of communicating with other people. One of these is to communicate through the use of sign language. Objective: Developing a system for sign language recognition becomes essential for deaf as well as a mute person. The recognition system acts as a translator between a disabled and an able person. This eliminates the hindrances in exchange of ideas. Most of the existing systems are very poorly designed with limited support for the needs of their day to day facilities. Methods: The proposed system embedded with gesture recognition capability has been introduced here which extracts signs from a video sequence and displays them on screen. On the other hand, a speech to text as well as text to speech system is also introduced to further facilitate the grieved people. To get the best out of human computer relationship, the proposed solution consists of various cutting-edge technologies and Machine Learning based sign recognition models which have been trained by using Tensor Flow and Keras library. Result: The proposed architecture works better than several gesture recognition techniques like background elimination and conversion to HSV because of sharply defined image provided to the model for classification. The results of testing indicate reliable recognition systems with high accuracy that includes most of the essential and necessary features for any deaf and dumb person in his/her day to day tasks. Conclusion: It’s the need of current technological advances to develop reliable solutions which can be deployed to assist deaf and dumb people to adjust to normal life. Instead of focusing on a standalone technology, a plethora of them have been introduced in this proposed work. Proposed Sign Recognition System is based on feature extraction and classification. The trained model helps in identification of different gestures.

A Gait Energy Image-Based System for Brazilian Sign Language Recognition

Sign language is the main type of communication of the deaf community. However, most people do not know this language, which causes communication problems for many people. Many technological solutions have been proposed to overcome this issue. Some use the concept of wearable devices, but gesture recognition in a video sequence is a cheaper and less intrusive solution. In this work, we address the problem of gesture recognition in video. To do so, we employ a two-step method with feature space mapping and classification. First, the body parts of each subject in a video are segmented through a deep neural network architecture. Then, we use Gait Energy Image to encode the motion of the body parts in a compact feature space. Small datasets are usually a problem in this type of application, leading to sparse representation in the feature space. To contour this problem, we evaluate SMOTE as a data augmentation technique in the feature space and classical dimensionality reduction techniques. We evaluate our method on three challenging Brazilian sign language (Libras) datasets, CEFET/RJ-Libras, MINDS-Libras, and LIBRAS-UFOP, achieving global accuracies of 85.40±3.13%, 84.66±1.78%, and 64.91±3.79%, respectively, with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">singular value decomposition</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">support vector machine</i> .

Autonomous Driving Robot That Drives and Returns along a Planned Route in Underground Mines by Recognizing Road Signs

In this study, an autonomous driving robot that drives and returns along a planned route in an underground mine tunnel was developed using a machine-vision-based road sign recognition algorithm. The robot was designed to recognize road signs at the intersection of a tunnel using a geometric matching algorithm of machine vision, and the autonomous driving mode was switched according to the shape of the road sign to drive the robot according to the planned route. The autonomous driving mode recognized the shape of the tunnel using the distance data from the LiDAR sensor; it was designed to drive while maintaining a fixed distance from the centerline or one wall of the tunnel. A machine-vision-based road sign recognition system and an autonomous driving robot for underground mines were used in a field experiment. The results reveal that all road signs were accurately recognized, and the average matching score was 979.14 out of 1000, confirming stable driving along the planned route.

Arabic Sign Language Recognition System for Alphabets Using Machine Learning Techniques

In recent years, the role of pattern recognition in systems based on human computer interaction (HCI) has spread in terms of computer vision applications and machine learning, and one of the most important of these applications is to recognize the hand gestures used in dealing with deaf people, in particular to recognize the dashed letters in surahs of the Quran. In this paper, we suggest an Arabic Alphabet Sign Language Recognition System (AArSLRS) using the vision-based approach. The proposed system consists of four stages: the stage of data processing, preprocessing of data, feature extraction, and classification. The system deals with three types of datasets: data dealing with bare hands and a dark background, data dealing with bare hands, but with a light background, and data dealing with hands wearing dark colored gloves. AArSLRS begins with obtaining an image of the alphabet gestures, then revealing the hand from the image and isolating it from the background using one of the proposed methods, after which the hand features are extracted according to the selection method used to extract them. Regarding the classification process in this system, we have used supervised learning techniques for the classification of 28-letter Arabic alphabet using 9240 images. We focused on the classification for 14 alphabetic letters that represent the first Quran surahs in the Quranic sign language (QSL). AArSLRS achieved an accuracy of 99.5% for the K-Nearest Neighbor (KNN) classifier.

Recognition of Persian Sign Language Alphabet Using Gaussian Distribution, Radial Distance and Centroid-Radii

Aims: Presenting a novel approach to identify Persian static symptoms in that the proposed sign language recognition system consists of two segmentation and feature extraction phases. In the segmentation phase, the hand region is separated by an effective segmentation method from the original image. Objective: (1) Introducing an effective framework to solve sensitivity to light conditions in identifying and recognizing letters of the Persian sign language alphabet. (2) Proposing a new feature extraction method to eliminate sensitivity to rotation and scale changes. (3) Creating a new dataset includes 480 images of the sign language alphabet symptoms in order to show the performance of our method. Methods: The proposed sign language recognition system uses two segmentation and feature extraction phases. In the segmentation phase, the hand region is separated by an effective segmentation method from the original image. This method is based on the unique Gaussian model in the YCbCr color space. The Bayes rule is used to precisely identify the hand region too. In the feature extraction phase, the radial model is used to obtain a one-dimensional function to display the hand region boundary and to compute the combined feature vector. In order to normalize this method, the Fourier Transformation method is applied. Results: The system was trained and tested using 480 image samples of Persian sign language characters, 15 images per sign, with the .jpg extension. Extensive experimental evaluations indicate that the proposed recognition system is less susceptible to displacement, scale, and rotation, and can detect symptoms at an accuracy of 95.62%. Conclusion: In this article, a new system was proposed for recognizing the alphabet letters in Persian sign language. This system consists of two main phases: the segmentation phase or the hand region identification and feature extraction phase. In the segmentation phase of the detection system, at first, the hand region is separated by an effective segmentation method from the original image. This method is based on the single Gaussian model in the color space of the YCbCr and the Bayes rule. Applying this method in hand segmentation, enables the system, to recognize the hand region well. In this method, after hand region detection, using the Sobel edge detector, the image edge is extracted. In the feature extraction phase, the radial distance model was used to obtain a one-dimensional function display of the hand region boundary and compute its feature vector. This model is based on the edge and image centroid and computes the centroid distance to the shape boundary as a function of the angle. Since this model by itself is sensitive to the scale variations and tilt, in order to normalize this method to tilt variations, the Fourier transformation was applied to feature vector computed by the radial model. Then the vector elements are divided by the largest vector element, thus, using this method the computed feature vector will be distinct for different images and will show less sensitivity to displacement, scale, and gradient. A database consist of 480 images of Persian sign symptoms was created to the overall accuracy evaluation of the system, the results show that the proposed recognition system can identify 32 letters of the Persian sign language alphabet with a detection rate 95.62%.

CNN Based Approach for Traffic Sign Recognition System

Machine Learning (ML) involves making a machine able to learn and take decisions on real-life problems by working with an efficient set of algorithms. The generated ML models find application in different areas of research and management. One such field, automotive technology, employs ML enabled commercialized advanced driver assistance systems (ADAS) which include traffic sign recognition as a part. With the increasing demand for the intelligence of vehicles, and the advent of self-driving cars, it is extremely necessary to detect and recognize traffic signs automatically through computer technology. For this, neural networks can be applied for analyzing images of traffic signs for cognitive decision making by autonomous vehicles. Neural networks are the computing systems which act as a means of performing ML. In this work, a convolutional neural network (CNN) based ML model is built for recognition of traffic signs accurately for decision making, when installed in driverless vehicles.

AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove

Sign language recognition, especially the sentence recognition, is of great significance for lowering the communication barrier between the hearing/speech impaired and the non-signers. The general glove solutions, which are employed to detect motions of our dexterous hands, only achieve recognizing discrete single gestures (i.e., numbers, letters, or words) instead of sentences, far from satisfying the meet of the signers’ daily communication. Here, we propose an artificial intelligence enabled sign language recognition and communication system comprising sensing gloves, deep learning block, and virtual reality interface. Non-segmentation and segmentation assisted deep learning model achieves the recognition of 50 words and 20 sentences. Significantly, the segmentation approach splits entire sentence signals into word units. Then the deep learning model recognizes all word elements and reversely reconstructs and recognizes sentences. Furthermore, new/never-seen sentences created by new-order word elements recombination can be recognized with an average correct rate of 86.67%. Finally, the sign language recognition results are projected into virtual space and translated into text and audio, allowing the remote and bidirectional communication between signers and non-signers.

The Implications of Weather and Reflectivity Variations on Automatic Traffic Sign Recognition Performance

Automatic recognition of traffic signs in complex, real-world environments has become a pressing research concern with rapid improvements of smart technologies. Hence, this study leveraged an industry-grade object detection and classification algorithm (You-Only-Look-Once, YOLO) to develop an automatic traffic sign recognition system that can identify widely used regulatory and warning signs in diverse driving conditions. Sign recognition performance was assessed in terms of weather and reflectivity to identify the limitations of the developed system in real-world conditions. Furthermore, we produced several editions of our sign recognition system by gradually increasing the number of training images in order to account for the significance of training resources in recognition performance. Analysis considering variable weather conditions, including fair (clear and sunny) and inclement (cloudy and snowy), demonstrated a lower susceptibility of sign recognition in the highly trained system. Analysis considering variable reflectivity conditions, including sheeting type, lighting conditions, and sign age, showed that older engineering-grade sheeting signs were more likely to go unnoticed by the developed system at night. In summary, this study incorporated automatic object detection technology to develop a novel sign recognition system to determine its real-world applicability, opportunities, and limitations for future integration with advanced driver assistance technologies.

Transformation Invariant Real-time Recognition of Indian Sign Language using Feature Fusion

Sign Language Recognition (SLR) is immerging as current area of research in the field of machine learning. SLR system recognizes gestures of sign language and converts them into text/voice thus making the communication possible between deaf and ordinary people. Acceptable performance of such system demands invariance of the output with respect to certain transformations of the input. In this paper, we introduce the real time hand gesture recognition system for Indian Sign Language (ISL). In order to obtain very high recognition accuracy, we propose a hybrid feature vector by combining shape oriented features like Fourier Descriptors and region oriented features like Hu Moments &amp; Zernike Moments. Support Vector Machine (SVM) classifier is trained using feature vectors of images of training dataset. During experiment it is found that the proposed hybrid feature vector enhanced the performance of the system by compactly representing the fundamentals of invariance with respect transformation like scaling, translation and rotation. Being invariant with respect to transformation, system is easy to use and&#x0D; achieved a recognition rate of 95.79%.

A deep learning approach for Indian sign language gestures classification with different backgrounds

The sign language recognition system recently has drawn the attention of various researchers as there is no universal sign language, moreover, it consists of many patterns and postures. Many methods for extracting features and classifying sign language have been proposed in the literature, most of them are based on machine learning techniques. In this article, a deep learning method has been adopted by designing a Convolution Neural Network (CNN) model to extract the sign language features where for classification softmax layer is used. All alphabets in simple as well as complex backgrounds have been considered, where data is collected from 100 subjects in different lighting conditions. The effect of various optimization techniques (Adam, Sgdm, RMSProp), activation functions (ReLU and Leaky ReLU) for generalization ability is also observed. The proposed approach has succeeded in attaining the testing accuracy of 99.10%, 92.69%, and 95.95% on the Indian sign language dataset with simple, complex backgrounds and on a mixed background scenarios, respectively. The model is also tested on NUS dataset-I, NUS dataset-II, their combination, and achieved the accuracy of 100%, 95.95%, and 97.22% respectively.

Toward Improving Confidence in Autonomous Vehicle Software: A Study on Traffic Sign Recognition Systems

This article proposes an approach named SafeML II, which applies empirical cumulative distribution function-based statistical distance measures in a designed human-in-the loop procedure to ensure the safety of machine learning-based classifiers in autonomous vehicle software.

A Research Paper on Convolutional Neural Network (CNN) Theory based Automated Traffic and Road Sign Detection and Recognition System

India is a country with a dense road network and has a complex system to maintain road safety. As we all know that we have a complex traffic system in which we have more than 100 types of traffic symbols in it. While driving, it is tough to take care of all the symbols placed at the road end. Sometimes the driver does not know what that symbol says. In this system sometimes the driver misses the road signs because the attention of the driver is overdriving the vehicle safe which leads to an accident or issuing Challan. Sometimes the traffic signs don't notice by the driver. So all the drivers or the vehicle need a system which is capable to read and recognize the traffic symbol placed at the road end and the system must be capable of giving simple instruction to the driver. So that system can automatically detect which type of symbol is this and can notify the driver. The system must have a good accuracy rate, as well as the system, must have a very good speed of working. This system can also be used in driverless cars to notify the system about the road signals and hence the system can tackle all the symbols carefully.

Towards Hybrid Multimodal Manual and Non-Manual Arabic Sign Language Recognition: mArSL Database and Pilot Study

Sign languages are the main visual communication medium between hard-hearing people and their societies. Similar to spoken languages, they are not universal and vary from region to region, but they are relatively under-resourced. Arabic sign language (ArSL) is one of these languages that has attracted increasing attention in the research community. However, most of the existing and available works on sign language recognition systems focus on manual gestures, ignoring other non-manual information needed for other language signals such as facial expressions. One of the main challenges of not considering these modalities is the lack of suitable datasets. In this paper, we propose a new multi-modality ArSL dataset that integrates various types of modalities. It consists of 6748 video samples of fifty signs performed by four signers and collected using Kinect V2 sensors. This dataset will be freely available for researchers to develop and benchmark their techniques for further advancement of the field. In addition, we evaluated the fusion of spatial and temporal features of different modalities, manual and non-manual, for sign language recognition using the state-of-the-art deep learning techniques. This fusion boosted the accuracy of the recognition system at the signer-independent mode by 3.6% compared with manual gestures.

Smart Communication Assistant for Deaf and Dumb People

Communication is the medium by which we can share our thoughts or convey the messages with other person. Nowadays we can give commands using voice recognition. But what if one absolutely cannot hear anything and eventually cannot speak. So the Sign Language is the main communicating tool for hearing impaired and mute people, and also to ensure an independent life for them, the automatic interpretation of sign language is an extensive research area. Sign language recognition (SLR) aims to interpret sign languages automatically by an application in order to help the deaf people to communicate with hearing society conveniently. Our aim is to design a system to help the Deaf and Dumb person to communicate with the rest of the world using sign language. With the use of image processing and artificial intelligence, many techniques and algorithms have been developed in this area. Every sign language recognition system is trained for recognizing the signs and converting them into required pattern. The proposed system aim to provide speech to speechless, in this paper we have introduced Sign Language Recognition using CNN for dynamic gestures to achieve faster results with high accuracy.

Applying deep neural networks for the automatic recognition of sign language words: A communication aid to deaf agriculturists

One of the major challenges that deaf people face in modern societal life is communication. For those engaged in agricultural jobs, efficiency at work and productivity are deeply related to the quality of deciphering the sign language used by the deaf farmers. Employing sign language interpreters is not a pragmatic solution to this problem. There comes the need for developing a reliable system for automatic sign language recognition (SLR). This paper reports a work on the recognition of hand gestures for the Indian sign language (ISL) words commonly used by deaf farmers. A hybrid deep learning model with convolutional long short term memory (LSTM) network has been exploited for gesture classification. The model has attained an average classification accuracy of 76.21% on the proposed dataset of ISL words from the agricultural domain.

Sign Language to Text-Speech Translator Using Machine Learning

Communication with deaf and dumb people is quite difficult task for others. So, through sign language can communicate with deaf and mute persons but it is difficult for normal people to understand the sign language hence it creates a huge gap between them and it's uneasy to exchange their ideas, thoughts with others. This gap has existed for years in order to minimize this, new technologies should be emerged. Therefore, an interpreter is necessary which acts as a bridge between deaf-mute and others. This paper proposed system which is a sign language translator. The system used American Sign Language (ASL) dataset which is pre-processed based on threshold and intensity. This system recognizes sign language alphabet and by joining the letters it creates a sentence then it converts the text to speech. As the system is based on hand, hand gesture is used in sign language recognition system, for that the efficient hand tracking technique which is given by media pipe cross platform is used and it exactly detects the hand after that by using the ANN architecture the model has trained and which classifies the images. The system has achieved 74% accuracy and recognize almost all the letters. The system which also converts sign text to speech so that it will also helpful for blind people.

Smart, comfortable wearable system for recognizing Arabic Sign Language in real-time using IMUs and features-based fusion

Automatic sign language recognition systems have been presented in the literature to avail as a helpful device for deaf and mute individuals to incorporate them into the community without impediments. Gesture recognition is usually done by using vision-based techniques or hand-attached sensors that track hand and finger movements. This paper presents a sensors-based Arabic Sign Language recognition system. The proposed system depends on the information fusion from a 3-axis accelerometer and gyroscope Inertial Measurement Units (IMUs) sensors. The architecture of the proposed system consists of two main modules: a wearable device (glove), which includes sensors, a communication unit, and micro-controller unit, and a software module running on a mobile device. The proposed glove system utilizes six IMUs sensors where five of them placed at the fingers and the sixth sensor placed at the back of the palm. The collected data from sensors are transmitted to a mobile device through a Bluetooth low-energy wireless transmission for data analysis and gesture recognition. For each gesture, the beginning and endpoints of the significative gesture part are discovered automatically using the intensity of the gyroscope signal. The system employs a features-based sensor fusion that combines accelerometer and gyroscope time-domain and frequency-domain features. In a comprehensive set of experiments with seven participants, the system was trained and tested with 28 Arabic alphabets, and the best recognition accuracy achieved was 98.6% for the user-dependent data and 96% for the user-independent data under support vector machine classifier. The achieved results are outperforming existing systems that have been presented in recent decades for alphabet-level gesture recognition.

Analysis of Market-Ready Traffic Sign Recognition Systems in Cars: A Test Field Study

Advanced Driver Assistance System (ADAS) represents a collection of vehicle-based intelligent safety systems. One in particular, Traffic Sign Recognition System (TSRS), is designed to detect and interpret roadside information in the form of signage. Even though TSRS has been on the market for more than a decade now, the available ones differ in hardware and software solutions they use, as well as in quantity and typology of signs they recognize. The aim of this study is to determine whether differences between detection and readability accuracy of market-ready TSRS exist and to what extent, as well as how different levels of “graphical changes” on the signs affect their accuracy. For this purpose, signs (“speed limit” and “prohibition of overtaking”) were placed on a test field and 17 vehicles from 14 different car brands underwent testing. Overall, the results showed that sign detection and readability by TSRS differ between car brands and that even small changes in the design of signs can drastically affect TSRS accuracy. Even in a controlled environment where no sign has been altered, there has been a 5% margin of misread signs.

Indian Sign Language Recognition through Hybrid ConvNet-LSTM Networks

Dynamic hand gesture recognition is a challenging task of Human-Computer Interaction (HCI) and Computer Vision. The potential application areas of gesture recognition include sign language translation, video gaming, video surveillance, robotics, and gesture-controlled home appliances. In the proposed research, gesture recognition is applied to recognize sign language words from real-time videos. Classifying the actions from video sequences requires both spatial and temporal features. The proposed system handles the former by the Convolutional Neural Network (CNN), which is the core of several computer vision solutions and the latter by the Recurrent Neural Network (RNN), which is more efficient in handling the sequences of movements. Thus, the real-time Indian sign language (ISL) recognition system is developed using the hybrid CNN-RNN architecture. The system is trained with the proposed CasTalk-ISL dataset. The ultimate purpose of the presented research is to deploy a real-time sign language translator to break the hurdles present in the communication between hearing-impaired people and normal people. The developed system achieves 95.99% top-1 accuracy and 99.46% top-3 accuracy on the test dataset. The obtained results outperform the existing approaches using various deep models on different datasets.

Sign language recognition using Kinect sensor based on color stream and skeleton points

This paper presents a sign language recognition system based on color stream and skeleton points. Several approaches have been established to address sign language recognition problems. However, most of the previous approaches still have poor recognition accuracy. The proposed approach uses Kinect sensor based on color stream and skeleton points from the depth stream to improved recognition accuracy. Techniques within this approach use hand trajectories and hand shapes in combating sign recognition challenges. Therefore, for a particular sign a representative feature vector is extracted, which consists of hand trajectories and hand shapes. A sparse dictionary learning algorithm, Label Consistent K-SVD (LC-KSVD) is applied to obtain a discriminative dictionary. Based on that, the system was further developed to a new classification approach for better results. The proposed system was fairly evaluated based on 21 sign words including one-handed signs and two-handed signs. It was observed that the proposed system gets high recognition accuracy of 98.25%, and obtained an average accuracy of 95.34% for signer independent recognition.&#x0D; Keywords: Sign language, Color stream, Skeleton points, Kinect sensor, Discriminative dictionary.