Sign Language Recognition System Research Articles

Improved Traffic Sign Recognition System (Itsrs) for Autonomous Vehicle Based on Deep Convolutional Neural Network

Improved Traffic Sign Recognition System (Itsrs) for Autonomous Vehicle Based on Deep Convolutional Neural Network

Traffic_Sign_Classification Using CNN

Autonomous driving cars are booming these days and the demand for a robust traffic sign recognition system that assures safety by recognizing traffic signs accurately and fast is increasing. We all must have heard about self-driving cars in which the passenger can fully depend on the car for traveling. But to achieve level 5 autonomy, it is necessary for all the vehicles to understand and follow the traffic rules. In the global world of Artificial Intelligence (AI) and advancement in technologies, many researchers and big companies like Tesla, Uber, Google, Mercedes-Benz, Toyota, Ford, Audi, etc are working on autonomous vehicles and self-driving cars Here in this paper we planned to use CNN (convolution_neural_network) algorithm is a part of a deep learning model to classify the images present in the dataset into different categories. It can classify 43 different signs of traffic from the German Traffic Sign Recognition benchmark dataset.

Gestures Controlled Home Automation using Deep Learning: A Review

This paper presents the review of the studies carried out on the application using computer vision for hand gesture recognition. Hand gestures can be used to operate the electronic devices. Therefore, it is very much essential to review on the scientific studies whose aim is to develop a technique to achieve more precise and faster sign language recognition system for plain and cluttered backgrounds with different humans to help speech and hearing impaired in their life, robot control, human–computer interaction (HCI), home automation and medical fields. The objective of this paper is to give an overall literature review of the work done related to identify the current technology and methodology used in hand gesture recognition as well as home automation system. This paper mainly describes various images or vision based sign language recognition system comprising pre-processing, feature extraction and classification. The system conversion of sign language to text. The recognized text can be sent to the control model. The gesture based automation communicates directly with control model to control home appliances.

Extension of interval-valued Pythagorean FDOSM for evaluating and benchmarking real-time SLRSs based on multidimensional criteria of hand gesture recognition and sensor glove perspectives

Several DataGlove wearable electronic devices based on real-time Sign Language Recognition Systems (SLRSs) have been developed recently to assist the deaf and dumb community in translating hand gestures to their spoken language equivalents. Multidimensional evaluation and benchmarking of these systems are critical for determining the most desirable approach for meeting all essential requirements. However, this process is considered a Multicriteria Decision-Making (MCDM) problem due to the presence of several issues, including multiple evaluation criteria, criteria importance and criteria confliction. Hence, the MCDM approach is required to solve these issues. In this study, a new extension of the Fuzzy Decision by Opinion Score Method (FDOSM) for evaluating and benchmarking SLRSs is developed under an Interval-Valued Pythagorean Fuzzy Set (IVPFS) named IVP-FDOSM. Fundamentally, the methodology includes two phases. In the first phase, a decision matrix is formulated on the basis of identified ‘multidimensional criteria of hand gesture recognition and sensor glove perspectives’ and ‘real-time SLRSs’. The second phase introduces the development of IVP-FDOSM in two stages. The decision matrix is transformed into an opinion matrix in the first stage (data transformation unit). Meanwhile, in the second stage (data-processing unit), the opinion matrix is converted into fuzzy opinion decision matrices with the assistance of three experts by transforming the opinion matrix’s linguistic terms to Interval-Value Pythagorean Fuzzy Numbers (IVPFNs). Results indicate the following: (1) individual benchmarking results of real-time SLRS showed high variation (90%) based on the preference of each Decision Maker (DM), with only 10% of preferences being identical. (2) The results of group benchmarking reveal that the 10th real-time SLRS was the optimal one and the 6th was the worst. In addition, the rates of ranking match between the group benchmarking and each DM (first DM, second DM and third DM) were 23%, 37% and 43%, respectively. (3) For the results evaluation, the statistical test-based objective assessment shows that the first group received the lowest mean value (2.80333), while the third group received the highest mean (3.8). These values indicate that the group benchmarked systems resulting from IVP-FDOSM are undergoing a systematic ranking. Furthermore, the comparative analysis reveals that IVP-FDOSM is superior to IVP-TOPSIS and IVP-AHP in terms of ranking and weighting.

Analytical review of models and methods for automatic recognition of gestures and sign languages

Introduction: Currently, the recognition of gestures and sign languages is one of the most intensively developing areas in computer vision and applied linguistics. The results of current investigations are applied in a wide range of areas, from sign language translation to gesture-based interfaces. In that regard, various systems and methods for the analysis of gestural data are being developed. Purpose: A detailed review of methods and a comparative analysis of current approaches in automatic recognition of gestures and sign languages. Results: The main gesture recognition problems are the following: detection of articulators (mainly hands), pose estimation and segmentation of gestures in the flow of speech. The authors conclude that the use of two-stream convolutional and recurrent neural network architectures is generally promising for efficient extraction and processing of spatial and temporal features, thus solving the problem of dynamic gestures and coarticulations. This solution, however, heavily depends on the quality and availability of data sets. Practical relevance: This review can be considered a contribution to the study of rapidly developing sign language recognition, irrespective to particular natural sign languages. The results of the work can be used in the development of software systems for automatic gesture and sign language recognition.

Intelligent Malaysian Sign Language Translation System Using Convolutional-Based Attention Module with Residual Network.

The deaf-mutes population always feels helpless when they are not understood by others and vice versa. This is a big humanitarian problem and needs localised solution. To solve this problem, this study implements a convolutional neural network (CNN), convolutional-based attention module (CBAM) to recognise Malaysian Sign Language (MSL) from images. Two different experiments were conducted for MSL signs, using CBAM-2DResNet (2-Dimensional Residual Network) implementing “Within Blocks” and “Before Classifier” methods. Various metrics such as the accuracy, loss, precision, recall, F1-score, confusion matrix, and training time are recorded to evaluate the models' efficiency. The experimental results showed that CBAM-ResNet models achieved a good performance in MSL signs recognition tasks, with accuracy rates of over 90% through a little of variations. The CBAM-ResNet “Before Classifier” models are more efficient than “Within Blocks” CBAM-ResNet models. Thus, the best trained model of CBAM-2DResNet is chosen to develop a real-time sign recognition system for translating from sign language to text and from text to sign language in an easy way of communication between deaf-mutes and other people. All experiment results indicated that the “Before Classifier” of CBAMResNet models is more efficient in recognising MSL and it is worth for future research.

SIGN LANGUAGE RECOGNITION: A SURVEY

Sign Language Recognition is a growing field in research where different people are trying to develop or propose a system that can help the specially-able people of the society in the best possible way so that they can interact and communicate with the people of the society easily and without the problem of them learning the sign language. This language is basically used by the deaf and dumb people of society. This paper describes various ways a sign language recognition system has been built or has been proposed by different researchers. It also aims to have a better understanding of various recognition systems and various methods of recognizing and predicting the hand signs/hand gestures of the specially able people of the society. In return to predict a value which is in a humanreadable format that is text. This paper describes in detail about the various ways and how one method of sign language recognition is different from other based on performance, results, difficulty in developing the model, etc.

Turkish Traffic Sign Recognition: Comparison of Training Step Numbers and Lighting Conditions

With the ever increasing number of vehicles on the roads, traffic signs are becoming more and more important every passing day. Despite the fact that traffic signs are simple and easy to understand, in congested traffic drivers may miss them. Considering that even milliseconds can make a huge difference in preventing accidents, it would make a big help if a system could assist the driver with traffic signs. In order to achieve this, a traffic sign recognition system needs to be implemented. Accordingly, this study aims to develop a Turkish traffic sign detection and recognition system using the Faster R-CNN algorithm. The proposed solution utilizes TensorFlow framework and specifically makes use of the Faster R-CNN Inception-v2-COCO to train the object detection model. For training purposes, indigenous dataset is created containing 54 classes and 10842 Turkish traffic sign images. The training process of the model is carried out twice with step numbers 51,217 and 200,000, respectively. Then, these two models are used to detect 10 Turkish traffic sign images taken both daytime and nighttime. The results indicate that the proposed system’s average precision is 67.2% and average recall is 78.3% when trained with 51,217 steps; on the other hand, the average precision increases to 76% and average recall to 82.8% when trained with 200,000 steps.

Machine learning methods for sign language recognition: A critical review and analysis

• A comprehensive review of artificial intelligent methods applied in sign language recognition systems. • Bibliometric analysis of research progress and publication pattern in intelligent based sign language recognition. • Overview of feature extraction and sign image segmentation techniques. • The publication trend in intelligent sign language recognition system is consistent with the rapid increase in people affected by hearing disability. Sign language is an essential tool to bridge the communication gap between normal and hearing-impaired people. However, the diversity of over 7000 present-day sign languages with variability in motion position, hand shape, and position of body parts making automatic sign language recognition (ASLR) a complex system. In order to overcome such complexity, researchers are investigating better ways of developing ASLR systems to seek intelligent solutions and have demonstrated remarkable success. This paper aims to analyse the research published on intelligent systems in sign language recognition over the past two decades. A total of 649 publications related to decision support and intelligent systems on sign language recognition (SLR) are extracted from the Scopus database and analysed. The extracted publications are analysed using bibliometric VOSViewer software to (1) obtain the publications temporal and regional distributions, (2) create the cooperation networks between affiliations and authors and identify productive institutions in this context. Moreover, reviews of techniques for vision-based sign language recognition are presented. Various features extraction and classification techniques used in SLR to achieve good results are discussed. The literature review presented in this paper shows the importance of incorporating intelligent solutions into the sign language recognition systems and reveals that perfect intelligent systems for sign language recognition are still an open problem. Overall, it is expected that this study will facilitate knowledge accumulation and creation of intelligent-based SLR and provide readers, researchers, and practitioners a roadmap to guide future direction. Flowchart of publications gathering in Scopus database and the procedure for bibliometric analysis .

Technological Aids for Deaf and Mute in The Modern World

Background: People suffering from hearing and speaking disabilities have a few ways of communicating with other people. One of these is to communicate through the use of sign language. Objective: Developing a system for sign language recognition becomes essential for deaf as well as a mute person. The recognition system acts as a translator between a disabled and an able person. This eliminates the hindrances in exchange of ideas. Most of the existing systems are very poorly designed with limited support for the needs of their day to day facilities. Methods: The proposed system embedded with gesture recognition capability has been introduced here which extracts signs from a video sequence and displays them on screen. On the other hand, a speech to text as well as text to speech system is also introduced to further facilitate the grieved people. To get the best out of human computer relationship, the proposed solution consists of various cutting-edge technologies and Machine Learning based sign recognition models which have been trained by using Tensor Flow and Keras library. Result: The proposed architecture works better than several gesture recognition techniques like background elimination and conversion to HSV because of sharply defined image provided to the model for classification. The results of testing indicate reliable recognition systems with high accuracy that includes most of the essential and necessary features for any deaf and dumb person in his/her day to day tasks. Conclusion: It’s the need of current technological advances to develop reliable solutions which can be deployed to assist deaf and dumb people to adjust to normal life. Instead of focusing on a standalone technology, a plethora of them have been introduced in this proposed work. Proposed Sign Recognition System is based on feature extraction and classification. The trained model helps in identification of different gestures.

A Gait Energy Image-Based System for Brazilian Sign Language Recognition

Sign language is the main type of communication of the deaf community. However, most people do not know this language, which causes communication problems for many people. Many technological solutions have been proposed to overcome this issue. Some use the concept of wearable devices, but gesture recognition in a video sequence is a cheaper and less intrusive solution. In this work, we address the problem of gesture recognition in video. To do so, we employ a two-step method with feature space mapping and classification. First, the body parts of each subject in a video are segmented through a deep neural network architecture. Then, we use Gait Energy Image to encode the motion of the body parts in a compact feature space. Small datasets are usually a problem in this type of application, leading to sparse representation in the feature space. To contour this problem, we evaluate SMOTE as a data augmentation technique in the feature space and classical dimensionality reduction techniques. We evaluate our method on three challenging Brazilian sign language (Libras) datasets, CEFET/RJ-Libras, MINDS-Libras, and LIBRAS-UFOP, achieving global accuracies of 85.40±3.13%, 84.66±1.78%, and 64.91±3.79%, respectively, with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">singular value decomposition</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">support vector machine</i> .

Autonomous Driving Robot That Drives and Returns along a Planned Route in Underground Mines by Recognizing Road Signs

In this study, an autonomous driving robot that drives and returns along a planned route in an underground mine tunnel was developed using a machine-vision-based road sign recognition algorithm. The robot was designed to recognize road signs at the intersection of a tunnel using a geometric matching algorithm of machine vision, and the autonomous driving mode was switched according to the shape of the road sign to drive the robot according to the planned route. The autonomous driving mode recognized the shape of the tunnel using the distance data from the LiDAR sensor; it was designed to drive while maintaining a fixed distance from the centerline or one wall of the tunnel. A machine-vision-based road sign recognition system and an autonomous driving robot for underground mines were used in a field experiment. The results reveal that all road signs were accurately recognized, and the average matching score was 979.14 out of 1000, confirming stable driving along the planned route.

Arabic Sign Language Recognition System for Alphabets Using Machine Learning Techniques

In recent years, the role of pattern recognition in systems based on human computer interaction (HCI) has spread in terms of computer vision applications and machine learning, and one of the most important of these applications is to recognize the hand gestures used in dealing with deaf people, in particular to recognize the dashed letters in surahs of the Quran. In this paper, we suggest an Arabic Alphabet Sign Language Recognition System (AArSLRS) using the vision-based approach. The proposed system consists of four stages: the stage of data processing, preprocessing of data, feature extraction, and classification. The system deals with three types of datasets: data dealing with bare hands and a dark background, data dealing with bare hands, but with a light background, and data dealing with hands wearing dark colored gloves. AArSLRS begins with obtaining an image of the alphabet gestures, then revealing the hand from the image and isolating it from the background using one of the proposed methods, after which the hand features are extracted according to the selection method used to extract them. Regarding the classification process in this system, we have used supervised learning techniques for the classification of 28-letter Arabic alphabet using 9240 images. We focused on the classification for 14 alphabetic letters that represent the first Quran surahs in the Quranic sign language (QSL). AArSLRS achieved an accuracy of 99.5% for the K-Nearest Neighbor (KNN) classifier.

Recognition of Persian Sign Language Alphabet Using Gaussian Distribution, Radial Distance and Centroid-Radii

Aims: Presenting a novel approach to identify Persian static symptoms in that the proposed sign language recognition system consists of two segmentation and feature extraction phases. In the segmentation phase, the hand region is separated by an effective segmentation method from the original image. Objective: (1) Introducing an effective framework to solve sensitivity to light conditions in identifying and recognizing letters of the Persian sign language alphabet. (2) Proposing a new feature extraction method to eliminate sensitivity to rotation and scale changes. (3) Creating a new dataset includes 480 images of the sign language alphabet symptoms in order to show the performance of our method. Methods: The proposed sign language recognition system uses two segmentation and feature extraction phases. In the segmentation phase, the hand region is separated by an effective segmentation method from the original image. This method is based on the unique Gaussian model in the YCbCr color space. The Bayes rule is used to precisely identify the hand region too. In the feature extraction phase, the radial model is used to obtain a one-dimensional function to display the hand region boundary and to compute the combined feature vector. In order to normalize this method, the Fourier Transformation method is applied. Results: The system was trained and tested using 480 image samples of Persian sign language characters, 15 images per sign, with the .jpg extension. Extensive experimental evaluations indicate that the proposed recognition system is less susceptible to displacement, scale, and rotation, and can detect symptoms at an accuracy of 95.62%. Conclusion: In this article, a new system was proposed for recognizing the alphabet letters in Persian sign language. This system consists of two main phases: the segmentation phase or the hand region identification and feature extraction phase. In the segmentation phase of the detection system, at first, the hand region is separated by an effective segmentation method from the original image. This method is based on the single Gaussian model in the color space of the YCbCr and the Bayes rule. Applying this method in hand segmentation, enables the system, to recognize the hand region well. In this method, after hand region detection, using the Sobel edge detector, the image edge is extracted. In the feature extraction phase, the radial distance model was used to obtain a one-dimensional function display of the hand region boundary and compute its feature vector. This model is based on the edge and image centroid and computes the centroid distance to the shape boundary as a function of the angle. Since this model by itself is sensitive to the scale variations and tilt, in order to normalize this method to tilt variations, the Fourier transformation was applied to feature vector computed by the radial model. Then the vector elements are divided by the largest vector element, thus, using this method the computed feature vector will be distinct for different images and will show less sensitivity to displacement, scale, and gradient. A database consist of 480 images of Persian sign symptoms was created to the overall accuracy evaluation of the system, the results show that the proposed recognition system can identify 32 letters of the Persian sign language alphabet with a detection rate 95.62%.

CNN Based Approach for Traffic Sign Recognition System

Machine Learning (ML) involves making a machine able to learn and take decisions on real-life problems by working with an efficient set of algorithms. The generated ML models find application in different areas of research and management. One such field, automotive technology, employs ML enabled commercialized advanced driver assistance systems (ADAS) which include traffic sign recognition as a part. With the increasing demand for the intelligence of vehicles, and the advent of self-driving cars, it is extremely necessary to detect and recognize traffic signs automatically through computer technology. For this, neural networks can be applied for analyzing images of traffic signs for cognitive decision making by autonomous vehicles. Neural networks are the computing systems which act as a means of performing ML. In this work, a convolutional neural network (CNN) based ML model is built for recognition of traffic signs accurately for decision making, when installed in driverless vehicles.

AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove

Sign language recognition, especially the sentence recognition, is of great significance for lowering the communication barrier between the hearing/speech impaired and the non-signers. The general glove solutions, which are employed to detect motions of our dexterous hands, only achieve recognizing discrete single gestures (i.e., numbers, letters, or words) instead of sentences, far from satisfying the meet of the signers’ daily communication. Here, we propose an artificial intelligence enabled sign language recognition and communication system comprising sensing gloves, deep learning block, and virtual reality interface. Non-segmentation and segmentation assisted deep learning model achieves the recognition of 50 words and 20 sentences. Significantly, the segmentation approach splits entire sentence signals into word units. Then the deep learning model recognizes all word elements and reversely reconstructs and recognizes sentences. Furthermore, new/never-seen sentences created by new-order word elements recombination can be recognized with an average correct rate of 86.67%. Finally, the sign language recognition results are projected into virtual space and translated into text and audio, allowing the remote and bidirectional communication between signers and non-signers.

The Implications of Weather and Reflectivity Variations on Automatic Traffic Sign Recognition Performance

Automatic recognition of traffic signs in complex, real-world environments has become a pressing research concern with rapid improvements of smart technologies. Hence, this study leveraged an industry-grade object detection and classification algorithm (You-Only-Look-Once, YOLO) to develop an automatic traffic sign recognition system that can identify widely used regulatory and warning signs in diverse driving conditions. Sign recognition performance was assessed in terms of weather and reflectivity to identify the limitations of the developed system in real-world conditions. Furthermore, we produced several editions of our sign recognition system by gradually increasing the number of training images in order to account for the significance of training resources in recognition performance. Analysis considering variable weather conditions, including fair (clear and sunny) and inclement (cloudy and snowy), demonstrated a lower susceptibility of sign recognition in the highly trained system. Analysis considering variable reflectivity conditions, including sheeting type, lighting conditions, and sign age, showed that older engineering-grade sheeting signs were more likely to go unnoticed by the developed system at night. In summary, this study incorporated automatic object detection technology to develop a novel sign recognition system to determine its real-world applicability, opportunities, and limitations for future integration with advanced driver assistance technologies.

Transformation Invariant Real-time Recognition of Indian Sign Language using Feature Fusion

Sign Language Recognition (SLR) is immerging as current area of research in the field of machine learning. SLR system recognizes gestures of sign language and converts them into text/voice thus making the communication possible between deaf and ordinary people. Acceptable performance of such system demands invariance of the output with respect to certain transformations of the input. In this paper, we introduce the real time hand gesture recognition system for Indian Sign Language (ISL). In order to obtain very high recognition accuracy, we propose a hybrid feature vector by combining shape oriented features like Fourier Descriptors and region oriented features like Hu Moments &amp; Zernike Moments. Support Vector Machine (SVM) classifier is trained using feature vectors of images of training dataset. During experiment it is found that the proposed hybrid feature vector enhanced the performance of the system by compactly representing the fundamentals of invariance with respect transformation like scaling, translation and rotation. Being invariant with respect to transformation, system is easy to use and&#x0D; achieved a recognition rate of 95.79%.

A deep learning approach for Indian sign language gestures classification with different backgrounds

The sign language recognition system recently has drawn the attention of various researchers as there is no universal sign language, moreover, it consists of many patterns and postures. Many methods for extracting features and classifying sign language have been proposed in the literature, most of them are based on machine learning techniques. In this article, a deep learning method has been adopted by designing a Convolution Neural Network (CNN) model to extract the sign language features where for classification softmax layer is used. All alphabets in simple as well as complex backgrounds have been considered, where data is collected from 100 subjects in different lighting conditions. The effect of various optimization techniques (Adam, Sgdm, RMSProp), activation functions (ReLU and Leaky ReLU) for generalization ability is also observed. The proposed approach has succeeded in attaining the testing accuracy of 99.10%, 92.69%, and 95.95% on the Indian sign language dataset with simple, complex backgrounds and on a mixed background scenarios, respectively. The model is also tested on NUS dataset-I, NUS dataset-II, their combination, and achieved the accuracy of 100%, 95.95%, and 97.22% respectively.

Toward Improving Confidence in Autonomous Vehicle Software: A Study on Traffic Sign Recognition Systems

This article proposes an approach named SafeML II, which applies empirical cumulative distribution function-based statistical distance measures in a designed human-in-the loop procedure to ensure the safety of machine learning-based classifiers in autonomous vehicle software.