Language Recognition System Research Articles

A Gait Energy Image-Based System for Brazilian Sign Language Recognition

Sign language is the main type of communication of the deaf community. However, most people do not know this language, which causes communication problems for many people. Many technological solutions have been proposed to overcome this issue. Some use the concept of wearable devices, but gesture recognition in a video sequence is a cheaper and less intrusive solution. In this work, we address the problem of gesture recognition in video. To do so, we employ a two-step method with feature space mapping and classification. First, the body parts of each subject in a video are segmented through a deep neural network architecture. Then, we use Gait Energy Image to encode the motion of the body parts in a compact feature space. Small datasets are usually a problem in this type of application, leading to sparse representation in the feature space. To contour this problem, we evaluate SMOTE as a data augmentation technique in the feature space and classical dimensionality reduction techniques. We evaluate our method on three challenging Brazilian sign language (Libras) datasets, CEFET/RJ-Libras, MINDS-Libras, and LIBRAS-UFOP, achieving global accuracies of 85.40±3.13%, 84.66±1.78%, and 64.91±3.79%, respectively, with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">singular value decomposition</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">support vector machine</i> .

Machine learning methods for sign language recognition: A critical review and analysis

• A comprehensive review of artificial intelligent methods applied in sign language recognition systems. • Bibliometric analysis of research progress and publication pattern in intelligent based sign language recognition. • Overview of feature extraction and sign image segmentation techniques. • The publication trend in intelligent sign language recognition system is consistent with the rapid increase in people affected by hearing disability. Sign language is an essential tool to bridge the communication gap between normal and hearing-impaired people. However, the diversity of over 7000 present-day sign languages with variability in motion position, hand shape, and position of body parts making automatic sign language recognition (ASLR) a complex system. In order to overcome such complexity, researchers are investigating better ways of developing ASLR systems to seek intelligent solutions and have demonstrated remarkable success. This paper aims to analyse the research published on intelligent systems in sign language recognition over the past two decades. A total of 649 publications related to decision support and intelligent systems on sign language recognition (SLR) are extracted from the Scopus database and analysed. The extracted publications are analysed using bibliometric VOSViewer software to (1) obtain the publications temporal and regional distributions, (2) create the cooperation networks between affiliations and authors and identify productive institutions in this context. Moreover, reviews of techniques for vision-based sign language recognition are presented. Various features extraction and classification techniques used in SLR to achieve good results are discussed. The literature review presented in this paper shows the importance of incorporating intelligent solutions into the sign language recognition systems and reveals that perfect intelligent systems for sign language recognition are still an open problem. Overall, it is expected that this study will facilitate knowledge accumulation and creation of intelligent-based SLR and provide readers, researchers, and practitioners a roadmap to guide future direction. Flowchart of publications gathering in Scopus database and the procedure for bibliometric analysis .

Arabic Sign Language Recognition System for Alphabets Using Machine Learning Techniques

In recent years, the role of pattern recognition in systems based on human computer interaction (HCI) has spread in terms of computer vision applications and machine learning, and one of the most important of these applications is to recognize the hand gestures used in dealing with deaf people, in particular to recognize the dashed letters in surahs of the Quran. In this paper, we suggest an Arabic Alphabet Sign Language Recognition System (AArSLRS) using the vision-based approach. The proposed system consists of four stages: the stage of data processing, preprocessing of data, feature extraction, and classification. The system deals with three types of datasets: data dealing with bare hands and a dark background, data dealing with bare hands, but with a light background, and data dealing with hands wearing dark colored gloves. AArSLRS begins with obtaining an image of the alphabet gestures, then revealing the hand from the image and isolating it from the background using one of the proposed methods, after which the hand features are extracted according to the selection method used to extract them. Regarding the classification process in this system, we have used supervised learning techniques for the classification of 28-letter Arabic alphabet using 9240 images. We focused on the classification for 14 alphabetic letters that represent the first Quran surahs in the Quranic sign language (QSL). AArSLRS achieved an accuracy of 99.5% for the K-Nearest Neighbor (KNN) classifier.

Bangla Sign alphabet recognition with zero-shot and transfer learning

Bangla, being the fifth most spoken language in the world has its own distinct sign language with two methods (one-handed and two-handed) of representation. However, a standard automatic recognition system of Bangla sign language (BdSL) is still to be achieved. Though widely studied and explored by researchers in the past years, certain unaddressed issues like identifying unseen signs and both types of BdSL or lack of evaluation of the models in versatile environmental conditions demarcate the real-world implementation of the automatic recognition of BdSL. To find a probable solution to the shortcomings in the existing works, this paper proposes two approaches based on conventional transfer learning and contemporary Zero-shot learning (ZSL) for automatic BdSL alphabet recognition of both seen and unseen data. The performance of the proposed system is evaluated for both types of Bangla sign representations as well as on a large dataset with 35,149 images from over 350 subjects, varying in terms of backgrounds, camera angle, light contrast, skin tone, hand size, and orientation. For the ZSL approach, a new semantic descriptor dedicated to BdSL is created and a split of the dataset into seen and unseen classes is proposed. Our model achieved 68.21%, 91.57%, and 54.34% of harmonic mean accuracy, seen accuracy, and zero-shot accuracy with six unseen classes respectively. For the transfer learning-based approach, we found pre-trained DenseNet201 architecture to be the best performing feature extractor and Linear Discriminant Analysis as the best classifier with an overall accuracy of 93.68% on the large dataset after conducting quantitative experimentation on 18 CNN architectures and 21 classifiers. The satisfactory result from our models supports its very probative potential to serve extensively for the hearing and speaking impaired community.

Recognition of Persian Sign Language Alphabet Using Gaussian Distribution, Radial Distance and Centroid-Radii

Aims: Presenting a novel approach to identify Persian static symptoms in that the proposed sign language recognition system consists of two segmentation and feature extraction phases. In the segmentation phase, the hand region is separated by an effective segmentation method from the original image. Objective: (1) Introducing an effective framework to solve sensitivity to light conditions in identifying and recognizing letters of the Persian sign language alphabet. (2) Proposing a new feature extraction method to eliminate sensitivity to rotation and scale changes. (3) Creating a new dataset includes 480 images of the sign language alphabet symptoms in order to show the performance of our method. Methods: The proposed sign language recognition system uses two segmentation and feature extraction phases. In the segmentation phase, the hand region is separated by an effective segmentation method from the original image. This method is based on the unique Gaussian model in the YCbCr color space. The Bayes rule is used to precisely identify the hand region too. In the feature extraction phase, the radial model is used to obtain a one-dimensional function to display the hand region boundary and to compute the combined feature vector. In order to normalize this method, the Fourier Transformation method is applied. Results: The system was trained and tested using 480 image samples of Persian sign language characters, 15 images per sign, with the .jpg extension. Extensive experimental evaluations indicate that the proposed recognition system is less susceptible to displacement, scale, and rotation, and can detect symptoms at an accuracy of 95.62%. Conclusion: In this article, a new system was proposed for recognizing the alphabet letters in Persian sign language. This system consists of two main phases: the segmentation phase or the hand region identification and feature extraction phase. In the segmentation phase of the detection system, at first, the hand region is separated by an effective segmentation method from the original image. This method is based on the single Gaussian model in the color space of the YCbCr and the Bayes rule. Applying this method in hand segmentation, enables the system, to recognize the hand region well. In this method, after hand region detection, using the Sobel edge detector, the image edge is extracted. In the feature extraction phase, the radial distance model was used to obtain a one-dimensional function display of the hand region boundary and compute its feature vector. This model is based on the edge and image centroid and computes the centroid distance to the shape boundary as a function of the angle. Since this model by itself is sensitive to the scale variations and tilt, in order to normalize this method to tilt variations, the Fourier transformation was applied to feature vector computed by the radial model. Then the vector elements are divided by the largest vector element, thus, using this method the computed feature vector will be distinct for different images and will show less sensitivity to displacement, scale, and gradient. A database consist of 480 images of Persian sign symptoms was created to the overall accuracy evaluation of the system, the results show that the proposed recognition system can identify 32 letters of the Persian sign language alphabet with a detection rate 95.62%.

AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove

Sign language recognition, especially the sentence recognition, is of great significance for lowering the communication barrier between the hearing/speech impaired and the non-signers. The general glove solutions, which are employed to detect motions of our dexterous hands, only achieve recognizing discrete single gestures (i.e., numbers, letters, or words) instead of sentences, far from satisfying the meet of the signers’ daily communication. Here, we propose an artificial intelligence enabled sign language recognition and communication system comprising sensing gloves, deep learning block, and virtual reality interface. Non-segmentation and segmentation assisted deep learning model achieves the recognition of 50 words and 20 sentences. Significantly, the segmentation approach splits entire sentence signals into word units. Then the deep learning model recognizes all word elements and reversely reconstructs and recognizes sentences. Furthermore, new/never-seen sentences created by new-order word elements recombination can be recognized with an average correct rate of 86.67%. Finally, the sign language recognition results are projected into virtual space and translated into text and audio, allowing the remote and bidirectional communication between signers and non-signers.

A deep learning approach for Indian sign language gestures classification with different backgrounds

The sign language recognition system recently has drawn the attention of various researchers as there is no universal sign language, moreover, it consists of many patterns and postures. Many methods for extracting features and classifying sign language have been proposed in the literature, most of them are based on machine learning techniques. In this article, a deep learning method has been adopted by designing a Convolution Neural Network (CNN) model to extract the sign language features where for classification softmax layer is used. All alphabets in simple as well as complex backgrounds have been considered, where data is collected from 100 subjects in different lighting conditions. The effect of various optimization techniques (Adam, Sgdm, RMSProp), activation functions (ReLU and Leaky ReLU) for generalization ability is also observed. The proposed approach has succeeded in attaining the testing accuracy of 99.10%, 92.69%, and 95.95% on the Indian sign language dataset with simple, complex backgrounds and on a mixed background scenarios, respectively. The model is also tested on NUS dataset-I, NUS dataset-II, their combination, and achieved the accuracy of 100%, 95.95%, and 97.22% respectively.

Towards Hybrid Multimodal Manual and Non-Manual Arabic Sign Language Recognition: mArSL Database and Pilot Study

Sign languages are the main visual communication medium between hard-hearing people and their societies. Similar to spoken languages, they are not universal and vary from region to region, but they are relatively under-resourced. Arabic sign language (ArSL) is one of these languages that has attracted increasing attention in the research community. However, most of the existing and available works on sign language recognition systems focus on manual gestures, ignoring other non-manual information needed for other language signals such as facial expressions. One of the main challenges of not considering these modalities is the lack of suitable datasets. In this paper, we propose a new multi-modality ArSL dataset that integrates various types of modalities. It consists of 6748 video samples of fifty signs performed by four signers and collected using Kinect V2 sensors. This dataset will be freely available for researchers to develop and benchmark their techniques for further advancement of the field. In addition, we evaluated the fusion of spatial and temporal features of different modalities, manual and non-manual, for sign language recognition using the state-of-the-art deep learning techniques. This fusion boosted the accuracy of the recognition system at the signer-independent mode by 3.6% compared with manual gestures.

Applying deep neural networks for the automatic recognition of sign language words: A communication aid to deaf agriculturists

One of the major challenges that deaf people face in modern societal life is communication. For those engaged in agricultural jobs, efficiency at work and productivity are deeply related to the quality of deciphering the sign language used by the deaf farmers. Employing sign language interpreters is not a pragmatic solution to this problem. There comes the need for developing a reliable system for automatic sign language recognition (SLR). This paper reports a work on the recognition of hand gestures for the Indian sign language (ISL) words commonly used by deaf farmers. A hybrid deep learning model with convolutional long short term memory (LSTM) network has been exploited for gesture classification. The model has attained an average classification accuracy of 76.21% on the proposed dataset of ISL words from the agricultural domain.

Smart Communication Assistant for Deaf and Dumb People

Communication is the medium by which we can share our thoughts or convey the messages with other person. Nowadays we can give commands using voice recognition. But what if one absolutely cannot hear anything and eventually cannot speak. So the Sign Language is the main communicating tool for hearing impaired and mute people, and also to ensure an independent life for them, the automatic interpretation of sign language is an extensive research area. Sign language recognition (SLR) aims to interpret sign languages automatically by an application in order to help the deaf people to communicate with hearing society conveniently. Our aim is to design a system to help the Deaf and Dumb person to communicate with the rest of the world using sign language. With the use of image processing and artificial intelligence, many techniques and algorithms have been developed in this area. Every sign language recognition system is trained for recognizing the signs and converting them into required pattern. The proposed system aim to provide speech to speechless, in this paper we have introduced Sign Language Recognition using CNN for dynamic gestures to achieve faster results with high accuracy.

Sign Language to Text-Speech Translator Using Machine Learning

Communication with deaf and dumb people is quite difficult task for others. So, through sign language can communicate with deaf and mute persons but it is difficult for normal people to understand the sign language hence it creates a huge gap between them and it's uneasy to exchange their ideas, thoughts with others. This gap has existed for years in order to minimize this, new technologies should be emerged. Therefore, an interpreter is necessary which acts as a bridge between deaf-mute and others. This paper proposed system which is a sign language translator. The system used American Sign Language (ASL) dataset which is pre-processed based on threshold and intensity. This system recognizes sign language alphabet and by joining the letters it creates a sentence then it converts the text to speech. As the system is based on hand, hand gesture is used in sign language recognition system, for that the efficient hand tracking technique which is given by media pipe cross platform is used and it exactly detects the hand after that by using the ANN architecture the model has trained and which classifies the images. The system has achieved 74% accuracy and recognize almost all the letters. The system which also converts sign text to speech so that it will also helpful for blind people.

Smart, comfortable wearable system for recognizing Arabic Sign Language in real-time using IMUs and features-based fusion

Automatic sign language recognition systems have been presented in the literature to avail as a helpful device for deaf and mute individuals to incorporate them into the community without impediments. Gesture recognition is usually done by using vision-based techniques or hand-attached sensors that track hand and finger movements. This paper presents a sensors-based Arabic Sign Language recognition system. The proposed system depends on the information fusion from a 3-axis accelerometer and gyroscope Inertial Measurement Units (IMUs) sensors. The architecture of the proposed system consists of two main modules: a wearable device (glove), which includes sensors, a communication unit, and micro-controller unit, and a software module running on a mobile device. The proposed glove system utilizes six IMUs sensors where five of them placed at the fingers and the sixth sensor placed at the back of the palm. The collected data from sensors are transmitted to a mobile device through a Bluetooth low-energy wireless transmission for data analysis and gesture recognition. For each gesture, the beginning and endpoints of the significative gesture part are discovered automatically using the intensity of the gyroscope signal. The system employs a features-based sensor fusion that combines accelerometer and gyroscope time-domain and frequency-domain features. In a comprehensive set of experiments with seven participants, the system was trained and tested with 28 Arabic alphabets, and the best recognition accuracy achieved was 98.6% for the user-dependent data and 96% for the user-independent data under support vector machine classifier. The achieved results are outperforming existing systems that have been presented in recent decades for alphabet-level gesture recognition.

Indian Sign Language Recognition through Hybrid ConvNet-LSTM Networks

Dynamic hand gesture recognition is a challenging task of Human-Computer Interaction (HCI) and Computer Vision. The potential application areas of gesture recognition include sign language translation, video gaming, video surveillance, robotics, and gesture-controlled home appliances. In the proposed research, gesture recognition is applied to recognize sign language words from real-time videos. Classifying the actions from video sequences requires both spatial and temporal features. The proposed system handles the former by the Convolutional Neural Network (CNN), which is the core of several computer vision solutions and the latter by the Recurrent Neural Network (RNN), which is more efficient in handling the sequences of movements. Thus, the real-time Indian sign language (ISL) recognition system is developed using the hybrid CNN-RNN architecture. The system is trained with the proposed CasTalk-ISL dataset. The ultimate purpose of the presented research is to deploy a real-time sign language translator to break the hurdles present in the communication between hearing-impaired people and normal people. The developed system achieves 95.99% top-1 accuracy and 99.46% top-3 accuracy on the test dataset. The obtained results outperform the existing approaches using various deep models on different datasets.

Sign language recognition using Kinect sensor based on color stream and skeleton points

This paper presents a sign language recognition system based on color stream and skeleton points. Several approaches have been established to address sign language recognition problems. However, most of the previous approaches still have poor recognition accuracy. The proposed approach uses Kinect sensor based on color stream and skeleton points from the depth stream to improved recognition accuracy. Techniques within this approach use hand trajectories and hand shapes in combating sign recognition challenges. Therefore, for a particular sign a representative feature vector is extracted, which consists of hand trajectories and hand shapes. A sparse dictionary learning algorithm, Label Consistent K-SVD (LC-KSVD) is applied to obtain a discriminative dictionary. Based on that, the system was further developed to a new classification approach for better results. The proposed system was fairly evaluated based on 21 sign words including one-handed signs and two-handed signs. It was observed that the proposed system gets high recognition accuracy of 98.25%, and obtained an average accuracy of 95.34% for signer independent recognition.&#x0D; Keywords: Sign language, Color stream, Skeleton points, Kinect sensor, Discriminative dictionary.

ELM based two-handed dynamic Turkish Sign Language (TSL) word recognition

Hearing impaired individuals can easily overcome the barriers in communicating with other members of the society via computer technology. In this study, the recognition of dynamic words in Turkish Sign Language (TSL) with two hands was studied using the Leap Motion Controller (LMC) device. 50 dynamic words were determined considering the similarities and differences among themselves, and a dataset was created using 4 signers. For the system proposed in this paper, a comprehensive feature extraction process is executed. Applying feature selection algorithms and PCA, LDA and PCA+LDA dimension reduction methods to this dataset, new datasets with less dimension were obtained. For the first time, ELM architectures were operated as a classifier in a sign language recognition system. Recognition performance was tested with 5 different ELM networks and 2 classical classifiers and the results were compared. In addition, comparisons of classical and ELM based classifiers were presented. The 10-fold cross validation method was used to test the validity of the proposed system and the accuracy of the results obtained. Based on the results obtained by a comprehensive analysis, it was observed that the ML-KELM classifier maintains its performance rate and gives the highest performance rate. At the same time, it has been observed that ML-KELM classifier has a stable structure, which offers less user intervention.

Design and Development of a Humanoid Robot for Sign Language Interpretation

With an indisputable complexity of communication for hearing and speaking impaired people, most sign language recognition systems utilize virtual reality or onscreen robots. This paper presents the design and development of a special and low-cost humanoid robot that can perform as a sign language interpreter. To the best of our knowledge, this is the first endeavor to fabricate a humanoid robot for Bangla sign language (BdSL) and Medical signs interpretation. Considering the plethora of design criteria and balancing between rigidity and flexibility 3D models of the robotics parts are designed and 3D printed ensuring cost efficiency. With the help of modern fabrication technology, the robot is developed and assembled with proper actuators and circuitry. An image dataset is built comprising 950 images for BdSL recognition and made publicly available. We utilized the Recurrent neural network (RNN) and Convolutional neural network (CNN) for deep learning model establishment and feature extraction from video and image data. The developed humanoid robot has 43 Degrees of freedom (DoF) which includes two 15 DoF hands. It can imitate 16 BdSL alphabets in sign language, can capture a video or image input in real-time from the user, and recognize 10 medical signs and 38 alphabets of BdSL. The learning model for video-based medical sign recognition achieved 87.5% test accuracy. Image-based Bangla sign language recognition achieved an overall test accuracy of 98.19% in our dataset and 93.8% in another available dataset. Compared to the state-of-the-art robotic systems for sign language interpretation, our approach has achieved higher kinematic characteristics, remarkable results in sign recognition, and impressive competency in sign imitation; all at almost 10 times lower cost than the state-of-the-art systems. The results are evidence that our approach is efficient and suitable in helping hearing and speaking impaired people. Moreover, this work initiates a research scope that can be further extended for creating equal opportunities for the hearing and speaking impaired community.

Turkish sign language recognition based on multistream data fusion

Sign languages are nonverbal, visual languages that hearing- or speech-impaired people use for communication.Aside from hands, other communication channels such as body posture and facial expressions are also valuable insign languages. As a result of the fact that the gestures in sign languages vary across countries, the significance ofcommunication channels in each sign language also differs. In this study, representing the communication channels usedin Turkish sign language, a total of 8 different data streams-4 RGB, 3 pose, 1 optical flow-were analyzed. Inception3D was used for RGB and optical flow; and LSTM-RNN was used for pose data streams. Experiments were conductedby merging the data streams in different combinations, and then a sign language recognition system that merged themost suitable streams with the help of a multistream late fusion mechanism was proposed. Considering each data streamindividually, the accuracies of the RGB streams were between 28% and 79%; pose stream accuracies were between9% and 50%; and optical flow data accuracy was 78.5%. When these data streams were used in combination, the signlanguage recognition performance was higher in comparison to any of the data streams alone. The proposed sign languagerecognition system uses a multistream data fusion mechanism and gives an accuracy of 89.3% on BosphorusSign Generaldataset. The multistream data fusion mechanisms have a great potential for improving sign language recognition results.

Recognition of Amharic sign language with Amharic alphabet signs using ANN and SVM

Sign language is a natural language mostly used by persons with hearing- and speech-based impairments to communicate with other people. In these modern times, sign language guides are used to eliminate the communication gap between people having hearing impairments and those with or without hearing impairments; however, they are very limited in number. To solve this challenge, automatic sign language recognition systems are developed to better reduce the communication gap for people with hearing disabilities. This paper presents the development of an automatic Amharic sign language translator which translates Amharic alphabet signs into their corresponding text using digital image processing and machine learning algorithms. The proposed system has four major developmental stages which include preprocessing, segmentation, feature extraction and classification. A total number of thirty-four features were extracted from shape, motion and color of hand gestures to represent both the base and derived class of Amharic sign characters. Classification models were built using artificial neural network (ANN) and multi-class support vector machine (SVM). The results show that the recognition system is capable of recognizing the Amharic alphabet signs with an average accuracy of 80.82% and 98.06% using the ANN and SVM classifiers, respectively.

Sign Language Recognition using Principal Component Analysis and Support Vector Machine

The World Health Organization (WHO) estimates that over five percent of the world's population are hearing-impaired. One of the communication problems that often arise between deaf or speech impaired with normal people is the low level of knowledge and understanding of the deaf or speech impaired's normal sign language in their daily communication. To overcome this problem, we build a sign language recognition system, especially for the Indonesian language. The sign language system for Bahasa Indonesia, called Bisindo, is unique from the others. Our work utilizes two image processing algorithms for the pre-processing, namely the grayscale conversion and the histogram equalization. Subsequently, the principal component analysis (PCA) is employed for dimensional reduction and feature extraction. Finally, the support vector machine (SVM) is applied as the classifier. Results indicate that the use of the histogram equalization significantly enhances the accuracy of the recognition. Comprehensive experiments by applying different random seeds for testing data confirm that our method achieves 76.8% accuracy. Accordingly, a more robust method is still open to enhance the accuracy in sign language recognition.

Performance analysis of Otsu thresholding for sign language segmentation

Sign language recognition system generally consists of three main processes, which are segmentation, modelling, and classification. Image segmentation plays a crucial role as the initial step in sign language recognition. Despite the many sign language recognition system algorithms proposed in the literature and their well-understood usage, their performance analyses are relatively limited. As such, the main motivation of this paper is to critically analyse the feasibility of successful sign language segmentation under variation of dynamic scene parameters such as noise, hand size, and intensity difference between hand and background. The focus is on image thresholding using Otsu technique, since it is the most commonly used in initial process of sign language segmentation. The analysis of this work was developed based on Monte Carlo statistical method, which showed that the success of sign language segmentation depends on hand size, hand background intensity difference, and noise measurement. The result showed that the sign alphabets with handheld shape like A, E, I, M, N, S, and T is easier to segment, while sign alphabets with finger-extend shape like C, D, F, G, H, K, L, P, R, U, V, W, and Y is harder to segment. Experiment using real images demonstrate the capability of the conditions to correctly predict the outcome of sign language segmentation using Otsu technique. In conclusion, the success of sign language segmentation could be predicted beforehand with obtainable scene parameters.