Arabic Sign Language Recognition System Research Articles

Improved Arithmetic Optimization Algorithm with Transfer Learning based Arabic Sign Language Identification System

The Arabic sign language (ArSL) has witnessed ground-breaking research activities to identify hand gestures and signs through the deep learning (DL) model. SL is a unique communication tool that bridges the gaps between people with hearing impairment and ordinary people. The ArSL recognition system is of immense importance for different groups of people since it enables individuals with hearing impairment to communicate effectively. In SLs, signs are characterized by discrepancies in hand positions, shapes, motions, body parts, and facial expressions, posing a crucial threat to visual recognition in computer vision (CV). An automated sign detection technique needs two primary courses of action: the recognition of specific features and the classification of the input dataset. Previously, several approaches for detecting and classifying SLs had been proposed. In this study, an Improved Metaheuristics with Transfer Learning based Arabic Sign Language Identification System (IMTL-ArSL) method is developed. The primary objective of the IMTL-ArSL method is to detect and classify the existence of various signs in the Arabic language. In the IMTL-ArSL model, the bilateral filtering (BF) model is initially used for preprocessing. Besides, the Residual Network (ResNet50v2) model is applied for extracting features, and an improved arithmetic optimization algorithm (IAOA) model is utilized for hyperparameter tuning. Finally, a gated recurrent unit (GRU) network is exploited to identify sign languages. The empirical analysis of the IMTL-ArSL approach is tested using a benchmark dataset. The experimental values of the IMTL-ArSL approach portrayed a superior accuracy value of 93.87% over other techniques.

Integrated Mediapipe with a CNN Model for Arabic Sign Language Recognition

Deaf and dumb people struggle with communicating on a day-to-day basis. Current advancements in artificial intelligence (AI) have allowed this communication barrier to be removed. A letter recognition system for Arabic sign language (ArSL) has been developed as a result of this effort. The deep convolutional neural network (CNN) structure is used by the ArSL recognition system in order to process depth data and to improve the ability for hearing-impaired to communicate with others. In the proposed model, letters of the hand-sign alphabet and the Arabic alphabet would be recognized and identified automatically based on user input. The proposed model should be able to identify ArSL with a rate of accuracy of 97.1%. In order to test our approach, we carried out a comparative study and discovered that it is able to differentiate between static indications with a higher level of accuracy than prior studies had achieved using the same dataset.

Arabic Sign Language Recognition System: Using an Image-Based hand Gesture Detection Method to help Deaf and Dump Children to Engage in Education

Hand gestures help individuals communicate in daily life. Sign languages, however, employ hand motions extensively. People with hearing impairments use sign language. Pattern recognition in computer vision may be used to interpret and translate Arabic Sign Language (ArSL) for deaf and dumb persons using image processing-based software systems. Arab hearing-impaired and deaf persons communicate with Arabic Sign Language Recognition (ArSLR).We propose an image-based ArSLR system employing gesture recognition to enable people with disabilities to engage with the outside world. This work proposes the design and development of an automated Arabic sign language-to-Arabic text translation system. This study extracts stem Arabic words and differentiates related terms. Hand motions capture alphabet letters in the proposed approach. Image isolation occurs after acquisition. Extraction, conclusion, and hand sign pattern characteristics comprise the third step. The fourth phase classifies hand gestures using a powerful classification algorithm. Finally, the sign language letter is interpreted and translated to the Arabic character.

A Deep Learning based Approach for Recognition of Arabic Sign Language Letters

No one can deny that the deaf-mute community has communication problems in daily life. Advances in artificial intelligence over the past few years have broken through this communication barrier. The principal objective of this work is creating an Arabic Sign Language Recognition system (ArSLR) for recognizing Arabic letters. The ArSLR system is developed using our image pre-processing method to extract the exact position of the hand and we proposed architecture of the Deep Convolutional Neural Network (CNN) using depth data. The goal is to make it easier for people who have hearing problems to interact with normal people. Based on user input, our method will detect and recognize hand-sign letters of the Arabic alphabet automatically. The suggested model is anticipated to deliver encouraging results in the recognition of Arabic sign language with an accuracy score of 97,07%. We conducted a comparison study in order to evaluate proposed system, the obtained results demonstrated that this method is able to recognize static signs with greater accuracy than the accuracy obtained by similar other studies on the same dataset used.

Smart, comfortable wearable system for recognizing Arabic Sign Language in real-time using IMUs and features-based fusion

Automatic sign language recognition systems have been presented in the literature to avail as a helpful device for deaf and mute individuals to incorporate them into the community without impediments. Gesture recognition is usually done by using vision-based techniques or hand-attached sensors that track hand and finger movements. This paper presents a sensors-based Arabic Sign Language recognition system. The proposed system depends on the information fusion from a 3-axis accelerometer and gyroscope Inertial Measurement Units (IMUs) sensors. The architecture of the proposed system consists of two main modules: a wearable device (glove), which includes sensors, a communication unit, and micro-controller unit, and a software module running on a mobile device. The proposed glove system utilizes six IMUs sensors where five of them placed at the fingers and the sixth sensor placed at the back of the palm. The collected data from sensors are transmitted to a mobile device through a Bluetooth low-energy wireless transmission for data analysis and gesture recognition. For each gesture, the beginning and endpoints of the significative gesture part are discovered automatically using the intensity of the gyroscope signal. The system employs a features-based sensor fusion that combines accelerometer and gyroscope time-domain and frequency-domain features. In a comprehensive set of experiments with seven participants, the system was trained and tested with 28 Arabic alphabets, and the best recognition accuracy achieved was 98.6% for the user-dependent data and 96% for the user-independent data under support vector machine classifier. The achieved results are outperforming existing systems that have been presented in recent decades for alphabet-level gesture recognition.

Arabic Sign Language Recognition using Faster R-CNN

Deafness does not restrict its negative effect on the person’s hearing, but rather on all aspect of their daily life. Moreover, hearing people aggravated the issue through their reluctance to learn sign language. This resulted in a constant need for human translators to assist deaf person which represents a real obstacle for their social life. Therefore, automatic sign language translation emerged as an urgent need for the community. The availability and the widespread use of mobile phones equipped with digital cameras promoted the design of image-based Arabic Sign Language (ArSL) recognition systems. In this work, we introduce a new ArSL recognition system that is able to localize and recognize the alphabet of the Arabic sign language using a Faster Region-based Convolutional Neural Network (R-CNN). Specifically, faster R-CNN is designed to extract and map the image features, and learn the position of the hand in a given image. Additionally, the proposed approach alleviates both challenges; the choice of the relevant features used to encode the sign visual descriptors, and the segmentation task intended to determine the hand region. For the implementation and the assessment of the proposed Faster R-CNN based sign recognition system, we exploited VGG-16 and ResNet-18 models, and we collected a real ArSL image dataset. The proposed approach yielded 93% accuracy and confirmed the robustness of the proposed model against drastic background variations in the captured scenes.

An Automatic Arabic Sign Language Recognition System based on Deep CNN: An Assistive System for the Deaf and Hard of Hearing

An Automatic Arabic Sign Language Recognition System based on Deep CNN: An Assistive System for the Deaf and Hard of Hearing

An Intelligent Arabic Sign Language Recognition System Using a Pair of LMCs With GMM Based Classification

In this paper, we propose a dual leap motion controllers (LMC)-based Arabic sign language recognition system. More specifically, we introduce the concept of using both front and side LMCs to cater for the challenges of finger occlusions and missing data. For feature extraction, an optimum set of geometric features is selected from both controllers, while for classification, we used both a Bayesian approach with a Gaussian mixture model (GMM) and a simple linear discriminant analysis (LDA) approach. To combine the information from the two LMCs, we introduce an evidence-based fusion approach; namely, the Dempster–Shafer (DS) theory of evidence. Data were collected from two native adult signers, for 100 isolated Arabic dynamic signs. Ten observations were collected for each of the signs. The proposed framework uses an intelligent strategy to handle the case of missing data from one or both controllers. A recognition accuracy of about 92% was achieved. The proposed system outperforms state-of-the-art glove-based systems and single-sensor-based techniques.

Supervised learning classifiers for Arabic gestures recognition using Kinect V2

Human–Computer Interaction (HCI) refers to the interaction between the computers and human. One of the most important applications of (HCI) is sign language recognition. Several research works aimed to interpret and translate the sign language to a spoken language to help the hear impaired persons in integrating with communities. Sign language is the main way of communication for the deaf persons and hearing impaired which enable them to communicate with their societies and between each other’s. According to the World Health Organization, there are 466 million hearing loss people (i.e. 5% of the world population), 432 million or 93% of them are adults and 34 million or 17% of them are children. The hearing-impaired persons often need to the same level of mental capability as the normal persons. The main problem is the most of the hearing persons who cannot understand the sign language and most of hearing impaired cannot read or write our spoken language, this represents a barrier between the deaf persons and their societies so developing an automatic sign language recognition system is very necessary. This research introduces dynamic Arabic Sign Language recognition system using Microsoft Kinect. The recognition depends on using two machine learning algorithms (a) Decision Tree and (b) Bayesian Network then applied Ada-Boosting technique to enhance the recognition of the system, we compared the results with two direct matching techniques: (a) Dynamic Time Wrapping and Hidden Markov Model the system was applied on 42 Arabic gestures related to medical field. The experimental results showed that the proposed system recognition rate reached 91.18% for Decision Tree classifier, 92.50% for Bayesian classifier and 93.7% after applying Ada-Boosting.

Image based Arabic Sign Language Recognition System

Through history, humans have used many ways of communication such as gesturing, sounds, drawing, writing, and speaking. However, deaf and speaking impaired people cannot use speaking to communicate with others, which may give them a sense of isolation within their societies. For those individuals, sign language is their principal way to communicate. However, most people (who can hear) do not know the sign language. In this paper, we aim to automatically recognize Arabic Sign Language (ArSL) alphabets using an image-based methodology. More specifically, various visual descriptors are investigated to build an accurate ArSL alphabet recognizer. The extracted visual descriptors are conveyed to One-Versus-All Support Vector Machine (SVM). The analysis of the results shows that Histograms of Oriented Gradients (HOG) descriptor outperforms the other considered descriptors. Thus, the ArSL gesture models that are learned by One-Versus-All SVM using HOG descriptors are deployed in the proposed system.

The Performance of Individual and Ensemble Classifiers for an Arabic Sign Language Recognition System

The objective of this paper is to compare different classifiers’ recognition accuracy for the 28 Arabic alphabet letters gestured by participants as Sign Language and captured by two depth sensors. The accuracy results of three individual classifiers: (1) the support vector machine (SVM), (2) random forest (RF), and (3) nearest neighbour (kNN), using the original gestured dataset were compared with the accuracy results using an ensemble of the results of each classifier, as recommended by the literature. SVM produced higher overall accuracy when running as an individual classifier regardless of the number of observations for each letter. However, for letters with fewer than 65 observations each, which created a far smaller dataset, RF had higher accuracy than SVM did when using the ensemble approach. Although RF produced higher accuracy results for classes with limited class observation data, the difference between the accuracy results of RF in phase 2 and SVM in phase 1 was negligible. The researchers conclude that such a difference does not warrant using the ensemble approach for this experiment, which adds more processing complexity without a significant increase in accuracy.

Glove-Based Continuous Arabic Sign Language Recognition in User-Dependent Mode

In this paper, we propose a glove-based Arabic sign language recognition system using a novel technique for sequential data classification. We compile a sensor-based dataset of 40 sentences using an 80-word lexicon. In the dataset, hand movements are captured using two DG5-VHand data gloves. Data labeling is performed using a camera to synchronize hand movements with their corresponding sign language words. Low-complexity preprocessing and feature extraction techniques are applied to capture and emphasize the temporal dependence of the data. Subsequently, a Modified k-Nearest Neighbor (MKNN) approach is used for classification. The proposed MKNN makes use of the context of feature vectors for the purpose of accurate classification. The proposed solution achieved a sentence recognition rate of 98.9%. The results are compared against an existing vision-based approach that uses the same set of sentences. The proposed solution is superior in terms of classification rates while eliminating restrictions of vision-based systems.

Arabic sign language continuous sentences recognition using PCNN and graph matching

Many previous researchers have tried developing sign languages recognition systems in general and Arabic sign language specifically. They succeeded to achieve acceptable results for isolated gestures level, but none of them investigated the recognition of connected sequence of gestures. This paper focuses on how to recognize real-time connected sequence of gestures using graph-matching technique, also how the continuous input gestures are segmented and classified. Graphs are a general and powerful data structure useful for the representation of various objects and concepts. This work is a component of a real-time Arabic Sign Language Recognition system that applied pulse-coupled neural network for static posture recognition in its first phase. This work can be adapted and applied to different sign languages and other recognition problems.

A signer-independent Arabic Sign Language recognition system using face detection, geometric features, and a Hidden Markov Model

In this paper, we propose an image-based system for Arabic Sign Language (ArSL) recognition. The algorithm starts by detecting the face of the signer using a Gaussian skin color model. The centroid of the detected face is then used as a reference point for tracking the hands’ movements. The hands regions are segmented using a region growing algorithm assuming the signer wears a yellow and an orange colored gloves. From the segmented hands regions, an optimal set of features is extracted. To represent the time varying feature patterns, a Hidden Markov Model (HMM) is then used. Before using HMM in testing, the number of states and the number of Gaussian mixtures are optimized. The proposed system was implemented for both signer dependent and signer independent conditions. The experimental results show that an accuracy of more than 95% can be achieved with a large database of 300 signs. The results outperform previous work on ArSL mainly restricted to small vocabulary size.