Arabic Sign Language Recognition Research Articles

Isolated Arabic Sign Language Recognition Using a Transformer-based Model and Landmark Keypoints

Pose-based approaches for sign language recognition provide light-weight and fast models that can be adopted in real-time applications. This article presents a framework for isolated Arabic sign language recognition using hand and face keypoints. We employed MediaPipe pose estimator for extracting the keypoints of sign gestures in the video stream. Using the extracted keypoints, three models were proposed for sign language recognition: Long-Term Short Memory, Temporal Convolution Networks, and Transformer-based models. Moreover, we investigated the importance of non-manual features for sign language recognition systems and the obtained results showed that combining hand and face keypoints boosted the recognition accuracy by around 4% compared with only hand keypoints. The proposed models were evaluated on Arabic and Argentinian sign languages. Using the KArSL-100 dataset, the proposed pose-based Transformer achieved the highest accuracy of 99.74% and 68.2% in signer-dependent and -independent modes, respectively. Additionally, the Transformer was evaluated on the LSA64 dataset and obtained an accuracy of 98.25% and 91.09% in signer-dependent and -independent modes, respectively. Consequently, the pose-based Transformer outperformed the state-of-the-art techniques on both datasets using keypoints from the signer’s hands and face.

Recent Progress in Arabic Sign Language Recognition: Utilizing Convolutional Neural Networks (CNN)

The advancement of assistive communication technology for the deaf and hard-of-hearing community is an area of significant research interest. In this study, we present a Convolutional Neural Network (CNN) model tailored for the recognition of Arabic Sign Language (ArSL). Our model incorporates a meticulous preprocessing pipeline that transforms input images through grayscale conversion, Gaussian blur, histogram equalization, and resizing to standardize input data and enhance feature visibility. Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) are employed for feature extraction to retain critical discriminative information while reducing dimensionality. The proposed CNN architecture leverages a blend of one-dimensional convolutional layers, max pooling, Leaky ReLU activation functions, and Long Short-Term Memory (LSTM) layers to efficiently capture both spatial and temporal patterns within the data. Our experiments on two separate datasets—one consisting of images and the other of videos—demonstrate exceptional recognition rates of 99.7% and 99.9%, respectively. These results significantly surpass the performance of existing models referenced in the literature. This paper discusses the methodologies, architectural considerations, and the training approach of the proposed model, alongside a comparative analysis of its performance against previous studies. The research outcomes suggest that our model not only sets a new benchmark in sign language recognition but also offers a promising foundation for the development of real-time, assistive sign language translation tools. The potential applications of such technology could greatly enhance communication accessibility, fostering greater inclusion for individuals who rely on sign language as their primary mode of communication. Future work will aim to expand the model's capabilities to more diverse datasets and investigate its deployment in practical, everyday scenarios to bridge the communication gap for the deaf and hard of hearing community.

Vision Transformers and Transfer Learning Approaches for Arabic Sign Language Recognition

Sign languages are complex, but there are ongoing research efforts in engineering and data science to recognize, understand, and utilize them in real-time applications. Arabic sign language recognition (ArSL) has been examined and applied using various traditional and intelligent methods. However, there have been limited attempts to enhance this process by utilizing pretrained models and large-sized vision transformers designed for image classification tasks. This study aimed to create robust transfer learning models trained on a dataset of 54,049 images depicting 32 alphabets from an ArSL dataset. The goal was to accurately classify these images into their corresponding Arabic alphabets. This study included two methodological parts. The first one was the transfer learning approach, wherein we utilized various pretrained models namely MobileNet, Xception, Inception, InceptionResNet, DenseNet, and BiT, and two vision transformers namely ViT, and Swin. We evaluated different variants from base-sized to large-sized pretrained models and vision transformers with weights initialized from the ImageNet dataset or otherwise randomly. The second part was the deep learning approach using convolutional neural networks (CNNs), wherein several CNN architectures were trained from scratch to be compared with the transfer learning approach. The proposed methods were evaluated using the accuracy, AUC, precision, recall, F1 and loss metrics. The transfer learning approach consistently performed well on the ArSL dataset and outperformed other CNN models. ResNet and InceptionResNet obtained a comparably high performance of 98%. By combining the concepts of transformer-based architecture and pretraining, ViT and Swin leveraged the strengths of both architectures and reduced the number of parameters required for training, making them more efficient and stable than other models and existing studies for ArSL classification. This demonstrates the effectiveness and robustness of using transfer learning with vision transformers for sign language recognition for other low-resourced languages.

Integrated Mediapipe with a CNN Model for Arabic Sign Language Recognition

Deaf and dumb people struggle with communicating on a day-to-day basis. Current advancements in artificial intelligence (AI) have allowed this communication barrier to be removed. A letter recognition system for Arabic sign language (ArSL) has been developed as a result of this effort. The deep convolutional neural network (CNN) structure is used by the ArSL recognition system in order to process depth data and to improve the ability for hearing-impaired to communicate with others. In the proposed model, letters of the hand-sign alphabet and the Arabic alphabet would be recognized and identified automatically based on user input. The proposed model should be able to identify ArSL with a rate of accuracy of 97.1%. In order to test our approach, we carried out a comparative study and discovered that it is able to differentiate between static indications with a higher level of accuracy than prior studies had achieved using the same dataset.

Signer-Independent Arabic Sign Language Recognition System Using Deep Learning Model.

Every one of us has a unique manner of communicating to explore the world, and such communication helps to interpret life. Sign language is the popular language of communication for hearing and speech-disabled people. When a sign language user interacts with a non-sign language user, it becomes difficult for a signer to express themselves to another person. A sign language recognition system can help a signer to interpret the sign of a non-sign language user. This study presents a sign language recognition system that is capable of recognizing Arabic Sign Language from recorded RGB videos. To achieve this, two datasets were considered, such as (1) the raw dataset and (2) the face-hand region-based segmented dataset produced from the raw dataset. Moreover, operational layer-based multi-layer perceptron "SelfMLP" is proposed in this study to build CNN-LSTM-SelfMLP models for Arabic Sign Language recognition. MobileNetV2 and ResNet18-based CNN backbones and three SelfMLPs were used to construct six different models of CNN-LSTM-SelfMLP architecture for performance comparison of Arabic Sign Language recognition. This study examined the signer-independent mode to deal with real-time application circumstances. As a result, MobileNetV2-LSTM-SelfMLP on the segmented dataset achieved the best accuracy of 87.69% with 88.57% precision, 87.69% recall, 87.72% F1 score, and 99.75% specificity. Overall, face-hand region-based segmentation and SelfMLP-infused MobileNetV2-LSTM-SelfMLP surpassed the previous findings on Arabic Sign Language recognition by 10.970% accuracy.

A Deep Learning based Approach for Recognition of Arabic Sign Language Letters

No one can deny that the deaf-mute community has communication problems in daily life. Advances in artificial intelligence over the past few years have broken through this communication barrier. The principal objective of this work is creating an Arabic Sign Language Recognition system (ArSLR) for recognizing Arabic letters. The ArSLR system is developed using our image pre-processing method to extract the exact position of the hand and we proposed architecture of the Deep Convolutional Neural Network (CNN) using depth data. The goal is to make it easier for people who have hearing problems to interact with normal people. Based on user input, our method will detect and recognize hand-sign letters of the Arabic alphabet automatically. The suggested model is anticipated to deliver encouraging results in the recognition of Arabic sign language with an accuracy score of 97,07%. We conducted a comparison study in order to evaluate proposed system, the obtained results demonstrated that this method is able to recognize static signs with greater accuracy than the accuracy obtained by similar other studies on the same dataset used.

Arabic Sign Language Recognition System: Using an Image-Based hand Gesture Detection Method to help Deaf and Dump Children to Engage in Education

Hand gestures help individuals communicate in daily life. Sign languages, however, employ hand motions extensively. People with hearing impairments use sign language. Pattern recognition in computer vision may be used to interpret and translate Arabic Sign Language (ArSL) for deaf and dumb persons using image processing-based software systems. Arab hearing-impaired and deaf persons communicate with Arabic Sign Language Recognition (ArSLR).We propose an image-based ArSLR system employing gesture recognition to enable people with disabilities to engage with the outside world. This work proposes the design and development of an automated Arabic sign language-to-Arabic text translation system. This study extracts stem Arabic words and differentiates related terms. Hand motions capture alphabet letters in the proposed approach. Image isolation occurs after acquisition. Extraction, conclusion, and hand sign pattern characteristics comprise the third step. The fourth phase classifies hand gestures using a powerful classification algorithm. Finally, the sign language letter is interpreted and translated to the Arabic character.

Two-Stage Deep Learning Solution for Continuous Arabic Sign Language Recognition Using Word Count Prediction and Motion Images

Two-Stage Deep Learning Solution for Continuous Arabic Sign Language Recognition Using Word Count Prediction and Motion Images

Correction to: Arabic Sign Language Recognition Using Convolutional Neural Network and MobileNet

Correction to: Arabic Sign Language Recognition Using Convolutional Neural Network and MobileNet

Atom Search Optimization with Deep Learning Enabled Arabic Sign Language Recognition for Speaking and Hearing Disability Persons.

Sign language has played a crucial role in the lives of impaired people having hearing and speaking disabilities. They can send messages via hand gesture movement. Arabic Sign Language (ASL) recognition is a very difficult task because of its high complexity and the increasing intraclass similarity. Sign language may be utilized for the communication of sentences, letters, or words using diverse signs of the hands. Such communication helps to bridge the communication gap between people with hearing impairment and other people and also makes it easy for people with hearing impairment to express their opinions. Recently, a large number of studies have been ongoing in developing a system that is capable of classifying signs of dissimilar sign languages into the given class. Therefore, this study designs an atom search optimization with a deep convolutional autoencoder-enabled sign language recognition (ASODCAE-SLR) model for speaking and hearing disabled persons. The presented ASODCAE-SLR technique mainly aims to assist the communication of speaking and hearing disabled persons via the SLR process. To accomplish this, the ASODCAE-SLR technique initially pre-processes the input frames by a weighted average filtering approach. In addition, the ASODCAE-SLR technique employs a capsule network (CapsNet) feature extractor to produce a collection of feature vectors. For the recognition of sign language, the DCAE model is exploited in the study. At the final stage, the ASO algorithm is utilized as a hyperparameter optimizer which in turn increases the efficacy of the DCAE model. The experimental validation of the ASODCAE-SLR model is tested using the Arabic Sign Language dataset. The simulation analysis exhibit the enhanced performance of the ASODCAE-SLR model compared to existing models.

Retraction Note to: A study on Arabic sign language recognition for differently abled using advanced machine learning classifiers

Retraction Note to: A study on Arabic sign language recognition for differently abled using advanced machine learning classifiers

Automatic recognition of Arabic alphabets sign language using deep learning

<span>Technological advancements are helping people with special needs overcome many communications’ obstacles. Deep learning and computer vision models are innovative leaps nowadays in facilitating unprecedented tasks in human interactions. The Arabic language is always a rich research area. In this paper, different deep learning models were applied to test the accuracy and efficiency obtained in automatic Arabic sign language recognition. In this paper, we provide a novel framework for the automatic detection of Arabic sign language, based on transfer learning applied on popular deep learning models for image processing. Specifically, by training AlexNet, VGGNet and GoogleNet/Inception models, along with testing the efficiency of shallow learning approaches based on support vector machine (SVM) and nearest neighbors algorithms as baselines. As a result, we propose a novel approach for the automatic recognition of Arabic alphabets in sign language based on VGGNet architecture which outperformed the other trained models. The proposed model is set to present promising results in recognizing Arabic sign language with an accuracy score of 97%. The suggested models are tested against a recent fully-labeled dataset of Arabic sign language images. The dataset contains 54,049 images, which is considered the first large and comprehensive real dataset of Arabic sign language to the furthest we know.</span>

Dynamic hand gesture recognition of Arabic sign language by using deep convolutional neural networks

<p>In computer vision, one of the most difficult problems is human gestures in videos recognition Because of certain irrelevant environmental variables. This issue has been solved by using single deep networks to learn spatiotemporal characteristics from video data, and this approach is still insufficient to handle both problems at the same time. As a result, the researchers fused various models to allow for the effective collection of important shape information as well as precise spatiotemporal variation of gestures. In this study, we collected the dynamic dataset for twenty meaningful words of Arabic sign language (ArSL) using a Microsoft Kinect v2 camera. The recorded data included 7350 red, green, and blue (RGB) videos and 7350 depth videos. We proposed four deep neural networks models using 2D and 3D convolutional neural network (CNN) to cover all feature extraction methods and then passing these features to the recurrent neural network (RNN) for sequence classification. Long short-term memory (LSTM) and gated recurrent unit (GRU) are two types of using RNN. Also, the research included evaluation fusion techniques for several types of multiple models. The experiment results show the best multi-model for the dynamic dataset of the ArSL recognition achieved 100% accuracy.</p>

Dynamic Sign Language Recognition Based on Real-Time Videos

Sign language is the main communication tool for the deaf and hard of hearing. Deaf people cannot interact with others without a sign language interpreter. Accordingly, sign language recognition automation has become an important application in artificial intelligence and deep learning. Specifically, the recognition of Arabic sign language has been studied using many smart and traditional methods. This research provides a system to recognize dynamic Saudi sign language based on real time videos to solve this problem. We constructed a dataset for Saudi sign language in terms of videos in the proposed system. The dataset was then used to train a deep learning model using convolutional long short-term memory (convLSTM) to recognize the dynamic signs. Implementing such a system provides a platform for deaf people to interact with the rest of the world without an interpreter to reduce deaf isolation in society.

Arabic Sign Language Recognition using Lightweight CNN-based Architecture

Communication is a critical skill for humans. People who have been deprived from communicating through words like the rest of humans, usually use sign language. For sign language, the main signs features are the handshape, the location, the movement, the orientation and the non-manual component. The vast spread of mobile phones presents an opportunity for hearing-disabled people to engage more into their communities. Designing and implementing a novel Arabic Sign Language (ArSL) recognition system would significantly affect their quality of life. Deep learning models are usually heavy for mobile phones. The more layers a neural network has, the heavier it is. However, typical deep neural network necessitates a large number of layers to attain adequate classification performance. This project aims at addressing the Arabic Sign Language recognition problem and ensuring a trade-off between optimizing the classification performance and scaling down the architecture of the deep network to reduce the computational cost. Specifically, we adapted Efficient Network (EfficientNet) models and generated lightweight deep learning models to classify Arabic Sign Language gestures. Furthermore, a real dataset collected by many different signers to perform hand gestures for thirty different Arabic alphabets. Then, an appropriate performance metrics used in order to assess the classification outcomes obtained by the proposed lightweight models. Besides, preprocessing and data augmentation techniques were investigated to enhance the models generalization. The best results were obtained using the EfficientNet-Lite 0 architecture and the Label smooth as loss function. Our model achieved 94% and proved to be effective against background variations.

Arabic Sign Language Recognition System for Alphabets Using Machine Learning Techniques

In recent years, the role of pattern recognition in systems based on human computer interaction (HCI) has spread in terms of computer vision applications and machine learning, and one of the most important of these applications is to recognize the hand gestures used in dealing with deaf people, in particular to recognize the dashed letters in surahs of the Quran. In this paper, we suggest an Arabic Alphabet Sign Language Recognition System (AArSLRS) using the vision-based approach. The proposed system consists of four stages: the stage of data processing, preprocessing of data, feature extraction, and classification. The system deals with three types of datasets: data dealing with bare hands and a dark background, data dealing with bare hands, but with a light background, and data dealing with hands wearing dark colored gloves. AArSLRS begins with obtaining an image of the alphabet gestures, then revealing the hand from the image and isolating it from the background using one of the proposed methods, after which the hand features are extracted according to the selection method used to extract them. Regarding the classification process in this system, we have used supervised learning techniques for the classification of 28-letter Arabic alphabet using 9240 images. We focused on the classification for 14 alphabetic letters that represent the first Quran surahs in the Quranic sign language (QSL). AArSLRS achieved an accuracy of 99.5% for the K-Nearest Neighbor (KNN) classifier.

Static hand gesture recognition of Arabic sign language by using deep CNNs

An Arabic sign language recognition using two concatenated deep convolution neural network models DenseNet121 & VGG16 is presented. The pre-trained models are fed with images, and then the system can automatically recognize the Arabic sign language. To evaluate the performance of concatenated two models in the Arabic sign language recognition, the red-green-blue (RGB) images for various static signs are collected in a dataset. The dataset comprises 220,000 images for 44 categories: 32 letters, 11 numbers (0:10), and 1 for none. For each of the static signs, there are 5000 images collected from different volunteers. The pre-trained models were used and trained on prepared Arabic sign language data. These models were used after some modification. Also, an attempt has been made to adopt two models from the previously trained models, where they are trained in parallel deep feature extractions. Then they are combined and prepared for the classification stage. The results demonstrate the comparison between the performance of the single model and multi-model. It appears that most of the multi-model is better in feature extraction and classification than the single models. And also show that when depending on the total number of incorrect recognize sign image in training, validation and testing dataset, the best convolutional neural networks (CNN) model in feature extraction and classification Arabic sign language is the DenseNet121 for a single model using and DenseNet121 & VGG16 for multi-model using.

Isolated Video-Based Arabic Sign Language Recognition Using Convolutional and Recursive Neural Networks

Isolated Video-Based Arabic Sign Language Recognition Using Convolutional and Recursive Neural Networks

Smart, comfortable wearable system for recognizing Arabic Sign Language in real-time using IMUs and features-based fusion

Automatic sign language recognition systems have been presented in the literature to avail as a helpful device for deaf and mute individuals to incorporate them into the community without impediments. Gesture recognition is usually done by using vision-based techniques or hand-attached sensors that track hand and finger movements. This paper presents a sensors-based Arabic Sign Language recognition system. The proposed system depends on the information fusion from a 3-axis accelerometer and gyroscope Inertial Measurement Units (IMUs) sensors. The architecture of the proposed system consists of two main modules: a wearable device (glove), which includes sensors, a communication unit, and micro-controller unit, and a software module running on a mobile device. The proposed glove system utilizes six IMUs sensors where five of them placed at the fingers and the sixth sensor placed at the back of the palm. The collected data from sensors are transmitted to a mobile device through a Bluetooth low-energy wireless transmission for data analysis and gesture recognition. For each gesture, the beginning and endpoints of the significative gesture part are discovered automatically using the intensity of the gyroscope signal. The system employs a features-based sensor fusion that combines accelerometer and gyroscope time-domain and frequency-domain features. In a comprehensive set of experiments with seven participants, the system was trained and tested with 28 Arabic alphabets, and the best recognition accuracy achieved was 98.6% for the user-dependent data and 96% for the user-independent data under support vector machine classifier. The achieved results are outperforming existing systems that have been presented in recent decades for alphabet-level gesture recognition.

KArSL

Sign language is the major means of communication for the deaf community. It uses body language and gestures such as hand shapes, lib patterns, and facial expressions to convey a message. Sign language is geography-specific, as it differs from one country to another. Arabic Sign language is used in all Arab countries. The availability of a comprehensive benchmarking database for ArSL is one of the challenges of the automatic recognition of Arabic Sign language. This article introduces KArSL database for ArSL, consisting of 502 signs that cover 11 chapters of ArSL dictionary. Signs in KArSL database are performed by three professional signers, and each sign is repeated 50 times by each signer. The database is recorded using state-of-art multi-modal Microsoft Kinect V2. We also propose three approaches for sign language recognition using this database. The proposed systems are Hidden Markov Models, deep learning images’ classification model applied on an image composed of shots of the video of the sign, and attention-based deep learning captioning system. Recognition accuracies of these systems indicate their suitability for such a large number of Arabic signs. The techniques are also tested on a publicly available database. KArSL database will be made freely available for interested researchers.