Gesture Images Research Articles

The ASL Dataset for Real-Time Recognition and Integration with LLM Services

This study aims to investigate the impact of hand gesture recognition techniques on the efficiency of American Sign Language (ASL) interpretation, addressing a critical gap in the existing literature. The research seeks new insights into the challenges of automated sign language recognition, contributing to a deeper understanding of accessibility in communication for the deaf and hard-of-hearing community. The study employs a quantitative approach, using a dataset comprising hand gesture images representing the static letters of the ASL alphabet collected from multiple users. Data were collected from various individuals to ensure diversity and analyzed using machine learning models to evaluate their effectiveness in recognizing ASL signs. The results reveal that the machine learning models implemented achieved a high accuracy rate in recognizing hand gestures, indicating that person-specific variations do not significantly hinder performance. These findings provide evidence that the proposed dataset and methodologies can improve the reliability of sign language recognition systems, offering significant implications for the development of more inclusive communication technologies. This research offers a novel perspective on sign language recognition, providing valuable insight that extends the current understanding of gesture-based communication systems. The study’s findings contribute to advancements in accessibility technologies, highlighting areas for future research and practical applications in improving communication for the Deaf and hard of hearing community.

Hand Gesture Enabled Operation for Computer Using Deep Learning

This project aims to develop a system that detects hand gestures and prints corresponding instructions for computer operators. The system utilizes Deep learning algorithms CNN and mediapipe to recognize hand gestures, and then prints instructions for the operator to perform specific computer operations.Operations like Notepad Open,Shutdown,Volume Up,Volume Down etc.The system will consist of a webcam, a Deep learning model, and a printer. The webcam will capture images of hand gestures, which will be processed using the Deep learning model to identify the specific gesture. The system will then print the corresponding instruction on the printer. The system will be designed to be user-friendly and adaptable to different environments and hardware configurations.The system aims to improve efficiency, reduce errors, and enhance accessibility for computer operators. The proposed system consists of a hand gesture detection module, instruction printing module, and computer operation module. The system is trained using a dataset of hand gestures and corresponding computer operations. Experimental results show that the system achieves high accuracy in gesture recognition and instruction printing. The system has potential applications in various fields, including gaming, education, and healthcare. Key Words: Hand gesture detection, Deep learning, instruction printing, computer operation, accessibility, Mediapipe, CNN.

CNN Algorithm with SIFT to Enhance the Arabic Sign Language Recognition

Sign language is used as a primary means of communication by millions of people who suffer from hearing problems. The unhearing people used visual language to interact with each other, Represented in sign language. There are features that the hearing impaired use to understand each other, which are difficult for normal people to understand. Therefore, deaf people will struggle to interact with society. This research aims to introduce a system for recognizing hand gestures in Arabic Sign Language (ArSL) through training the Convolutional Neural Network (CNN) on the images of ArSL gestures launched by the University of Prince Mohammad Bin Fahd, Saudi Arabia. A Scale Invariant Feature Transform (SIFT) algorithm is used for creating the feature vectors that contain shape, finger position, size, center points of palm, and hand margin by extracting the Important features for images of ArSL and transforming them to points of the vector. The accuracy of the proposed system is 97% using the SIFT with CNN, and equal to 94.8% nearly without SIFT. Finally, the proposed system was tried and tested on a group of persons and its effectiveness was proven after considering their observations.

A Sign Language Recognition using Improved Grey Wolf Optimization based neural networks

Sign language was developed to enable the deaf and hard of hearing community to communicate with society and convey information. It is the primary means by which they can interact with each other, as well as with the general population. The automatic gesture recognition system described in this paper uses Gray Wolf Optimization (GWO) to identify the most relevant features from a set of gesture images, with the aim of improving model performance. The main steps of the system are explained, including image preprocessing, feature extraction, GWO-based feature selection, classifier training, and evaluation. The proposed model achieved an accuracy of 99.9%.

The left side of gestures: left perceptual bias for meaningless hand gestures recognition is independent from handedness

The “Left Perceptual Bias” (LPB) is the effect for which the left side of a picture such as a face is used to a greater extent than its right side as shown in a wide variety of experimental tasks. This effect has been observed for faces, body parts and objects. The present work investigated the presence of a LPB in recognizing hand gestures in two experiments. The role of the side of stimuli presentation (left /right), stimuli orientation (palm/back), participants’ handedness and gender were analyzed. Participants were presented with images of meaningless gestures performed by an actor with the right or left hand, from a palm or a back view. Immediately afterward, participants were shown a drawing and had to discriminate whether it represented the same gesture or not. In the first experiment, the task was administered to a sample of right-handed participants, half males. Results showed shorter response times for stimuli presented on the left side and for those shown from the palm. No gender differences were observed. The second experiment included a sample of left-handed participants and the LPB was replicated. Taken together, our results suggest an interpretation of the bias in terms of asymmetries in perceptual processing rather than the involvement of motor representations.

Hybrid Convolution-Based Efficientnet-Based Hand Gesture Recognition Framework with Optimized Algorithm

The difficulties in communication and hearing are an important concern for deaf–dumb people, which stop access to their essential and basic needs. Many findings have been made to address sign languages even though this challenging problem is not still solved. Many methods aimed to propose vision-based classifiers through identical pattern investigation tasks by obtaining the difficult handcraft feature descriptions of gestures from the gathered images. However, the efficacy of all those models is less for performing with a huge signbook captured from uncontrolled and complex background conditions. So, an effective Indian Sign Language (ISL) classification method is developed by an advanced deep learning approach. At first, the hand gesture images are obtained from the data source. Only the image of the hand, even from a complicated background, is extracted from the obtained image. The features are extracted using the Scale-Invariant Feature Transform (SIFT) method and Multiscale Vision Transformer (MVT). Then, the extracted features are fed to the Hybrid Convolution-based EfficientNet (HCEN) model. The hyper-parameters in the developed HCEN model are tuned using the implemented Adaptive Political Optimizer (APO) algorithm. The recognized hand signs are obtained from the suggested HCEN model. Various experiments are conducted to determine the performance of the suggested deep learning-based hand gesture recognition model.

Editorial

Dear Readers, It gives me great pleasure to announce the seventh regular issue of 2024. In this issue, 5 papers by 12 authors from 6 countries - Brazil, Ecuador, Germany, Iraq, Spain, Turkey - cover various topical aspects of computer science. In a continuous effort to further strengthen our journal, I would like to expand the editorial board: If you are a tenured associate professor or above with a strong publication record, you are welcome to apply to join our editorial board. We are also interested in high-quality proposals for special issues on new topics and trends. As always, I would like to thank all the authors for their sound research and the editorial board and guest reviewers for their extremely valuable review effort and suggestions for improvement. I also want to thank the readers for their interest in our articles, which is reflected in the consistently high number of user accesses and PDF downloads. These contributions, together with the generous support of the consortium members, maintain the quality of our journal. In the seventh regular issue, I am very pleased to present the following 5 accepted articles: Vinicius Bischoff and Kleinner Farias from Brazil report their study on a controlled experiment with 22 participants (15 students, 7 professionals) in which the correctness and the effort required for the integration of feature models were investigated. Abdulkadir Buldu, Kaplan Kaplan and Melih Kuncan from Turkey present their research on an assistive system based on EEG data and Continuous Wavelet Transform (CWT), which aims to reduce the life-threatening risk for epilepsy patients. In a collaboration between researchers from Ecuador and Spain, Darwin Alulema, Maximiliano Paredes-Velasco and Ricardo de Arriba Lasso report in their manuscript on the LESCA system, which performs adaptive content feedback through scaffolding and supports the development of high-level competencies. In another joint research between colleagues from Iraq and Germany, Anwar Mira and Olaf Hellwich present their approach to improve recognition capabilities by optimizing deep learning features for hand gesture image recognition. Specifically, they propose to enhance features of well-trained DNNs using an improved radial basis function (RBF) neural network, targeting recognition within individual gesture categories. And last but not least, Rasim Çekik and Mahmut Kaya from Turkey propose in their research a new performance metric to evaluate filter feature selection methods in text classification.  Enjoy Reading! Best regards,  Christian Gütl, Managing Editor 

An Embedded Neural Network Approach for Reinforcing Deep Learning: Advancing Hand Gesture Recognition

Deep neural networks (DNNs) can face limitations during training for recognition, motivating this study to improve recognition capabilities by optimizing deep learning features for hand gesture image recognition. We propose a novel approach that enhances features from well-trained DNNs using an improved radial basis function (RBF) neural network, targeting recognition within individual gesture categories. We achieve this by clustering images with a self-organizing map (SOM) network to identify optimal centers for RBF training. Our enhanced SOM, employing the Hassanat distance metric, outperforms the traditional K-Means method across a comparative analysis of various distance functions and the expanded number of cluster centers, accurately identifying hand gestures in images. Our training pipeline learns from hand gesture videos and static images, addressing the growing need for machines to interact with gestures. Despite challenges posed by gesture videos, such as sensitivity to hand pose sequences within a single gesture category and overlapping hand poses due to the high similarities and repetitions, our pipeline achieved significant enhancement without requiring time-related training data. We also improve the recognition of static hand pose images within the same category. Our work advances DNNs by integrating deep learning features and incorporating SOM for RBF training.

Hand Gestures Recognition Model Using Adaptive Feature Extraction with Attention-Based Hybrid Deep Learning via Optimization Strategy

Research on continuous hand gesture recognition is motivated by applications of deaf people to provide effective communication between one another. Numerous techniques are adapted for effectively identifying the hand gesture. Moreover, the collection of data becomes a challenging task in the existing models. In some instances, hand gesture in low-resolution images becomes a complicated task. In this research work, a deep learning-based hand gesture detection model is designed using hand data points to uncover patterns that are used to convey information. Initially, hand gesture images are obtained from public sources. The collected inputs are preprocessed through Contrast-Limited Adaptive Histogram Equalization and filtering to eliminate the unnecessary blurs in the images. The preprocessed images undergo hand segmentation using four methods color space transformation, skin color detection, active contour, and morphological operation. The segmented image is subjected to data point feature extraction, where the Adaptive Weighted Scale-Invariant Feature Transform is utilized for further enhancement. The hand data point-extracted features are given to the recognition of hand gestures with the support of an attention-based hybrid network, where the Attention-based Hybrid 1D Convolutional Neural Network with Recurrent Neural Network recognizes the hand gestures from the hand data points. In the hybridization network, the values are optimized with the help of the investigated Adaptive Dandelion White Shark Optimizer to enhance recognition effectiveness. The test results are validated with the existing hand gesture recognition models using diverse evaluation metrics. The findings of the recommended method show 97% and 99% in terms of accuracy and NPV. Thus, the developed model is validated to outperform significant performance rather than the existing models.

Active convolutional neural networks sign language (ActiveCNN-SL) framework: a paradigm shift in deaf-mute communication

Real-time speech-to-text and text-to-speech technologies have significantly influenced the accessibility of communication for individuals who are deaf or mute. This research aims to assess the efficacy of these technologies in facilitating communication between deaf or mute individuals and those who are neither deaf nor mute. A mixed-method approach will incorporate qualitative and quantitative data collection and analysis techniques. The study will involve participants from deaf or mute and non-deaf or non-mute communities. The research will scrutinize the precision and efficiency of communication using these technologies and evaluate user experience and satisfaction. Furthermore, the study intends to pinpoint potential obstacles and limitations of these technologies and offer suggestions for enhancing their effectiveness in fostering inclusivity. The study proposes an active learning framework for sign language gesture recognition, termed Active Convolutional Neural Networks—Sign Language (ActiveCNN-SL). ActiveCNN-SL aims to minimize the labeled data required for training and augment the accuracy of sign language gesture recognition through iterative human feedback. This proposed framework holds the potential to enhance communication accessibility for deaf and mute individuals and encourage inclusivity across various environments. The proposed framework is trained using two primary datasets: (i) the Sign Language Gesture Images Dataset and (ii) the American Sign Language Letters (ASL)—v1. The framework employs Resnet50 and YoloV.8 to train the datasets. It has demonstrated high performance in terms of precision and accuracy. The ResNet model achieved a remarkable accuracy rate of 99.98% during training, and it also exhibited a validation accuracy of 100%, surpassing the baseline CNN and RNN models. The YOLOv8 model outperformed previous methods on the ASL alphabet dataset, achieving an overall mean average accuracy for all classes of 97.8%.

Sign Language Detection using CNN Model

Abstract: Sign Language is mainly used by deaf (hard hearing) and dumb people to exchange information between their own community and with other people. It is a language where people use their hand gestures to communicate as they can’t speak or hear. Sign Language Recognition (SLR) deals with recognizing the hand gestures acquisition and continues till text or speech is generated for corresponding hand gestures. Here hand gestures for sign language can be classified as static and dynamic. Deep Learning Computer Vision is used to recognize the hand gestures by building Deep Neural Network architectures (Convolution Neural Network Architectures) where the model will learn to recognize the hand gestures images over an epoch.

Sign Speak: Recogninzing Sign Language with Machine Learning

Sign language serves as a critical means of communication for individuals with hearing impairments, enabling them to integrate into society effectively and express themselves. However, interpreting and recognizing sign language gestures present unique challenges due to the dynamic nature of gestures and spatial dependencies inherent in sign language communication. As a response, the SignSpeak project employs advanced machine learning techniques to address these challenges and enhance accessibility for the deaf and hard of hearing community. The project leverages a diverse dataset sourced from Kaggle, comprising images of sign language gestures captured in various contexts. The integration of advanced algorithms, such as 3D Convolutional Neural Networks (CNNs), and Gated Recurrent Units (GRUs), enables SignSpeak to recognize and interpret sign language gestures accurately and in real-time. This integration allows the model to capture both spatial and temporal features inherent in sign language, thus enabling more robust and accurate recognition. The project encompasses several critical stages, including data preprocessing, model development, training, and evaluation. Data preprocessing involves converting the image data into a suitable format and applying augmentation techniques to enhance the diversity and robustness of the dataset. Model development entails designing a deep learning architecture that combines CNNs and GRUs to effectively capture spatial and temporal dependencies in sign language gestures. Training the model involves optimizing parameters and hyperparameters to achieve optimal performance. Evaluation metrics such as accuracy, F1 score, and recall are utilized to assess the model's performance on both training and validation datasets. The trained model is then tested on a separate test dataset to evaluate its real- world performance and generalization ability. Experimental results demonstrate the efficacy of the SignSpeak approach in accurately recognizing and interpreting sign language gestures. The model achieves high accuracy scores, demonstrating its potential to enhance accessibility and inclusion for individuals with hearing impairments. By providing real-time translation of sign language into text or speech, SignSpeak contributes to breaking down communication barriers and promoting equal participation for all members of society.

Cognitive Classifier of Hand Gesture Images for Automated Sign Language Recognition: Soft Robot Assistance Based on Neutrosophic Markov Chain Paradigm

In recent years, Sign Language Recognition (SLR) has become an additional topic of discussion in the human–computer interface (HCI) field. The most significant difficulty confronting SLR recognition is finding algorithms that will scale effectively with a growing vocabulary size and a limited supply of training data for signer-independent applications. Due to its sensitivity to shape information, automated SLR based on hidden Markov models (HMMs) cannot characterize the confusing distributions of the observations in gesture features with sufficiently precise parameters. In order to simulate uncertainty in hypothesis spaces, many scholars provide an extension of the HMMs, utilizing higher-order fuzzy sets to generate interval-type-2 fuzzy HMMs. This expansion is helpful because it brings the uncertainty and fuzziness of conventional HMM mapping under control. The neutrosophic sets are used in this work to deal with indeterminacy in a practical SLR setting. Existing interval-type-2 fuzzy HMMs cannot consider uncertain information that includes indeterminacy. However, the neutrosophic hidden Markov model successfully identifies the best route between states when there is vagueness. This expansion is helpful because it brings the uncertainty and fuzziness of conventional HMM mapping under control. The neutrosophic three membership functions (truth, indeterminate, and falsity grades) provide more layers of autonomy for assessing HMM’s uncertainty. This approach could be helpful for an extensive vocabulary and hence seeks to solve the scalability issue. In addition, it may function independently of the signer, without needing data gloves or any other input devices. The experimental results demonstrate that the neutrosophic HMM is nearly as computationally difficult as the fuzzy HMM but has a similar performance and is more robust to gesture variations.

Diver-robot communication dataset for underwater hand gesture recognition

In this paper, we present a dataset of diving gesture images used for human–robot interaction underwater. By offering this open access dataset, the paper aims at investigating the potential of using visual detection of diving gestures from an autonomous underwater vehicle (AUV) as a form of communication with a human diver. In addition to the image recording, the same dataset was recorded using a smart gesture recognition glove. The glove uses dielectric elastomer sensors and on-board processing to determine the selected gesture and transmit the command associated with the gesture to the AUV via acoustics. Although this method can be used under different visibility conditions and even without line of sight, it introduces a communication delay required for the acoustic transmission of the gesture command. To compare efficiency, the glove was equipped with visual markers proposed in a gesture-based language called CADDIAN and recorded with an underwater camera in parallel to the glove’s onboard recognition process. The dataset contains over 30,000 underwater frames of nearly 900 individual gestures annotated in corresponding snippet folders. The dataset was recorded in a balanced ratio with five different divers in sea and five different divers in pool conditions, with gestures recorded at 1, 2 and 3 metres from the camera. The glove gesture recognition statistics are reported in terms of average diver reaction time, average time taken to perform a gesture, recognition success rate, transmission times and more. The dataset presented should provide a good baseline for comparing the performance of state of the art visual diving gesture recognition techniques under different visibility conditions.

“Deep Learning Based Classification of Single-Hand South Indian Sign Language Gestures”

Gesture recognition is a branch of computer science and language technology dedicated to utilizing mathematical algorithms for the analysis of human gestures. Within the realm of non-verbal communication, the pivotal role of human arm movements and gestures remains a focal point. This research introduces advanced multi-stream deep transfer learning models tailored for identifying signs from South Indian languages, specifically Kannada, Tamil, and Telugu. The primary aim is to offer support to individuals encountering speech disorders or disabilities. The key deep transfer learning models utilized include Inception-V3, VGG-16, and ResNet-50, which have been modified and improved to attain heightened classification efficacy. The dataset comprises 35,000 images capturing single-hand gestures. In the realm of models, Inception-V3 exhibits the utmost test accuracy, standing at 91.45%, alongside a validation accuracy of 93.45% when tasked with the classification of single-hand gesture images across thirty-five distinct categories. The importance of this study lies in its prospective utility for creating an automated system that can support and improve the functional capabilities of individuals facing speech disorders or disabilities.

Deep Learning Approaches for Age-based Gesture Classification in South Indian Sign Language

This study focuses on recognizing and categorizing South Indian Sign Language gestures based on different age groups through transfer learning models. Sign language serves as a natural and expressive communication method for individuals with hearing impairments. This study intends to develop deep transfer learning models, namely Inception-V3, VGG-16, and ResNet-50, to accurately identify and classify double-handed gestures in South Indian languages, like Kannada, Tamil, and Telugu. A dataset comprising 30,000 images of double-handed gestures, with 10,000 images for each considered age group (1-7, 8-25, and 25 and above), is utilized to enhance and modify the models for improved classification performance. Amongst the tested models, Inception-V3 achieves the best performance with a test precision of 95.20% and validation accuracy of 92.45%, demonstrating its effectiveness in accurately categorizing images of double-handed gestures into ten different classes.

Sign Language Detection and Recognition Using Media Pipe and Deep Learning Algorithm

People lacking the sense of hearing and the ability to speak have undeniable communication problems in their life. People with hearing and speech problems communicate using sign language with themselves and others. These communicating signs are made up of the shape of the hand and movement. Sign language is not essentially known to a more significant portion of the human population who uses spoken and written language for communication. Therefore, it is a necessity to develop technological tools for interpretation of sign language. Much research have been carried out to acknowledge sign language using technology for most global languages. But there are still scopes of development of tools and techniques for sign language development for local dialects. This work attempts to develop a technical approach for recognizing American Sign Language Using machine learning techniques, this work tried to establish a system for identifying the hand gestures from American Sign Language. A combination of two-dimensional and three-dimensional images of Assamese gestures has been used to prepare a dataset. The Media Pipe framework has been implemented to detect landmarks in the images. The results reveal that the method implemented in this work is effective for the recognition of the other alphabets and gestures in Sign Language. This method could also be tried and tested for the recognition of signs and gestures for various other local languages of India

“Deep Learning Based Classification of Single-Hand South Indian Sign Language Gestures”

Gesture recognition is a branch of computer science and language technology dedicated to utilizing mathematical algorithms for the analysis of human gestures. Within the realm of non-verbal communication, the pivotal role of human arm movements and gestures remains a focal point. This research introduces advanced multi-stream deep transfer learning models tailored for identifying signs from South Indian languages, specifically Kannada, Tamil, and Telugu. The primary aim is to offer support to individuals encountering speech disorders or disabilities. The key deep transfer learning models utilized include Inception-V3, VGG-16, and ResNet-50, which have been modified and improved to attain heightened classification efficacy. The dataset comprises 35,000 images capturing single-hand gestures. In the realm of models, Inception-V3 exhibits the utmost test accuracy, standing at 91.45%, alongside a validation accuracy of 93.45% when tasked with the classification of single-hand gesture images across thirty-five distinct categories. The importance of this study lies in its prospective utility for creating an automated system that can support and improve the functional capabilities of individuals facing speech disorders or disabilities.

A Wireframe-Based Approach for Classifying and Acquiring Proficiency in the American Sign Language (Student Abstract)

We describe our methodology for classifying ASL (American Sign Language) gestures. Rather than operate directly on raw images of hand gestures, we extract coor-dinates and render wireframes from individual images to construct a curated training dataset. This dataset is then used in a classifier that is memory efficient and provides effective performance (94% accuracy). Because we con-struct wireframes that contain information about several angles in the joints that comprise hands, our methodolo-gy is amenable to training those interested in learning ASL by identifying targeted errors in their hand gestures.

DEVELOPMENT OF MOBILE APPLICATION FOR SIGN LANGUAGE TRANSLATION USING DEEP LEARNING TECHNIQUE

Sign Language is a visual and gestural language used by deaf and hard-of-hearing people to communicate. However, communication between hearing and non-hearing individuals can be challenging due to the language barrier. In order to overcome this barrier, a Mobile based Translator is being proposed using Convolutional Neural Network (CNN) to recognize and translate hand gestures in real-time. The proposed system consists of a CNN model trained on a large dataset of hand gestures that includes various signs, such as the alphabet, numbers, and common phrases to recognize various signs, and a backend server to handle the translation of the recognized signs into text. The system was implemented using various CNN architectures like ResNet, MobileNet and VGG-16, where the later gave the best accuracy of 97.69%. The trained VGG-16 model recognizes the signs by extracting features from the images of the hand gestures and using these features to classify the gestures. Once a gesture is recognized, the backend server translates it into text using a pre-defined mapping of signs to words or phrases. The translated text will then be displayed to the user in real-time on the mobile app, enabling seamless communication between hearing and non-hearing individuals. The proposed system was implemented as a mobile app using Flutter, which is a cross-platform development framework. This mobile app make communication easier for vulnerable and enable sharing of information without any discrimination.