Sign Language Gestures Research Articles

AI empowered Auslan learning for parents of deaf children and children of deaf adults

AbstractCommunication poses a challenge for the deaf and hearing loss community. This difficulty is even more pronounced in the families of Children of Deaf Adults (CODAs) and Parents of Deaf Children (PODCs). To help these families overcome this challenge, we design an AI-empowered interactive bi-directional Australian Sign Language (i.e., Auslan) dictionary application to facilitate communication within a household. Technically, our APP can not only look up sign gestures for the given English words but also translate isolated Auslan gestures into English. Through an inviting user interface and experience design, we can further improve engagement within the CODA and PODC families while enabling Auslan education at home. The positive user experience underscores the success of our APP not only in leveraging AI to revolutionise Auslan education but also in promoting cross-generational language acquisition and communication.

The Crossover: Integrating Indian Sign Language with the Mudras in an Indian Classical Dance Performance Creation

Indian classical dance uses a variety of non-verbal communication devices, with gestures called hasta or mudra being one of the most ancient sign languages powerful to transmit the meaning of the lyrics through dance. The classical dance forms of Kerala have an elaborate scheme of gestures drawn from the ancient text on aesthetic sign language, called Hastalakṣanadīpika, inspired by the Vedic, Nāṭya and Tantric traditions. The following is an attempt to document a performance-oriented, socially inclusive effort that makes Indian traditional dance performance discernible to the Deaf and Hard of Hearing (DHH) community, a disparate group that is hardly considered as a spectator during performance creation by mainstream Indian classical dancers. The Indian Sign Language (ISL) of individuals with hearing impairments is elaborate, yet is rarely used as a gestural language along with other aesthetic gestural sign languages to disseminate a narrative in a work of art. In her role as both creator and performer, Methil Devika creates work in Mohiniyāṭṭam, a graceful dance form of Kerala, integrating the sign language of those with hearing impairments along with the gestural language prescribed in the ancient treatise Hastalakṣanadīpika. She applies it strategically in her work of art, as an interlinking tool between the art and the audience. In the first few moments, “all walls came crumbling down,” something the performer quotes in the visual documentation of the work. Any separation between the hearing impaired and the artiste, as well as the spectators, become inconsequential. The performance and the presentation have been video documented in a film called The Crossover, and the following is an attempt to describe the creative process.

BISINDO (Bahasa Isyarat Indonesia) Sign Language Recognition Using Deep Learning

<p>Sign language recognition plays a crucial role in<br />facilitating communication for individuals with hearing<br />impairments. This paper presents a deep learning-based<br />approach for recognizing Bahasa Isyarat Indonesia (BISINDO),<br />the sign language used in Indonesia. The proposed system<br />employs convolutional neural networks (CNNs) and recurrent<br />neural networks (RNNs) to automatically extract features from<br />sign language gestures and classify them into corresponding<br />linguistic units. The dataset used for training and evaluation<br />consists of annotated BISINDO sign language videos.<br />Preprocessing techniques such as normalization and<br />augmentation are applied to enhance the robustness of the<br />model. Experimental results demonstrate the effectiveness of the<br />proposed approach in accurately recognizing BISINDO sign<br />language gestures, achieving state-of-the-art performance<br />compared to existing methods. The developed system shows<br />promising potential for real-world applications in enhancing<br />communication accessibility for the hearing-impaired<br />community in Indonesia.</p>

Sign-to-Text Translation from Panamanian Sign Language to Spanish in Continuous Capture Mode with Deep Neural Networks

Convolutional neural networks (CNN) have provided great advances for the task of sign language recognition (SLR). However, recurrent neural networks (RNN) in the form of long–short-term memory (LSTM) have become a means for providing solutions to problems involving sequential data. This research proposes the development of a sign language translation system that converts Panamanian Sign Language (PSL) signs into text in Spanish using an LSTM model that, among many things, makes it possible to work with non-static signs (as sequential data). The deep learning model presented focuses on action detection, in this case, the execution of the signs. This involves processing in a precise manner the frames in which a sign language gesture is made. The proposal is a holistic solution that considers, in addition to the seeking of the hands of the speaker, the face and pose determinants. These were added due to the fact that when communicating through sign languages, other visual characteristics matter beyond hand gestures. For the training of this system, a data set of 330 videos (of 30 frames each) for five possible classes (different signs considered) was created. The model was tested having an accuracy of 98.8%, making this a valuable base system for effective communication between PSL users and Spanish speakers. In conclusion, this work provides an improvement of the state of the art for PSL–Spanish translation by using the possibilities of translatable signs via deep learning.

Frequency and Spatial Domain-Based Approaches for Recognition of Indian Sign Language Gestures

Objectives: The objective of this paper is to introduce and demonstrate an innovative approach for the recognition of Indian sign language gestures, with a focus on bridging communication gap between the deaf and hearing communities. The goal is to contribute to the development of effective tools and technologies that facilitate seamless communication between individuals using sign language and the people with no knowledge about sign language. Methods: The methodology consists of three key steps. First, data pre-processing involves resizing and contours extraction. Next, feature extraction employs Fourier descriptors for frequency domain analysis and gray-level-co-occurrence matrix for spatial domain analysis. Finally, various machine learning models including SVM, Random Forest, Logistic Regression, K-Nearest Neighbor and Naive Bayes are trained on a standard dataset. Findings: In our controlled experimental setup, we applied a diverse set of machine learning classifiers to evaluate the proposed approach for gesture recognition. Among the classifiers tested, K-Nearest Neighbors demonstrated the highest accuracy, achieving 99.82%. To validate the robustness of our approach, we employed k-fold cross-validation with 5 folds. Novelty: This study presents an innovative method for sign language recognition by employing a dual-domain fusion strategy that prominently emphasizes the frequency domain. Through the integration of Fourier descriptors, the research conducts a detailed frequency domain analysis to characterize the contour shapes of sign language gestures. The synergy with gray-level co-occurrence matrix texture features in the spatial domain analysis, contributes to the creation of a comprehensive feature vector. The proposed approach ensures a thorough exploration of gesture features, there by advancing the precision and efficacy of sign language recognition. Keywords: Indian Sign Language (ISL), Sign Language Recognition (SLR), Frequency domain, Spatial domain, Fourier descriptors, Gray level co­occurrence matrix (GLCM), K­ Fold

A Comprehensive Study on Relative Distances of Hand Landmarks Approach for American Sign Language Gesture

A Comprehensive Study on Relative Distances of Hand Landmarks Approach for American Sign Language Gesture

TRANSLATION GLOVE WITH SENSOR FOR SIGN LANGUAGE

Abstract-The translating glove with sign language sensors is a cutting-edge wearable device designed to close the communication gap between sign language users and non- users. The glove has a number of sensors, such as accelerometers, gyroscopes, and flex sensors, which are strategically placed on the hand and fingers to precisely detect the motions and gestures used in sign language.The glove can detect and interpret a wide range of sign language gestures by using sophisticated algorithms to assess sensor data in real-time. When the sensors record the movements of a sign language user, the algorithms convert their gestures into either spoken or written language. Consequently, real- time communication is possible between people who use sign language and others who do not.Reverse translation is another feature of the glove. The device's sensors convert spoken or written input into motions that are displayed on the glove's surface as the user talks or writes into it. By making it possible for sign language users to comprehend and respond to spoken information, this promotes inclusive communication.

Hand Sign Language Recognition and Conversion into Text Using Convolutional Neural Network

The current research looks into using convolutional neural networks, also known as CNNs, to develop a hand sign understanding and conversion system. The study addresses the urgent need for a precise and effective way to close the communication gap that exists between the hearing and the deaf communities. The study examines how well CNNs can identify and translate hand sign gestures into text, giving sign language users a real-time means of communication. We use three steps in our methodology: gathering datasets, training models, and thorough assessment. The outcomes show encouraging hand sign recognition accuracy, creating new opportunities for inclusive communication. Sign language is one of the oldest and most natural forms of language for communication, but since most people do not know sign language and interpreters are very difficult to come by, we have come up with a real-time method using neural networks for fingerspelling-based American sign language. In our method, the hand is first passed through a filter and after the filter is applied the hand is passed through a classifier which predicts the class of the hand gestures. Our method provides 95.7 % accuracy for the 26 letters of the alphabet. Keywords-Sign Language, ASL, Hearing disability, Convolutional Neural Network (CNN), Computer Vision, Machine Learning, Gesture recognition, Sign language recognition, Hue Saturation Value algorithm.

Gesture Recognition Based on a Convolutional Neural Network-Bidirectional Long Short-Term Memory Network for a Wearable Wrist Sensor with Multi-Walled Carbon Nanotube/Cotton Fabric Material.

Flexible pressure sensors play a crucial role in detecting human motion and facilitating human-computer interaction. In this paper, a type of flexible pressure sensor unit with high sensitivity (2.242 kPa-1), fast response time (80 ms), and remarkable stability (1000 cycles) is proposed and fabricated by the multi-walled carbon nanotube (MWCNT)/cotton fabric (CF) material based on a dip-coating method. Six flexible pressure sensor units are integrated into a flexible wristband and made into a wearable and portable wrist sensor with favorable stability. Then, seven wrist gestures (Gesture Group #1), five letter gestures (Gesture Group #2), and eight sign language gestures (Gesture Group #3) are performed by wearing the wrist sensor, and the corresponding time sequence signals of the three gesture groups (#1, #2, and #3) from the wrist sensor are collected, respectively. To efficiently recognize different gestures from the three groups detected by the wrist sensor, a fusion network model combined with a convolutional neural network (CNN) and the bidirectional long short-term memory (BiLSTM) neural network, named CNN-BiLSTM, which has strong robustness and generalization ability, is constructed. The three types of Gesture Groups were recognized based on the CNN-BiLSTM model with accuracies of 99.40%, 95.00%, and 98.44%. Twenty gestures (merged by Group #1, #2, and #3) were recognized with an accuracy of 96.88% to validate the applicability of the wrist sensor based on this model for gesture recognition. The experimental results denote that the CNN-BiLSTM model has very efficient performance in recognizing different gestures collected from the flexible wrist sensor.

Revealing sign language for individuals who are deaf and mute through the utilization of neural networks

Living without effective communication poses significant challenges for humans. Individuals employ various methods to convey and share their thoughts and ideas between the sender and receiver. Two of the most prevalent means of communication are verbal speech, which relies on auditory perception, and non-verbal communication through gestures involving bodily movements such as hand gestures and facial expressions. Sign language, specifically categorized as a gestural language, is a unique form of communication that relies on visual perception for understanding and expression. While many individuals incorporate gestures into their communication, for deaf individuals, sign language is often their primary and essential means of communication. Individuals who are deaf and dumb rely on communication to interact with others, gain knowledge, and engage in their surroundings. Sign language serves as a crucial link that reduces the distance between them and the broader society. In order to enhance this communication, we've created models with the ability to identify sign language gestures and translate them into conventional text. Through the training of these models on a dataset employing neural networks, remarkable outcomes have been attained. This technology enables individuals, without prior knowledge of sign language, to understand the intentions and messages of individuals with disabilities, fostering greater inclusivity and accessibility in our society. Three algorithms were used to achieve this work and the findings show very good outcomes, i.e. Random Forest at 98%, Logistic Regression at 99%, and Decision Tree at 91%. Keywords: Machine Learning, Sign Language, Random Forest, Neural Network, Logistic Regression Deaf and Dumb, Decision Tree

Isolated Arabic Sign Language Recognition Using a Transformer-based Model and Landmark Keypoints

Pose-based approaches for sign language recognition provide light-weight and fast models that can be adopted in real-time applications. This article presents a framework for isolated Arabic sign language recognition using hand and face keypoints. We employed MediaPipe pose estimator for extracting the keypoints of sign gestures in the video stream. Using the extracted keypoints, three models were proposed for sign language recognition: Long-Term Short Memory, Temporal Convolution Networks, and Transformer-based models. Moreover, we investigated the importance of non-manual features for sign language recognition systems and the obtained results showed that combining hand and face keypoints boosted the recognition accuracy by around 4% compared with only hand keypoints. The proposed models were evaluated on Arabic and Argentinian sign languages. Using the KArSL-100 dataset, the proposed pose-based Transformer achieved the highest accuracy of 99.74% and 68.2% in signer-dependent and -independent modes, respectively. Additionally, the Transformer was evaluated on the LSA64 dataset and obtained an accuracy of 98.25% and 91.09% in signer-dependent and -independent modes, respectively. Consequently, the pose-based Transformer outperformed the state-of-the-art techniques on both datasets using keypoints from the signer’s hands and face.

Towards a Bidirectional Mexican Sign Language–Spanish Translation System: A Deep Learning Approach

People with hearing disabilities often face communication barriers when interacting with hearing individuals. To address this issue, this paper proposes a bidirectional Sign Language Translation System that aims to bridge the communication gap. Deep learning models such as recurrent neural networks (RNN), bidirectional RNN (BRNN), LSTM, GRU, and Transformers are compared to find the most accurate model for sign language recognition and translation. Keypoint detection using MediaPipe is employed to track and understand sign language gestures. The system features a user-friendly graphical interface with modes for translating between Mexican Sign Language (MSL) and Spanish in both directions. Users can input signs or text and obtain corresponding translations. Performance evaluation demonstrates high accuracy, with the BRNN model achieving 98.8% accuracy. The research emphasizes the importance of hand features in sign language recognition. Future developments could focus on enhancing accessibility and expanding the system to support other sign languages. This Sign Language Translation System offers a promising solution to improve communication accessibility and foster inclusivity for individuals with hearing disabilities.

Deep convolutional neural network-based Leveraging Lion Swarm Optimizer for gesture recognition and classification

<abstract> <p>Vision-based human gesture detection is the task of forecasting a gesture, namely clapping or sign language gestures, or waving hello, utilizing various video frames. One of the attractive features of gesture detection is that it makes it possible for humans to interact with devices and computers without the necessity for an external input tool like a remote control or a mouse. Gesture detection from videos has various applications, like robot learning, control of consumer electronics computer games, and mechanical systems. This study leverages the Lion Swarm optimizer with a deep convolutional neural network (LSO-DCNN) for gesture recognition and classification. The purpose of the LSO-DCNN technique lies in the proper identification and categorization of various categories of gestures that exist in the input images. The presented LSO-DCNN model follows a three-step procedure. At the initial step, the 1D-convolutional neural network (1D-CNN) method derives a collection of feature vectors. In the second step, the LSO algorithm optimally chooses the hyperparameter values of the 1D-CNN model. At the final step, the extreme gradient boosting (XGBoost) classifier allocates proper classes, i.e., it recognizes the gestures efficaciously. To demonstrate the enhanced gesture classification results of the LSO-DCNN approach, a wide range of experimental results are investigated. The brief comparative study reported the improvements in the LSO-DCNN technique in the gesture recognition process.</p> </abstract>

Recognition of Wh-Question Sign Gestures in Video Streams using an Attention Driven C3D-BiLSTM Network

Human sign language is a visual and gestural means of communication used by people with hearing impairments to interact with others. It has the potential to enable interaction between individuals who struggle with verbal communication and those who lack the essential proficiency to comprehend sign language. Wh-questions constitute a substantial portion of daily sign language interactions. Automatic recognition of Wh-question words in sign language, encompassing interrogative words, has received limited attention in gesture recognition. In this context, a novel dataset named the American Question Sign Video dataset (AQSVd) has been introduced as a significant contribution. Recognizing Wh-question signs in video streams empowers individuals to effectively convey messages through hand movements and gestures, fostering inclusivity and accessibility in communication for deaf and hearing-impaired individuals. The paper proposes a Deep Convolutional-3D BiLSTM Multi-head Attention network for recognizing American Wh-Question word sign gestures in video streams. Incorporating Multi-head attention in the proposed model enhances its ability to capture intricate spatial and temporal features, essential for accurate gesture recognition. The proposed model is thoroughly evaluated across a range of datasets, such as INCLUDE50, WLASL-100, and our AQSVd dataset, achieving remarkable results. Specifically, our model demonstrated an outstanding 98.91% validation accuracy on the AQSVd dataset. The C3D-BiLSTM MHAttention model outperforms other state-of-the-art models, demonstrating its superiority in sign language recognition tasks. The proposed dataset and C3D-BiLSTM Multi-head Attention model contribute significantly to this field, offering potential benefits in education, human-robot interaction, and overall communication for the hearing-impaired community.

Gesture recognition of the Kazakh alphabet based on machine and deep learning models

Currently, a growing body of research focuses on addressing problems using computer vision libraries and artificial intelligence tools. The predominant approaches involve employing machine and deep learning models of artificial neural networks to recognize gestures in the Kazakh Sign Alphabet (KSA) via supervised and deep learning techniques for sequential data processing. Pattern recognition in this context involves identifying an object within an image, where the object can be abstract and vary in shape. We have chosen to investigate the field of gesture recognition, specifically. For recognizing Kazakh Sign Language (KSL), the initial step involves mastering the KSA. Training a neural network to recognize KSL necessitates the collection of datasets in the form of images depicting hand gestures. In this research, prominent hand gesture recognition models such as the Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and Support Vector Machine (SVM) were analyzed. These models differ in their methodologies, processing times, and training data requirements. A significant aspect of this study is the application of unsupervised and supervised learning techniques including CNN, LSTM, and SVM. The experiments yielded diverse results when training neural networks for recognizing gestures in Kazakh sign language based on the dactyl alphabet. This article provides a comprehensive overview of each method, their specific purposes, and their effectiveness in terms of performance and training. Numerous experimental outcomes were documented in a table, showcasing the accuracy of recognizing each gesture. Additionally, specific hand gestures were tested in front of a camera to identify the gesture and display the result on the screen. A notable feature was the use of mathematical formulas and functions to elucidate the operating principles of the machine learning methods, as well as the logical structure and design of the LSTM model.

AIGC Enabling Non-Genetic Design Methods and Practices

Abstract Artificial Intelligence Generated Content (AIGC) technology aligns seamlessly with the design requirements of non-genetic heritage, offering a viable pathway for its modernization. This paper delineates the specific design needs of non-genetic heritage and utilizes a diffusion model to create themed images and animations related to this heritage. Additionally, AIGC is employed to enhance the creation of virtual reality interactive imagery. The Long Short-Term Memory (LSTM) network is deployed to classify time-series gesture data, facilitating the training and categorization of Chinese Sign Language (CSL) gestures for virtual interactive engagement with non-heritage themes. We have integrated the AIGC operation process into the theme of non-genetic inheritance, thereby constructing a robust development trajectory for AIGC-enhanced non-genetic heritage. The experimental setup is crafted to ascertain the optimal number of iterations and training durations through the control variable method. We evaluate the efficacy of the diffusion model for anti-implicit writing analysis and the performance of the speech recognition, text dialogue, and text response modules within the non-heritage multimodal interactive framework using Word Error Rate (WER) and Mean Opinion Score (MOS). A descriptive analysis of users’ interactive experiences with non-heritage content is also conducted. The results indicate that the speech recognition module achieved a WER of 0.365, while the text response module garnered an MOS of 4.49 with a standard deviation of 0.56. This multimodal, non-heritage virtual interaction, leveraging multiple modalities, enriches users’ experiences and deepens their understanding and appreciation of non-heritage content. Consequently, this enhances the high-quality development of non-genetic heritage.

Silent Expressions Unveiled: Deep Learning for British and American Sign Language Detection

Sign language is an efficient means of communication for the hearing impaired. The main goal of this research is to recognize both American and British sign languages using computer vision algorithms and neural networks. Our work addresses the critical need for inclusive and accessible communication for the Deaf community. For the study, we compiled a large dataset of her ASL and BSL gestures. Preprocess the data to improve feature extraction and reduce noise. Our model architecture uses a convolutional neural network (CNN) to capture the temporal patterns of finger gestures. This allows you to use Mediapipe keypoint detection within your model. The end result is a comprehensive model that can accurately and quickly recognize and classify sign language gestures. We further improve the usability of the system by integrating keyword recognition into the model, allowing sign language sentences to be decoded and translated into text. Our test results demonstrate how well our deep learning strategy works to achieve high efficiency and accuracy in both ASL and BSL detection. This program helps bridge the gap between those who use sign language and those who do not. Our new approach not only accurately recognizes American and British sign language, but also seamlessly integrates real-time feedback to help users make immediate improvements to improve their sign language skills. We can do better. Moreover, our approach is flexible enough to be used in a variety of settings, ensuring its usefulness in educational environments and providing sign language learners with a fun and dynamic learning environment. Explore and exploit the full potential of Mediapipe Keypoint using CNNs and also leverage the efficiency of computer vision.

Gesture-to-Text: A Real-Time Indian Sign Language Translator with Pose Estimation and LSTMs

In recent years, there have been notable advancements in technology, deep learning, and pose detection. One significant application of these advancements pertains to the real-time detection of sign language from video sources. The motivation behind this research stems from the pressing societal need to enhance the quality of life for individuals with speech impairments. Given the current prominence of online meetings, exacerbated by the COVID-19 pandemic, there is a growing need for systems that can provide individuals with speech impairments greater independence in communication, eliminating the requirement for a human translator. This research proposal advocates for a solution that leverages PoseNet algorithms for the extraction of key pose points, which are subsequently employed within LSTM models for the predictive modeling of sign language gestures. This research paper aims to make several notable contributions to the field of assistive technology and human-computer interaction. The achieved accuracy stands at an impressive 98%, underscoring the robustness and precision of our proposed system.

Sign Language Detection

Abstract: Sign language is a crucial means of communication for individuals with hearing impairments, serving as a rich and expressive form of language. As the world becomes increasingly interconnected, there is a growing need for technology to facilitate effective communication between sign language users and the broader community. Sign language detection, a branch of computer vision and artificial intelligence, plays a pivotal role in bridging these communication gaps. This abstract provides an overview of the key aspects related to sign language detection. The primary focus is on the technological advancements and applications that contribute to the seamless integration of sign language into various aspects of daily life. The paper explores the challenges associated with recognizing and interpreting sign language gestures, considering factors such as variations in signing styles, lighting conditions, and diverse signing communities. The proposed paper delves into the methodologies employed in sign language detection systems, including image and video processing techniques, deep learning algorithms, and sensor technologies. It highlights the significance of dataset diversity and the role of machine learning in enhancing the accuracy and robustness of sign language detection models. Additionally, the paper discusses the potential integration of sign language detection into mainstream devices, applications, and assistive technologies to empower individuals with hearing impairments.

Smart Glove for Sign Language Translation

Sign language is a vital mode of communication for deaf people, yet it presents a significant barrier when interacting with those who do not understand it. The advent of technology has paved the way for innovative solutions to bridge this communication gap. This abstract explores the development and implications of a smart glove designed for sign language translation (SLT). The primary aim of this study is to create a wearable device, the Smart Glove, capable of recognizing and translating sign language gestures into text or speech. Key objectives include designing a lightweight and ergonomic glove prototype, developing machine learning algorithms for sign language recognition, implementing real-time translation capabilities, evaluating the glove's accuracy and usability, and assessing the potential impact on facilitating communication for deaf people. The Smart Glove utilizes only one sensor, flex sensors, to capture hand movements and gestures. These data inputs are processed through a custom-built machine learning model trained on a comprehensive sign language dataset. Preliminary results indicate a high accuracy rate in recognizing sign language gestures, with an average recognition rate of over 90% across a diverse set of signs. While challenges such as expanding gesture recognition and refining translation algorithms remain, this technology offers a promising solution to break down communication barriers and enhance the quality of life for those who rely on sign language.