Sign Language Communication Research Articles

Soundboard-trained dogs produce non-accidental, non-random and non-imitative two-button combinations

Early studies attempting interspecies communication with great apes trained to use sign language and Augmented Interspecies Communication (AIC) devices were limited by methodological and technological constraints, as well as restrictive sample sizes. Evidence for animals’ intentional production of symbols was met with considerable criticisms which could not be easily deflected with existing data. More recently, thousands of pet dogs have been trained with AIC devices comprising soundboards of buttons that can be pressed to produce prerecorded human words or phrases. However, the nature of pets’ button presses remains an open question: are presses deliberate, and potentially meaningful? Using a large dataset of button presses by family dogs and their owners, we investigate whether dogs’ button presses are (i) non-accidental, (ii) non-random, and (iii) not mere repetitions of their owners’ presses. Our analyses reveal that, at the population level, soundboard use by dogs cannot be explained by random pressing, and that certain two-button concept combinations appear more often than expected by chance at the population level. We also find that dogs’ presses are not perfectly predicted by their owners’, suggesting that dogs’ presses are not merely repetitions of human presses, therefore suggesting that dog soundboard use is deliberate.

A flexible wearable sensor based on improving the piezoelectric layer performance of sandwich structures with micro-/mesopores for communication with the deaf-mute

Layered H-UiO-66@IL composites with both intraparticle micropores, interparticle mesopores, and multiple active sites were prepared. A key aspect of the technique was the addition of deionized water during solvothermal synthesis, which rapidly nucleated layered nanoparticles. A multifunctional, flexible, and stable polyacrylonitrile (PAN)-based membrane was successfully constructed by incorporating H-UiO-66 nanoparticles, modified by ionic liquid (1-allyl-3-butylimidazole tetrafluoroborate), into polymers with macromolecular chains under mild, controlled conditions. The resulting film exhibited excellent dielectric, ferroelectric, and mechanical properties due to the multilayer fiber design. Furthermore, the PAN matrix showed a higher content (97.41 %) of planar zigzag conformations, enhancing the piezoelectric properties. The composite film demonstrated ultra-high sensitivity (10.98 V/N) and an ultra-low detection limit of 3.5 × 10⁻⁴ N. In addition, the film passed 1,800-cycle stability tests and maintained its initial output value across various environmental conditions. This flexible piezoelectric sensor shows significant potential for aiding deaf and mute individuals in sign language or breathing communication.

Identifying Basic Sign Language Communication Abilities among Hearing Students to Communicate with their Deaf Peers at University Level

Identifying Basic Sign Language Communication Abilities among Hearing Students to Communicate with their Deaf Peers at University Level

DFCNet +: Cross-modal dynamic feature contrast net for continuous sign language recognition

In sign language communication, the combination of hand signs and facial expressions is used to convey messages in a fluid manner. Accurate interpretation relies heavily on understanding the context of these signs. Current methods, however, often focus on static images, missing the continuous flow and the story that unfolds through successive movements in sign language. To address this constraint, our research introduces the Dynamic Feature Contrast Net Plus (DFCNet+), a novel model that incorporates both dynamic feature extraction and cross-modal learning. The dynamic feature extraction module of DFCNet+ uses dynamic trajectory capture to monitor and record motion across frames and apply key features as an enhancement tool that highlights pixels that are critical for recognizing important sign language movements, allowing the model to follow the temporal variation of the signs. In the cross-modal learning module, we depart from the conventional approach of aligning video frames with textual descriptions. Instead, we adopt a gloss-level alignment, which provides a more detailed match between the visual signals and their corresponding text glosses, capturing the intricate relationship between what is seen and the associated text. The enhanced proficiency of DFCNet+ in discerning inter-frame details translates to heightened precision on benchmarks such as PHOENIX14, PHOENIX14-T and CSL-Daily. Such performance underscores its advantage in dynamic feature capture and inter-modal learning compared to conventional approaches to sign language interpretation. Our code is available at https://github.com/fyzjut/DFCNet_Plus.

Empowering sign language communication: Integrating sentiment and semantics for facial expression synthesis

Translating written sentences from oral languages to a sequence of manual and non-manual gestures plays a crucial role in building a more inclusive society for deaf and hard-of-hearing people. Facial expressions (non-manual), in particular, are responsible for encoding the grammar of the sentence to be spoken, applying punctuation, pronouns, or emphasizing signs. These non-manual gestures are closely related to the semantics of the sentence being spoken and also to the utterance of the speaker’s emotions. However, most Sign Language Production (SLP) approaches are centered on synthesizing manual gestures and do not focus on modeling the speaker’s expression. This paper introduces a new method focused in synthesizing facial expressions for sign language. Our goal is to improve sign language production by integrating sentiment information in facial expression generation. The approach leverages a sentence’s sentiment and semantic features to sample from a meaningful representation space, integrating the bias of the non-manual components into the sign language production process. To evaluate our method, we extend the Fréchet gesture distance (FGD) and propose a new metric called Fréchet Expression Distance (FED) and apply an extensive set of metrics to assess the quality of specific regions of the face. The experimental results showed that our method achieved state of the art, being superior to the competitors on How2Sign and PHOENIX14T datasets. Moreover, our architecture is based on a carefully designed graph pyramid that makes it simpler, easier to train, and capable of leveraging emotions to produce facial expressions. Our code and pretrained models will be available at: https://github.com/verlab/empowering-sign-language.

Quantification of Automatic Speech Recognition System Performance on d/Deaf and Hard of Hearing Speech.

To evaluate the performance of commercial automatic speech recognition (ASR) systems on d/Deaf and hard-of-hearing (d/Dhh) speech. A corpus containing 850 audio files of d/Dhh and normal hearing (NH) speech from the University of Memphis Speech Perception Assessment Laboratory was tested on four speech-to-text application program interfaces (APIs): Amazon Web Services, Microsoft Azure, Google Chirp, and OpenAI Whisper. We quantified the Word Error Rate (WER) of API transcriptions for 24 d/Dhh and nine NH participants and performed subgroup analysis by speech intelligibility classification (SIC), hearing loss (HL) onset, and primary communication mode. Mean WER averaged across APIs was 10 times higher for the d/Dhh group (52.6%) than the NH group (5.0%). APIs performed significantly worse for "low" and "medium" SIC (85.9% and 46.6% WER, respectively) as compared to "high" SIC group (9.5% WER, comparable to NH group). APIs performed significantly worse for speakers with prelingual HL relative to postlingual HL (80.5% and 37.1% WER, respectively). APIs performed significantly worse for speakers primarily communicating with sign language (70.2% WER) relative to speakers with both oral and sign language communication (51.5%) or oral communication only (19.7%). Commercial ASR systems underperform for d/Dhh individuals, especially those with "low" and "medium" SIC, prelingual onset of HL, and sign language as primary communication mode. This contrasts with Big Tech companies' promises of accessibility, indicating the need for ASR systems ethically trained on heterogeneous d/Dhh speech data. 3 Laryngoscope, 2024.

A novel case of Takenouchi-Kosaki syndrome: Focus on hearing and language rehabilitation

Abstract Takenouchi-Kosaki syndrome (TKS) is a rare congenital condition, caused by a mutation in the cell division cycle 42 gene. This gene typo impacts various physiological processes, including hearing, hematopoiesis, immunologic functioning, and neurodevelopment. Despite the high prevalence of hearing impairment in TKS, existing literature has concentrated primarily on pharmacological treatments for hematoimmunologic diseases. This case report presents a novel case of TKS, highlighting the importance of audiological assessment and rehabilitating interventions in children with TKS. We discuss the case of a prematurely born boy who was diagnosed with TKS and who presented with profound bilateral sensorineural hearing loss and associated developmental delays. Following comprehensive audiological evaluation and intervention – including hearing aids, sign language, and alternative augmentative communication systems – a multidisciplinary approach was adopted to enhance his communicative abilities. From this study, it emerges the increasing need for tailored rehabilitation strategies for TKS patients to improve communication skills and overall quality of life. Further researches are needed to develop standardized protocols for assessment and treatment in patients with TKS.

Millimeter wave gesture recognition using multi-feature fusion models in complex scenes

As a form of body language, the gesture plays an important role in smart homes, game interactions, and sign language communication, etc. The gesture recognition methods have been carried out extensively. The existing methods have inherent limitations regarding user experience, visual environment, and recognition granularity. Millimeter wave radar provides an effective method for the problems lie ahead gesture recognition because of the advantage of considerable bandwidth and high precision perception. Interfering factors and the complexity of the model raise an enormous challenge to the practical application of gesture recognition methods as the millimeter wave radar is applied to complex scenes. Based on multi-feature fusion, a gesture recognition method for complex scenes is proposed in this work. We collected data in variety places to improve sample reliability, filtered clutters to improve the signal’s signal-to-noise ratio (SNR), and then obtained multi features involves range-time map (RTM), Doppler-time map (DTM) and angle-time map (ATM) and fused them to enhance the richness and expression ability of the features. A lightweight neural network model multi-CNN-LSTM is designed to gestures recognition. This model consists of three convolutional neural network (CNN) for three obtained features and one long short-term memory (LSTM) for temporal features. We analyzed the performance and complexity of the model and verified the effectiveness of feature extraction. Numerous experiments have shown that this method has generalization ability, adaptability, and high robustness in complex scenarios. The recognition accuracy of 14 experimental gestures reached 97.28%.

Sign Speak: Recogninzing Sign Language with Machine Learning

Sign language serves as a critical means of communication for individuals with hearing impairments, enabling them to integrate into society effectively and express themselves. However, interpreting and recognizing sign language gestures present unique challenges due to the dynamic nature of gestures and spatial dependencies inherent in sign language communication. As a response, the SignSpeak project employs advanced machine learning techniques to address these challenges and enhance accessibility for the deaf and hard of hearing community. The project leverages a diverse dataset sourced from Kaggle, comprising images of sign language gestures captured in various contexts. The integration of advanced algorithms, such as 3D Convolutional Neural Networks (CNNs), and Gated Recurrent Units (GRUs), enables SignSpeak to recognize and interpret sign language gestures accurately and in real-time. This integration allows the model to capture both spatial and temporal features inherent in sign language, thus enabling more robust and accurate recognition. The project encompasses several critical stages, including data preprocessing, model development, training, and evaluation. Data preprocessing involves converting the image data into a suitable format and applying augmentation techniques to enhance the diversity and robustness of the dataset. Model development entails designing a deep learning architecture that combines CNNs and GRUs to effectively capture spatial and temporal dependencies in sign language gestures. Training the model involves optimizing parameters and hyperparameters to achieve optimal performance. Evaluation metrics such as accuracy, F1 score, and recall are utilized to assess the model's performance on both training and validation datasets. The trained model is then tested on a separate test dataset to evaluate its real- world performance and generalization ability. Experimental results demonstrate the efficacy of the SignSpeak approach in accurately recognizing and interpreting sign language gestures. The model achieves high accuracy scores, demonstrating its potential to enhance accessibility and inclusion for individuals with hearing impairments. By providing real-time translation of sign language into text or speech, SignSpeak contributes to breaking down communication barriers and promoting equal participation for all members of society.

EXPRESSIVE HAND: Sign Language to Audio Interpreter

The act of sharing or exchanging data, concepts, or emotions is known as communication. Both parties must be able to speak and comprehend the same language in order for communication to be established. It can be quite challenging for people with speech and hearing problems to communicate with the broader public. The way they communicate is through sign language. To communicate with the outside world, they are dependent on interpreters who use sign language. Being a minority, most people are unaware of the sign language they employ. Sign language communication is always necessary. Without an interpreter, communication might be challenging. One can create a machine learning-based model to get around this obstacle. It is possible to train a model to translate various sign language motions into English by recognizing them. Many individuals will find this useful while interacting and chatting with the deaf and ignorant.

Applying Swin Architecture to Diverse Sign Language Datasets

In an era where artificial intelligence (AI) bridges crucial communication gaps, this study extends AI’s utility to American and Taiwan Sign Language (ASL and TSL) communities through advanced models like the hierarchical vision transformer with shifted windows (Swin). This research evaluates Swin’s adaptability across sign languages, aiming for a universal platform for the unvoiced. Utilizing deep learning and transformer technologies, it has developed prototypes for ASL-to-English translation, supported by an educational framework to facilitate learning and comprehension, with the intention to include more languages in the future. This study highlights the efficacy of the Swin model, along with other models such as the vision transformer with deformable attention (DAT), ResNet-50, and VGG-16, in ASL recognition. The Swin model’s accuracy across various datasets underscore its potential. Additionally, this research explores the challenges of balancing accuracy with the need for real-time, portable language recognition capabilities and introduces the use of cutting-edge transformer models like Swin, DAT, and video Swin transformers for diverse datasets in sign language recognition. This study explores the integration of multimodality and large language models (LLMs) to promote global inclusivity. Future efforts will focus on enhancing these models and expanding their linguistic reach, with an emphasis on real-time translation applications and educational frameworks. These achievements not only advance the technology of sign language recognition but also provide more effective communication tools for the deaf and hard-of-hearing community.

Enhanced Sign Language Communication through Deep Learning-based Gesture and Action Recognition and Correction

Sign language is a crucial form of communication for the hearing impaired, presenting unique challenges for technological interpretation. This project focuses on developing a Sign Language Detection Using Action Recognition (SLDAR) system to address these challenges. Leveraging Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and multimodal fusion techniques, our system aims to accurately detect and recognize sign language gestures in real-time. The project involves collecting and annotating a diverse dataset comprising video, audio, and text data. We then design and train a deep learning model that combines features from these modalities to improve accuracy and robustness.The project involves collecting and annotating a diverse dataset, designing a deep learning model for multimodal fusion, and implementing a user-friendly interface for real-time interaction. The system's performance is evaluated using metrics such as accuracy, precision, and recall. Our goal is to enhance accessibility and communication for the hearing impaired through the development of this advanced sign language recognition system. Keywords:Acoustic Features, Linguistic Context, Mean Opinion Score (MOS), Natural Language Processing, NLP Algorithms, Signal-to-Noise Ratio (SNR), Speech Enhancement, Speech Quality, Telecommunication Systems, Voice Assistants.

A BIDIRECTIONAL AUDIO TO SIGN LANGUAGE TRANSLATION APP

The bidirectional audio to sign language translation app presented in this paper aims to improve communication between spoken language users and the sign language community. Leveraging natural language processing (NLP) techniques, including syntactic analysis and lemmatization, the app provides real-time translation services with enhanced accuracy and naturalness. Developed using Flask for web application development, NLTK for NLP tasks, and the Stanford Parser for syntactic parsing, the system offers an intuitive interface for seamless translation between spoken language and sign language. Evaluation of the app demonstrates notable improvements in translation quality and user satisfaction, indicating its potential for widespread adoption and positive impact on communication accessibility and inclusivity. Index Terms -Bidirectional Audio, Sign Language Translation, Natural Language Processing (NLP), Flask Framework, NLTK Library, Stanford Parser.

Digital Signature Verification System Using Convolutional Neural Networks

The design, development, and evaluation of a signature verification system, a critical component of biometric authentication. The study employs two primary datasets, the MCYT (MasterCard Young Teenager) Signature Dataset and the GPDS (Greek Sign Language Recognition) Signature Dataset, to assess the system's adaptability and accuracy. The MCYT dataset, featuring 6600 signature samples, provides variability in writing styles and demographic information for potential nuanced analyses. In contrast, the GPDS dataset, comprising 4000 signatures, introduces dynamic signing variations during Greek Sign Language communication. The project aims to develop a robust signature verification system capable of handling diverse scenarios, ultimately contributing to the advancement of biometric technology. The research combines insights from real-world variability and linguistic nuances, offering a comprehensive understanding of the system's effectiveness.

Sign Language Recognition by using Spatio-Temporal Features

The recognition of sign language has significant importance as an application that addresses the communication challenges faced by those within the Deaf and hard-of-hearing populations. This study introduces a methodology for the identification of sign language using spatio-temporal attributes, specifically concentrating on two distinct sign language datasets: the CSL (Chinese Sign Language) alphabet and the ArSL (Arabic Sign Language) alphabet. The study's objective is to create a powerful recognition system capable of reliably identifying sign language motions. A combination of spatio-temporal characteristics collected from video sequences of sign language motions is used in the suggested technique. These characteristics capture the spatial layout as well as the temporal dynamics of the motions, allowing the model to recognise signals with more precision. Experiments were carried out on the CSL and ArSL alphabet datasets to validate the method's efficacy. The findings of the trials are encouraging. The suggested system recognized CSL alphabet signs with an accuracy of 90.87% and ArSL alphabet signs with an accuracy of 89.46%. These high accuracy rates show the power of the spatiotemporal feature-based method for sign language identification. The system's success on two different sign languages implies that it is adaptable and useful in a wide variety of sign language applications. This study contributes to the evolution of assistive technology, making sign language recognition more accessible and efficient for those who communicate using sign language. Furthermore, the findings pave the way for additional research into spatiotemporal feature-based approaches in sign language recognition, with potential applications in real-world settings such as sign language interpretation and communication support.

Hand gesture recognition based on CNN and YOLO techniques

Communication is essential for humanity today and in the past. However, some individuals lack verbal communication due to their innate disability and physical losses from accidents. There are sign-language communication methods developed for such people to communicate. Artificial intelligence solutions are offered to remove the disadvantaged situations of people with disabilities due to communication in daily life. Nowadays, rapidly developing image processing and artificial intelligence methods are proper solutions for the problem focused on in this study. Convolution neural network techniques, which have become very popular recently, offer solutions to many problems. On the other hand, the YOLO algorithm shows very high performance in real-time object detection. In this study, we proposed a method for identifying the alphabets which each gesture delivers. This work studied hand detection on images and classification according to hand movements. The American Sign Language (ASL) standard was used as the sign language. The most recent version of YOLO, known as YOLOv5x, is used for gesture detection. Concentrating on the Static Sign-language problem, a study was conducted on the definition of hand movements. The letters “J” and “Z” are not included in the data set because movable hand signals are required. Apart from these two letters, a total number of 24 letters are classified. The proposed model achieved a training performance of 99.45% mAP@.5. Moreover, the proposed model has a performance of 97.9% mAP@.5 on the test dataset. The results demonstrate that the model's object detection performance is excellent. A statistical analysis of the training time shows that the training time has been drastically decreased, 4.5 hours with the current model as compared to the existing models in the literature.

Including the Sign Language Community in Language Research, Learning and Teaching: Video Reference Grammar of Slovenian Sign Language (SZJ)

Background: Deaf users of Slovenian Sign Language vary in competence between L1, delayed L1 and L2. They follow the grammatical patterns of their language but are unaware of them because the language is neither linguistically documented nor systematically used in the classroom. As a result, the available learning and teaching materials are inadequately prepared in many aspects.
 Purpose: The aim of this research was to create a reference grammar of SZJ by (i) using SZJ for metalinguistic descriptions and (ii) training Deaf signers as researchers and teachers. We observed their performance in order to answer the research question of whether it is possible to involve Deaf signers in the process of creating and disseminating a sign language reference grammar.
 Method: Training a group of deaf L1 signers to (i) start systematic research of their language within a selected formal approach, (ii) create didactic video materials for learning/teaching Slovenian Sign Language as L1 and L2, and (iii) use these materials in language courses for L1 and L2 deaf signers.
 Results: Eleven deaf L1 signers were trained as researchers in a 40-hour course. Five deaf and two hearing L1 signers were trained as language teachers in another 40-hour course and then presented the core concepts of Slovenian Sign Language grammar to 302 members of 12 local Slovenian deaf clubs in 24 editions of a 40-hour language course. For the presentations, they used didactic video materials (duration 5:46 hours). These materials were produced by the five L1 signers and later uploaded to two freely accessible online video platforms.
 Conclusion: The approach proved fruitful: signers were actively involved in the creation and dissemination of the first Slovenian Sign Language reference grammar. The available analytics show that the materials are continuously viewed by both deaf and hearing users.

Cross-Sectional Analysis of Caregiver-Reported Expressive Language Profiles and Associated Covariates in Individuals with Down Syndrome.

To describe the distribution of expressive language abilities of individuals with Down syndrome (DS) in a clinical sample and characterize demographic, environmental, and medical factors associated with varying expressive language profiles. Cross-sectional analysis was completed on a sample of 345 individuals with DS between the ages of 4 and 22 years who were enrolled into a longitudinal clinical database between March 2018 and August 2021. Expressive language-related items on a standardized caregiver-reported questionnaire assessing domains of functioning in neurodevelopmental disorders were used to conduct latent variable modeling and determine caregiver-reported expressive language (CREL) classes across the sample. Linear regression was used to explore associations between CREL classes and predictor variables. Latent variable modeling revealed 3 distinct classes of CREL abilities representing higher, middle, and lower CREL. Individuals in the lower CREL class were more likely to be female, to use sign language or visual communication systems, have reduced pronunciation, attend private or residential school, and to be in a substantially separate classroom. Membership was not predicted by complex medical histories or co-occurring neurodevelopmental diagnoses. Caregiver-reported expressive language abilities in a cohort of individuals with DS were variable, with most of the individuals belonging to higher or middle CREL classes, relative to one another. Additional studies are indicated to understand factors that predict higher expressive language ability and explore how to direct services to individuals who are at risk of more profound language delays.

Design of Sign Language Learning Media Based on Virtual Reality

Sign language has become a very important means of communication for individuals with hearing loss, allowing them to interact with people who are not hearing impaired. In recent years, virtual reality (VR) technology has advanced rapidly, opening up new opportunities to enhance the user experience in various situations. This study aims to design and develop an easy-to-use virtual reality-based interactive sign language application. The software development method used is the Software Life Cycle Development Method (MDLC), which includes the stages of analysis, design, implementation and testing. This app is designed to run on affordable and popular VR devices to make it accessible to as many individuals as possible. In the development process, we conducted an analysis of various sign languages used globally, focusing on hand gestures and facial expressions which have an important role in sign language communication. Users can learn and practice sign language through interactive experiences in a virtual environment that mimics real-life situations. This application is also equipped with visual and audio feedback features to help users improve and hone their skills. At the testing stage, this application was evaluated through trials involving participants with diverse backgrounds in the use of sign language. The test results show that users feel involved in the VR experience and experience increased confidence in using sign language after using this application. This research proves that VR technology can be an effective tool in sign language learning and development. Thus, the virtual reality-based sign language application that has been designed and developed in this study makes a significant contribution in increasing the accessibility of communication for individuals with hearing impairments. The implications of this research can be the basis for further development in the field of sign language and VR technology.

Sign language in immersive virtual reality: design, development, and evaluation of a virtual reality learning environment prototype

ABSTRACT Non-Verbal Cues (NVCs) add to communication effectiveness among individuals in both real and virtual world. Thus, NVCs transference between the two receives increased attention from both the industry and research community. Their efforts lead to sophisticated technological solutions which allow high fidelity NVCs to be transferred from real individuals to virtual characters/avatars, in real-time, adding to the communication effectiveness of both standalone Immersive Virtual Reality (IVR) applications and Social Virtual Reality (SVR) platforms. That combination triggered the development of Virtual Reality Learning Environments (VRLEs), aiming at assisting students towards learning sign language and communicating through it in both virtual and real world. Still, technological limitations affect the quality of such learning and communication experience. This work presents the design, development, and evaluation of an SVR compatible system for sign language learning and communication in virtual worlds. The system considers lessons learned from previous studies and was used to create a VRLE mimicking learning and examination processes students of sign language follow in real life. The results of our study show that despite the appreciation of technological solutions efficiency towards NVCs transference, there are still several issues to be investigated to increase the effectiveness of communication through sign language in virtual worlds.