Sign Language Research Articles

Beyond Granularity: Enhancing Continuous Sign Language Recognition with Granularity-Aware Feature Fusion and Attention Optimization

The advancement of deep learning techniques has significantly propelled the development of the continuous sign language recognition (cSLR) task. However, the spatial feature extraction of sign language videos in the RGB space tends to focus on the overall image information while neglecting the perception of traits at different granularities, such as eye gaze and lip shape, which are more detailed, or posture and gestures, which are more macroscopic. Exploring the efficient fusion of visual information of different granularities is crucial for accurate sign language recognition. In addition, applying a vanilla Transformer to sequence modeling in cSLR exhibits weak performance because specific video frames could interfere with the attention mechanism. These limitations constrain the capability to understand potential semantic characteristics. We introduce a feature fusion method for integrating visual features of disparate granularities and refine the metric of attention to enhance the Transformer’s comprehension of video content. Specifically, we extract CNN feature maps with varying receptive fields and employ a self-attention mechanism to fuse feature maps of different granularities, thereby obtaining multi-scale spatial features of the sign language framework. As for video modeling, we first analyze why the vanilla Transformer failed in cSLR and observe that the magnitude of the feature vectors of video frames could interfere with the distribution of attention weights. Therefore, we utilize the Euclidean distance among vectors to measure the attention weights instead of scaled-dot to enhance dynamic temporal modeling capabilities. Finally, we integrate the two components to construct the model MSF-ET (Multi-Scaled feature Fusion–Euclidean Transformer) for cSLR and train the model end-to-end. We perform experiments on large-scale cSLR benchmarks—PHOENIX-2014 and Chinese Sign Language (CSL)—to validate the effectiveness.

Sign language delays in deaf 3- to 5-year-olds with hearing parents.

Deaf children of hearing parents (DOH) are at risk for early language delays (ELD) due to environmental and etiological factors, compounding the previously reported higher incidence of ELD in deaf children of deaf parents (DOD) compared to the general population. Archival data from the online database of the Visual Communication and Sign Language Checklist yielded 147 ratings of DOH 3- to 5-year-old children whose parents reported American sign language (ASL) being used in the home at least equally to spoken language. Research goals included (1) examining the incidence of ELD in this cohort, (2) investigating differences in outcomes based on the scaling method used, (3) exploring patterns among the ratings of DOH children identified with ELD, (4) comparing the DOH outcomes to published DOD data, and (5) investigating differences between ratings and individual children. The DOH outcomes suggested an incidence of ELD twice that of the previous DOD sample, with a higher percentage of more severe delays. Even DOH children not identified with ELD demonstrated less well-developed ASL skills than their DOD peers. This emphasizes the need for ongoing monitoring of early ASL skills to allow for early identification of needs and adjustment of interventions to address ASL development.

Vocal usage learning and vocal comprehension learning in harbor seals

BackgroundWhich mammals show vocal learning abilities, e.g., can learn new sounds, or learn to use sounds in new contexts? Vocal usage and comprehension learning are submodules of vocal learning. Specifically, vocal usage learning is the ability to learn to use a vocalization in a new context; vocal comprehension learning is the ability to comprehend a vocalization in a new context. Among mammals, harbor seals (Phoca vitulina) are good candidates to investigate vocal learning. Here, we test whether harbor seals are capable of vocal usage and comprehension learning.ResultsWe trained two harbor seals to (i) switch contexts from a visual to an auditory cue. In particular, the seals first produced two vocalization types in response to two hand signs; they then transitioned to producing these two vocalization types upon the presentation of two distinct sets of playbacks of their own vocalizations. We then (ii) exposed the seals to a combination of trained and novel vocalization stimuli. In a final experiment, (iii) we broadcasted only novel vocalizations of the two vocalization types to test whether seals could generalize from the trained set of stimuli to only novel items of a given vocal category. Both seals learned all tasks and took ≤ 16 sessions to succeed across all experiments. In particular, the seals showed contextual learning through switching the context from former visual to novel auditory cues, vocal matching and generalization. Finally, by responding to the played-back vocalizations with distinct vocalizations, the animals showed vocal comprehension learning.ConclusionsIt has been suggested that harbor seals are vocal learners; however, to date, these observations had not been confirmed in controlled experiments. Here, through three experiments, we could show that harbor seals are capable of both vocal usage and comprehension learning.

Community-based interventions for mitigating cognitive-linguistic decline in older adults.

This study focuses on exploring the effectiveness of interventions in two community-based language rehabilitation care centers located in an economically underdeveloped region of northwest China. The research focuses on 35 older adults with cognitive-linguistic decline enrolled in both centers. The interventions provided to the participants include auditory training, sign language use enhancement, second language interest classes, and the promotion of elderly-mode applications on smartphones. The interventions were particularly effective in enhancing auditory processing and language comprehension skills. Additionally, the promotion of elderly-mode applications on smartphones proved to be a valuable tool in improving participants' language cognition. The study underscores the importance of tailored community-based interventions in improving language skills and cognition and provides a basis for further research in this area.

A lightweight network-based sign language robot with facial mirroring and speech system

Human–robot interaction is an essential capability for humanoid robots to enter the physical world and become companions in people’s lives, learning, and work. While the majority of current research focuses on the voice-based interactions of robots, yet over 60% of communication occurs through nonverbal behaviors, such as facial expressions and hand gestures. Endowing robots with the ability to communicate through nonverbal behavior not only enhances the interactive experience with robots but also provides a potential communication tool for individuals with hearing or speech impairments. Here, we develop a humanoid robot capable of adjusting facial movements by driving servos, and design a novel framework for the robot to integrate sign language recognition and facial landmark detection algorithms. This framework facilitates the robot recognize sign language and translate it into spoken language, while also imitating the facial expressions of the signers. To achieve this, we also propose a lightweight deep learning network called RealTimeSignNet for real-time sign language recognition. Leveraging lightweight 3D convolution modules and time-dependent constraints, this model adapts to various time scales, ensuring efficient processing of sign language recognition tasks. Experimental results demonstrate the outstanding performance of the RealTimeSignNet model on mainstream sign language datasets, achieving an accuracy of 88.1% on the large continuous sign language dataset (continuous SLR), 98.2% on the isolated sign language dataset (SLR 500), and 91.50% on the English sign language dataset (WLAS). The overall assessment demonstrates that our humanoid robot is capable of recognizing sign language and translating it into spoken language, while imitating the facial emotions, providing a comprehensive solution to the communication challenges faced by individuals with hearing and speech impairments.

A wearable sign language translation device utilizing silicone-hydrogel hybrid triboelectric sensor arrays and machine learning

Sign language is a crucial communication tool for the hearing impaired to interact with the outside world. However, the low prevalence of sign language leads to significant communication barriers between the hearing impaired and others. These barriers can be alleviated by using electronic sign language translation devices, but such devices typically face challenges related to their bulkiness, rigidity, and high cost. Herein, we present a cost-effective, flexible, and wearable device designed for the interpretation of sign language, leveraging machine learning algorithms to transform sign language movements into spoken language accurately. Our device is equipped with flexible and stretchable triboelectric sensor (FS-TS) arrays and a printed circuit board for efficient signal processing and wireless transmission. These FS-TSs are constructed from cost-effective materials including silicone, hydrogel, and fluorinated ethylene propylene (FEP) powders, which, while affordable, ensure rapid response and heightened sensitivity. By analyzing 1,000 sets of sign language gestures with machine learning, our system has achieved an impressive recognition accuracy of 98.5%. This achievement underscores the potential of our system as an economically viable and scalable solution for enhancing sign language recognition within the wearable electronics domain.

Differences in biological motion perception associated with hearing status and age of signed language exposure.

This study investigates how American Sign Language (ASL) fluency and hearing status influence the perception of biological motion, using three point-light display (PLD) tasks. Prior research indicates that early exposure to ASL among deaf signers results in more rapid and effortless recognition of biological motion than hearing nonsigners, potentially due to the expertise in deciphering complex human movements or possibly due to neuroplasticity in deaf brains. However, it remains uncertain whether this advantage stems from signed language proficiency or the experience of being deaf. To explore this, we designed three PLD tasks involving viewing randomly moving dots, identifying a person from biological motion PLDs, and determining whether right-side up and inverted PLDs depict actions involving a ball. A diverse cohort of participants (N = 224) with varying ASL fluencies and hearing statuses completed the tasks online, providing us with reaction time and accuracy data. Our results demonstrate that earlier ASL exposure is associated with accuracy, especially on complex action identification tasks. Furthermore, we discovered robust evidence for a speed-accuracy trade-off in deaf participants, in which they performed more quickly but less accurately. The speed-accuracy trade-off was evident in the most difficult task, the action identification task. Further analysis of this deaf group revealed that earlier signed language acquisition led to higher accuracy in action identification task. We conclude that age of ASL exposure and hearing status both significantly contribute to variations in biological motion perception, with implications for understanding visual expertise and cognitive processing in both deaf and signing populations. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Enhancing sign language recognition and accessibility for the deaf community in China

This paper explores the critical role of sign language in enabling communication and social inclusion for the deaf community in China, where approximately 27.8 million individuals experience hearing impairments. Despite the recognition of linguistic human rights, Chinese law lacks formal acknowledgment of sign language, leading to fragmented regulations and insufficient support. The study highlights deficiencies in existing legislation and proposes recommendations for enhancing legal frameworks to protect and promote the rights of deaf individuals. Additionally, the paper examines the disparities between local and national sign languages and underscores the potential of technological advancements, such as sign language translation gloves, in bridging communication gaps and fostering social integration.

Providing accessible health information for people with disability in a public health crisis: A qualitative study of the experiences of Australian accessible information provider organizations during the COVID-19 pandemic

BackgroundDuring the COVID-19 pandemic, people with disability had a right and an acute need to access accessible COVID-19 information, in formats such as sign language, Braille, large print and Easy Read/English. Yet such information was not always provided and many people with disability were without sufficient accessible information. ObjectiveThis paper explores the experiences of Australian accessible information provider organizations of producing accessible information during the pandemic. The intent is to understand why insufficient accessible information reached people with disability. MethodsIn-depth interviews were conducted with 17 accessible information provider organizations and one other stakeholder in Australia, and thematically analyzed. ResultsThe findings show that accessible information provider organizations felt an immense depth and breadth of responsibility for providing accessible information to people with disability during the pandemic. However, they were hampered by constantly changing information; workforce challenges; and a lack of financial, logistic and partnership assistance. Other notable difficulties included having no source of clear/accurate information to translate to accessible formats and not having medical/health expertise themselves. ConclusionThe findings have implications for better preparation for accessible information access in future public health crises. There should be greater funding and logistic support for accessible information provider organizations. Provision of clear/accurate information and subject matter checks of accessible products would be helpful. Responsibility for this should be scoped into existing outreach, education and communications roles – for example, in local health services, general practice clinics and government health departments.

Transfer learning with YOLOV8 for real-time recognition system of American Sign Language Alphabet

Sign language serves as a sophisticated means of communication vital to individuals who are deaf or hard of hearing, relying on hand movements, facial expressions, and body language to convey nuanced meaning. American Sign Language (ASL) exemplifies this linguistic complexity with its distinct grammar and syntax. The advancement of real-time ASL gesture recognition has explored diverse methodologies, including motion sensors and computer vision techniques. This study specifically addresses the recognition of ASL alphabet gestures using computer vision through Mediapipe for hand movement tracking and YOLOv8 for training the deep learning model. The model achieved notable performance metrics: precision of 98%, recall rate of 98%, F1 score of 99%, mean Average Precision (mAP) of 98%, and mAP50-95 of 93%, underscoring its exceptional accuracy and sturdy capabilities.

A large corpus for the recognition of Greek Sign Language gestures

Sign language recognition (SLR) from videos constitutes a captivating problem in gesture recognition, requiring the interpretation of hand movements, facial expressions, and body postures. The complexity of sign formation, signing variability among signers, and the technical hurdles of visual detection and tracking render SLR a challenging task. At the same time, the scarcity of large-scale SLR datasets, which are critical for developing robust data-intensive deep-learning SLR models, exacerbates these issues. In this article, we introduce a multi-signer video corpus of Greek Sign Language (GSL), which is the largest GSL database to date, serving as a valuable resource for SLR research. This corpus comprises an extensive RGB+D video collection that conveys rich lexical content in a multi-modal fashion, encompassing three subsets: (i) isolated signs; (ii) continuous signing; and (iii) continuous alphabet fingerspelling of words. Moreover, we introduce a comprehensive experimental setup that paves the way for more accurate and robust SLR solutions. In particular, except for the multi-signer (MS) and signer-independent (SI) settings, we employ a signer-adapted (SA) experimental paradigm, facilitating a comprehensive evaluation of system performance across various scenarios. Further, we provide three baseline SLR systems for isolated signs, continuous signing, and continuous fingerspelling. These systems leverage cutting-edge methods in deep learning and sequence modeling to capture the intricate temporal dynamics inherent in sign gestures. The models are evaluated on the three corpus subsets, setting their state-of-the-art recognition benchmark. The SL-ReDu GSL corpus, including its recommended experimental frameworks, is publicly available at https://sl-redu.e-ce.uth.gr/corpus.

Automatic sign language translation: a review of neural network methods for recognition and synthesis of spoken and signed language

Automatic sign language translation: a review of neural network methods for recognition and synthesis of spoken and signed language

Translation of a story into Chilean Sign Language: productions by deaf co-teachers

IntroductionThe incorporation of narratives in Chilean Sign Language enriches classroom practices, preserving and transmitting Deaf culture. However, that within the Chilean educational context, the narratives used for instructing Deaf students are translations from written Spanish. This study aims to describe and analyze the narrative structure and the use of Highly Iconic Structures in these translations conducted by Deaf co-teachers into Chilean Sign Language.MethodThe research adopts a qualitative approach and a descriptive case study design, involving two deaf teachers from an Inclusive Education Program in a school. The data analyzed focus on the video-recorded LSCh translations of the short story “The Greedy Squirrel”. Manual annotations were made on the corpus gloss transcription to first segment by narrative structure for description and then to identify the transfer operations.ResultsThe co-teachers make variations to the narrative structure of the translated text and incorporate specific visual-gestural elements such as the change of narrator to first person; the 45% of the translation is composed of transfer operations.DiscussionThese findings suggest that the variations in structure and the new visual-gestural information provided by the co-teachers reflect their understanding and use of sign language narrative norms for translation. The use of transfer operations enhances the storytelling experience, although it cannot be conclusively stated how and why teachers decide to use them in translation. The results are limited, as they require comparison with other corpora on translations from written Spanish to LSCh and software-assisted analysis to standardize data, which implies further research on the topic. For now, it is essential that the time allocated to the preparation of educational materials in LSCh be more extensive and better planned.

ViSL model: The model automatically generates sentences of Vietnamese sign language

ViSL model: The model automatically generates sentences of Vietnamese sign language

For a sustainable education in Brazilian Library Science

The study starts from the global goals of inequality reduction and legal frameworks towards people with disabilities to evaluate curricular components and institutional guidance of active undergraduate courses in Library Science in the southeast area of Brazil. This study, which is of an applied nature, aims to investigate if and how universities offer education oriented to qualification in and towards diversity. The empirical field consists of 22 (twenty-two) courses certified by the Ministry of Education of the Federal Government of Brazil. Once these courses were submitted to documental analysis, we located 13 (thirteen) political-pedagogic projects and 9 (nine) summarized course syllabi to execute the exploratory reading based on keywords related to the inclusion domain and later definition of analytical categories. The reading, analysis, and critical interpretation regarding the results show significant gaps in the librarian education about diversity. Disregarding the predominance of optional courses on Brazilian Sign Language (Libras), the undergraduate courses of the investigated area as an empirical field of this study fail to present, satisfactorily, components or specific objectives aimed toward the learning process of people with disabilities.

Editorial: Bridging the Gap: AI and Sign Language Recognition– A Path Toward Inclusive Communication

The integration of Artificial Intelligence (AI) and sign language recognition is a hot topic in the field of AI+Science, aimed at addressing communication barriers faced by the deaf and hard-of-hearing communities. This paper examines the integration of AI with sign language recognition, highlighting its potential to bridge communication gaps for the deaf and hard-of-hearing. It reviews the evolution of sign language recognition from data gloves to computer vision and underscores the role of extensive databases. The paper also discusses the benefits of multi-modal AI models in enhancing recognition accuracy. It highlights the importance of government and industry support, ethical data practices, and user-centered design in advancing this technology. The challenges and opportunities of integrating this technology into daily life, including technical, interface, and ethical considerations, are explored, emphasizing the need for user-focused solutions and innovative technical approaches.

Biodegradable, stretchable, and high-performance triboelectric nanogenerators through interfacial polarization in bilayer structure

The increasing demand for wearable electronics has led to the development of triboelectric nanogenerators (TENGs) as a promising energy harvesting and sensing technology. However, conventional TENGs often utilize non-biodegradable materials, contributing to environmental pollution. In this work, we present a stretchable and biodegradable TENG based on hydroxyethyl cellulose (HEC) and gelatin (HG-TENG). The HG-TENG features a bilayered structure, where the large difference in their relative permittivity between HEC and gelatin induces interfacial polarization, effectively mitigating charge recombination and enhancing triboelectric performance. The optimized HG-TENG achieves an open-circuit voltage (Voc) of 93 V, a maximum power density of 57.8 µW/cm2, and can power 38 blue light-emitting diodes. The device exhibits a stretchability of 150 % and biodegrades within 3 hours in phosphate-buffered saline. Furthermore, we demonstrate the application of the HG-TENG as a wearable sensor by modifying it with trichloro(1H, 1H, 2H, 2H-perfluorooctyl)silane (FOTS) (FHG-TENG). The FHG-TENG-based smart glove, integrated with machine learning algorithms, enables real-time monitoring of blood pressure waveforms and finger motions, showcasing its potential for human-machine interfaces. The smart glove, equipped with five FHG-TENGs on the proximal interphalangeal joints of each finger, detects diverse finger gestures and generates voltage signals that control a robotic hand in real-time, demonstrating effective human-machine interaction through synchronized motion. Moreover, the smart glove achieves a high recognition accuracy of 96.15 % for 10 different hand sign languages.

Generalizations of Wearable Device Placements and Sentences in Sign Language Recognition With Transformer-Based Model

Generalizations of Wearable Device Placements and Sentences in Sign Language Recognition With Transformer-Based Model

Translation of Mexican sign language into Spanish using machine learning and internet of things devices

Translation of Mexican sign language into Spanish using machine learning and internet of things devices

Kurdish Sign Language Recognition Using Convolutional Neural Network (CNN)

The present study examines the obstacles encountered by the deaf population, with a specific emphasis on the increasing significance of sign language in facilitating effective communication. The main mode of communication for deaf persons is Sign Language (SL), which is a visual and expressive method of communicating meaning via facial expressions, hand motions, and body gestures. The objective of this project is to automate the identification of sign language in order to improve accessibility and decrease reliance on interpreters. Specifically, the work focused on constructing an alphabet recognition system for Kurdish Sign Language (KSL). Due to its many intricacies and resemblances to the Arabic script, KSL requires a strong recognition model. The proposed method utilizes Convolutional Neural Networks (CNN) with an actual dataset to accurately recognize both numerical values and alphabetic characters in the Korean Sign Language (KSL). Real-time operation allows for rapid recognition of hand gestures and subsequent provision of textual output. The collection comprises 132,000 hand pictures, including 33 alphabetica and (0-9) numeral signs. Significantly, the use of MediaPipe, a method for processing 3D images, greatly improves the efficiency of detection. Multiple methodologies were investigated, and the integration of Convolutional Neural Networks (CNN), TensorFlow, and MediaPipe resulted in a remarkable accuracy of 99.87% with negligible dropout rates. This study establishes a foundation for enhanced communication and independence for the deaf population, representing a notable advancement in the automation of sign language recognition.