American Sign Language Signs Research Articles

Production of real signs but not pseudosigns affected by age of acquisition in American Sign Language.

Research shows that insufficient language access in early childhood significantly affects language processing. While the majority of this work focuses on syntax, phonology also appears to be affected, though it is unclear exactly how. Here we investigated phonological production across age of acquisition of American Sign Language (ASL). Participants were deaf adult signers who first learned ASL at ages ranging from birth to 14years and they performed both lexical decisions and repetitions of ASL signs and pseudosigns. Because phonological production has been understudied across age of acquisition, we were particularly interested in production accuracy for the sublexical phonological parameters of handshape, movement, and location. Lexical decision responses were slower and more accurate for impossible pseudosigns compared with possible pseudosigns, indicating participants were sensitive to ASL phonological structure regardless of age of acquisition. Despite this, age of acquisition affected repetition accuracy. Handshape errors were highest for those with earlier ages of acquisition, but movement errors were highest for those with later ages of acquisition, though this effect of age of acquisition was only seen for real ASL signs and not pseudosigns. The parameter error pattern for pseudosigns was not affected by age of acquisition. These results indicate that later age of acquisition does not inhibit the ability to produce ASL phonology but ultimately alters the processing of the phonological parameters when meaning and phonology are integrated.

The ASL Dataset for Real-Time Recognition and Integration with LLM Services

This study aims to investigate the impact of hand gesture recognition techniques on the efficiency of American Sign Language (ASL) interpretation, addressing a critical gap in the existing literature. The research seeks new insights into the challenges of automated sign language recognition, contributing to a deeper understanding of accessibility in communication for the deaf and hard-of-hearing community. The study employs a quantitative approach, using a dataset comprising hand gesture images representing the static letters of the ASL alphabet collected from multiple users. Data were collected from various individuals to ensure diversity and analyzed using machine learning models to evaluate their effectiveness in recognizing ASL signs. The results reveal that the machine learning models implemented achieved a high accuracy rate in recognizing hand gestures, indicating that person-specific variations do not significantly hinder performance. These findings provide evidence that the proposed dataset and methodologies can improve the reliability of sign language recognition systems, offering significant implications for the development of more inclusive communication technologies. This research offers a novel perspective on sign language recognition, providing valuable insight that extends the current understanding of gesture-based communication systems. The study’s findings contribute to advancements in accessibility technologies, highlighting areas for future research and practical applications in improving communication for the Deaf and hard of hearing community.

Argument ordering in simple sentences is affected by age of first language acquisition: Evidence from late first language signers of ASL.

Research on the language acquisition of deaf individuals who are exposed to accessible linguistic input at a variety of ages has provided evidence for a sensitive period of first language acquisition. Recent studies have shown that deaf individuals who first learn language after early childhood, late first-language learners (LL1), do not comprehend reversible Subject-Verb-Object (SVO) sentences. The present study analyzed 478 signed productions elicited with pictures depicting simple events with one or two arguments by 28 signers. The argument order patterns of native signers converged with one another and the word order patterns of American Sign Language (ASL). By contrast, the ordering patterns of the LL1 signers did not converge with one another or with the patterns of the native signers. This indicates that early childhood is a period of heightened sensitivity to basic word order and may help explain why complex structures are difficult for LL1 signers to learn.

Advancing human-computer interaction: AI-driven translation of American Sign Language to Nepali using convolutional neural networks and text-to-speech conversion application

Advanced technology that serves people with impairments is severely lacking in Nepal, especially when it comes to helping the hearing impaired communicate. Although sign language is one of the oldest and most organic ways to communicate, there aren't many resources available in Nepal to help with the communication gap between Nepali and American Sign Language (ASL). This study investigates the application of Convolutional Neural Networks (CNN) and AI-driven methods for translating ASL into Nepali text and speech to bridge the technical divide. Two pre-trained transfer learning models, ResNet50 and VGG16, were refined to classify ASL signs using extensive ASL image datasets. The system utilizes the Python gTTS package to translate signs into Nepali text and speech, integrating with an OpenCV video input TKinter-based Graphical User Interface (GUI). With both CNN architectures, the model's accuracy of over 99 % allowed for the smooth conversion of ASL to speech output. By providing a workable solution to improve inclusion and communication, the deployment of an AI-driven translation system represents a significant step in lowering the technological obstacles that disabled people in Nepal must overcome.

An ERP investigation of perceptual vs motoric iconicity in sign production

The type of form-meaning mapping for iconic signs can vary. For perceptually-iconic signs there is a correspondence between visual features of a referent (e.g., the beak of a bird) and the form of the sign (e.g., extended thumb and index finger at the mouth for the American Sign Language (ASL) sign BIRD). For motorically-iconic signs there is a correspondence between how an object is held/manipulated and the form of the sign (e.g., the ASL sign FLUTE depicts how a flute is played). Previous studies have found that iconic signs are retrieved faster in picture-naming tasks, but type of iconicity has not been manipulated. We conducted an ERP study in which deaf signers and a control group of English speakers named pictures that targeted perceptually-iconic, motorically-iconic, or non-iconic ASL signs. For signers (unlike the control group), naming latencies varied by iconicity type: perceptually-iconic < motorically-iconic < non-iconic signs. A reduction in the N400 amplitude was only found for the perceptually-iconic signs, compared to both non-iconic and motorically-iconic signs. No modulations of N400 amplitudes were observed for the control group. We suggest that this pattern of results arises because pictures eliciting perceptually-iconic signs can more effectively prime lexical access due to greater alignment between features of the picture and the semantic and phonological features of the sign. We speculate that naming latencies are facilitated for motorically-iconic signs due to later processes (e.g., faster phonological encoding via cascading activation from semantic features). Overall, the results indicate that type of iconicity plays role in sign production when elicited by picture-naming tasks.

DeafTech Vision: A Visual Computer's Approach to Accessible Communication through Deep Learning-Driven ASL Analysis

Sign language is commonly used by people with hearing and speech impairments, making it difficult for those without such disabilities to understand. However, sign language is not limited to communication within the deaf community alone. It has been officially recognized in numerous countries and is increasingly being offered as a second language option in educational institutions. In addition, sign language has shown its usefulness in various professional sectors, including interpreting, education, and healthcare, by facilitating communication between people with and without hearing impairments. Advanced technologies, such as computer vision and machine learning algorithms, are used to interpret and translate sign language into spoken or written forms. These technologies aim to promote inclusivity and provide equal opportunities for people with hearing impairments in different domains, such as education, employment, and social interactions. In this paper, we implement a DeafTech Vision (DTV-CNN) architecture based on the convolutional neural network to recognize American Sign Language (ASL) gestures using deep learning techniques. Our main objective is to develop a robust ASL sign classification model to enhance human-computer interaction and assist individuals with hearing impairments. Through extensive evaluation, our model consistently outperformed baseline methods in terms of precision. It achieved an outstanding accuracy rate of 99.87% on the ASL alphabet test dataset and 99.94% on the ASL digit dataset, significantly exceeding previous research, which reported an accuracy of 90.00%. We also illustrated the model's learning trends and convergence points using loss and error graphs. These results highlight the DTV-CNN's effectiveness and capability in distinguishing complex ASL gestures.

Multi-culture Sign Language Detection and Recognition Using Fine-tuned Convolutional Neural Network

The speech and hearing-impaired community use sign language as the primary means of communication. It is quite challenging for the general population to interpret or learn sign language completely. A sign language recognition system must be designed and developed to address this communication barrier. Most current sign language recognition systems rely on wearable sensors, keeping the recognition system unaffordable for most individuals. Moreover, the existing vision-based sign recognition frameworks do not consider all of the spatial and temporal information required for accurate recognition. A novel vision-based hybrid deep neural net methodology is proposed in this project for recognizing American Sign Language (ASL) and custom sign gestures. The proposed framework aims to establish a single framework for tracking and extracting multi-semantic properties, such as non-manual components and manual co-articulations. Furthermore, spatial feature extraction from the sign gestures is deployed using a Hybrid Deep Neural Network (HDNN) with atrous convolutions. The temporal and sequential feature extraction is carried out by employing attention-based HDNN. In addition, the distinguished abstract feature extraction is done using modified autoencoders. The discriminative feature extraction for differentiating the sign gestures from unwanted transition gestures is done by leveraging the hybrid attention module. The experimentation of the proposed model has been carried out on the novel multi-signer ASL and custom sign language dataset. The proposed sign language recognition framework with hybrid neural nets, specifically using HDNN, yields better results than other state-of-the-art frameworks. Additionally, a detection module is incorporated using Flask web, allowing manual input of images/signs for real-time recognition.

Interactive learning of natural sign language with radar

AbstractOver the past decade, there have been great advancements in radio frequency sensor technology for human–computer interaction applications, such as gesture recognition, and human activity recognition more broadly. While there is a significant amount of study on these topics, in most cases, experimental data are acquired in controlled settings by directing participants what motion to articulate. However, especially for communicative motions, such as sign language, such directed data sets do not accurately capture natural, in situ articulations. This results in a difference in the distribution of directed American Sign Language (ASL) versus natural ASL, which severely degrades natural sign language recognition in real‐world scenarios. To overcome these challenges and acquire more representative data for training deep models, the authors develop an interactive gaming environment, ChessSIGN, which records video and radar data of participants as they play the game without any external direction. The authors investigate various ways of generating synthetic samples from directed ASL data, but show that ultimately such data does not offer much improvement over just initialising using imagery from ImageNet. In contrast, an interactive learning paradigm is proposed by the authors in which model training is shown to improve as more and more natural ASL samples are acquired and augmented via synthetic samples generated from a physics‐aware generative adversarial network. The authors show that the proposed approach enables the recognition of natural ASL in a real‐world setting, achieving an accuracy of 69% for 29 ASL signs—a 60% improvement over conventional training with directed ASL data.

Turn-Taking Machinery in a Japanese Sign Language Triadic Conversation in an Online Environment

Abstract: This article explores turn-taking in Japanese Sign Language (JSL) by using Baker's (1977) framework. JSL as a language is wholly unrelated to American Sign Language (ASL), but because Baker and other discourse analysts have always been concerned mostly with ASL and European sign languages, it remains to be seen if Asian sign languages such as JSL have similar discourse mechanisms at work. Baker introduces three types of regulators that manage who has the floor in ASL conversations: initiation regulators, continuation regulators, and shift regulators. The sequential analysis, which looks closely at each case of regulatory behavior as they happen in chronological order, will be used. The video analyzed consists of three fluent deaf JSL users recorded on a virtual platform. While Baker's turn-taking system holds water for the most part, there are some crucial differences that deserve further examination. As the recording was done remotely rather than in person, a turn-taking mechanism not made explicit by Baker has been observed, and an addition to the sign language turn-taking process is put forward.

Better letter: iconicity in the manual alphabets of American Sign Language and Swedish Sign Language

Abstract While iconicity has sometimes been defined as meaning transparency, it is better defined as a subjective phenomenon bound to an individual’s perception and influenced by their previous language experience. In this article, I investigate the subjective nature of iconicity through an experiment in which 72 deaf, hard-of-hearing and hearing (signing and non-signing) participants rate the iconicity of individual letters of the American Sign Language (ASL) and Swedish Sign Language (STS) manual alphabets. It is shown that L1 signers of ASL and STS rate their own (L1) manual alphabet as more iconic than the foreign one. Hearing L2 signers of ASL and STS exhibit the same pattern as L1 signers, showing an iconic preference for their own (L2) manual alphabet. In comparison, hearing non-signers show no general iconic preference for either manual alphabet. Across all groups, some letters are consistently rated as more iconic in one sign language than the other, illustrating general iconic preferences. Overall, the results align with earlier findings from sign language linguistics that point to language experience affecting iconicity ratings and that one’s own signs are rated as more iconic than foreign signs with the same meaning, even if similar iconic mappings are used.

ASL champ!: a virtual reality game with deep-learning driven sign recognition

We developed an American Sign Language (ASL) learning platform in a Virtual Reality (VR) environment to facilitate immersive interaction and real-time feedback for ASL learners. We describe the first game to use an interactive teaching style in which users learn from a fluent signing avatar and the first implementation of ASL sign recognition using deep learning within the VR environment. Advanced motion-capture technology powers an expressive ASL teaching avatar within an immersive three-dimensional environment. The teacher demonstrates an ASL sign for an object, prompting the user to copy the sign. Upon the user’s signing, a third-party plugin executes the sign recognition process alongside a deep learning model. Depending on the accuracy of a user’s sign production, the avatar repeats the sign or introduces a new one. We gathered a 3D VR ASL dataset from fifteen diverse participants to power the sign recognition model. The proposed deep learning model’s training, validation, and test accuracy are 90.12%, 89.37%, and 86.66%, respectively. The functional prototype can teach sign language vocabulary and be successfully adapted as an interactive ASL learning platform in VR.

A helping hand

In this article, I examine responses to the taking of food items during snack time in an early childhood education classroom with deaf toddlers (18 months to three years old) who are native signers of American Sign Language (ASL). These children have grown up with exposure to ASL from deaf family members and are attending a classroom where all individuals use ASL. Through the combined analysis of ethnographic and interactional data, I argue that the teachers’ corporeal socialization of the deaf toddlers into a visual orientation leads to their development of a social and moral understanding of ownership rights within the classroom which is displayed in the children’s social awareness, social responsiveness and self-reliance in responding to food takings.

Multi-Modal Multi-Channel American Sign Language Recognition

In this paper, we propose a machine learning-based multi-stream framework to recognize American Sign Language (ASL) manual signs and nonmanual gestures (face and head movements) in real time from RGB-D videos. Our approach is based on 3D Convolutional Neural Networks (3D CNNs) by fusing the multi-modal features including hand gestures, facial expressions, and body poses from multiple channels (RGB, Depth, Motion, and Skeleton joints). To learn the overall temporal dynamics in a video, a proxy video is generated by selecting a subset of frames for each video which are then used to train the proposed 3D CNN model. We collected a new ASL dataset, ASL-100-RGBD, which contains 42 RGB-D videos captured by a Microsoft Kinect V2 camera. Each video consists of 100 ASL manual signs, along with RGB channel, Depth maps, Skeleton joints, Face features, and HD face. The dataset is fully annotated for each semantic region (i.e. the time duration of each sign that the human signer performs). Our proposed method achieves 92.88% accuracy for recognizing 100 ASL sign glosses in our newly collected ASL-100-RGBD dataset. The effectiveness of our framework for recognizing hand gestures from RGB-D videos is further demonstrated on a large-scale dataset, ChaLearn IsoGD, achieving the state-of-the-art results.

Poster Session I: Divided attention in Sign Language Recognition.

Previous work suggests that the brain has a limited ability to process multiple visual stimuli during divided attention: for example, people can recognize only one word at a time (White, Palmer, & Boynton, Psych Science 2018). Here, we examine whether American Sign Language (ASL) experience affects divided attention for stimuli: do signers & non-signers differ in their ability to process two signs at once? In a probe recognition paradigm, participants were presented with two letter signs, one or both of which were pre-cued as relevant (single- and dual-task conditions, respectively), and then responded whether a probe sign matched the cued sign(s). The dual-task deficit is the difference in performance between the single- and dual-task conditions and measures the cost of dividing attention. Preliminary data show that hearing non-signers and signers had a similar dual-task deficit (11.8% ± 2.2%, n = 5 vs. 11.4% ± 1.5%, n = 6), with no significant difference between the groups (t(7.27) = 0.18, p = 0.86). The magnitude of these divided attention effects is consistent with processing limits observed for object judgments (e.g. Popovkina, Palmer, Moore, & Boynton, JoV 2021). Thus, these preliminary results suggest that the attentional demands of ASL sign processing are similar in signers and non-signers.

L2 Learners’ Signed Language Processing Relates, in Part, to Perspective‐Taking Skills

AbstractThe comprehension of signed language requires linguistic and visual–spatial processing, such as perspective‐taking for correctly interpreting the layout of a spatial scene. However, little is known about how adult second‐language (L2) learners process visual–spatial constructions in a signed language that they are studying, including which angles of viewing are most challenging to process and whether there are relationships between perspective‐taking and the comprehension of non‐spatial (i.e., non‐scene based) constructions. We examine the performance of 95 intermediate signers of American Sign Language (ASL) on linguistic and non‐linguistic perspective‐taking tests. Half the participants completed a test of narrative comprehension that included visual–spatial scenes, and half took a test of signed phonological and morphophonological discrimination. Performance on linguistic perspective‐taking correlated moderately with performance on the narrative, but not with the discrimination test. These findings support the claim that perspective‐taking skills are yoked to some—but not all—aspects of signed language learning.

Attitudes and experiences regarding preventive strategies for the deaf population in Western New York.

People for whom English is a second language, such as the deaf population, often have unequal access to health information and low health literacy. In the context of a wider study on risk of tick-borne illness in deaf communities, we explored barriers, opportunities, and nuances to accessible health information and communication among deaf people. Semi-structured qualitative individual and group interviews were conducted with 40 deaf people in upstate New York, to explore factors associated with health literacy and health information accessibility. Interviews were conducted in American Sign Language (ASL) by a deaf researcher fluent in ASL. Data analysis included the translation of ASL signs into English words, systematic coding, and generation of themes. A total of 21 interview events (mean time per interview = 41 minutes) were conducted. Two main themes and multiple sub-themes emerged from the data: 1) Layers of obstacles faced by deaf people confirms (or reinforces) exclusion; and 2) preventive information is unavailable or inaccessible to deaf people. Sub- themes identified in the results were perceptions of the deaf community and deaf culture, complex layers of obstacles faced by deaf individuals, the digital divide, the culture of communication, awareness of tick and tick-borne disease (TBD) diseases, importance of using certified deaf interpreters (CDI), health information dissemination strategies and collaborations with the education system, and physical/virtual community engagement. The data suggested several challenges to health literacy in the deaf population, including healthcare and education inequalities and negative perceptions of deaf people by both deaf and hearing people. Improving health literacy in the deaf population requires more interpreters who themselves are deaf ("certified deaf interpreters"), provision of health information in ASL, and a greater engagement with the deaf population by education and healthcare systems.

Enhancing Sign Language Recognition through Fusion of CNN Models

This study introduces a pioneering hybrid model designed for the recognition of sign language, with a specific focus on American Sign Language (ASL) and Indian Sign Language (ISL). Departing from traditional machine learning methods, the model ingeniously blends hand-crafted techniques with deep learning approaches to surmount inherent limitations. Notably, the hybrid model achieves an exceptional accuracy rate of 96% for ASL and 97% for ISL, surpassing the typical 90-93% accuracy rates of previous models. This breakthrough underscores the efficacy of combining predefined features and rules with neural networks. What sets this hybrid model apart is its versatility in recognizing both ASL and ISL signs, addressing the global variations in sign languages. The elevated accuracy levels make it a practical and accessible tool for the hearing-impaired community. This has significant implications for real-world applications, particularly in education, healthcare, and various contexts where improved communication between hearing-impaired individuals and others is paramount. The study represents a noteworthy stride in sign language recognition, presenting a hybrid model that excels in accurately identifying ASL and ISL signs, thereby contributing to the advancement of communication and inclusivity.

A Deep Learning Approach for ASL Recognition and Text-to-Speech Synthesis using CNN

Abstract: Sign language is a visual language that is used by the deaf and hard-of-hearing community to communicate. However, sign language is not universally understood by non-signers, which can create communication barriers for the deaf and hard-ofhearing individuals. In this paper, we present a novel application for American Sign Language (ASL) to text to speech conversion using deep learning techniques. Our app aims to bridge the communication gap between hearing-impaired individuals who use ASL as their primary mode of communication and individuals who do not understand ASL. The app comprises three main components: a hand gesture recognition module that detects and recognizes ASL signs, a text-tospeech synthesis module that converts recognized ASL signs to text, and a speech synthesis module that converts the text to speech

American Sign Language Recognition and Generation : A CNN-based Approach

Although not a global language, sign language is an essential tool for the deaf community. Communication between these communities and hearing population is severely hampered by this, as human-based interpretation can be both costly and time-consuming. In this paper, we present a real-time American Sign Language (ASL) generation and recognition system that makes use of Convolutional Neural Networks and deep learning (CNNs). Despite differences in lighting, skin tones, and backdrops, our technology is capable of correctly identifying and generating ASL signs. We trained our model on a large dataset of ASL signs in order to obtain a high level of accuracy. Our findings show that, with accuracy rates of 98.53% and 98.84%, respectively, our system achieves high accuracy rates in both training and validation. Our approach uses the advantages of CNNs to accomplish quick and precise recognition of individual letters and words, making it particularly effective for sign fingerspelling recognition. We believe that our technology has the ability to transform communication between the hearing community and the deaf and hard-of-hearing communities by providing a dependable and cost-effective way of sign language interpretation. Our method could help people who use sign language communicate more easily and live better in a range of environments, including schools, hospitals, and public places.

SmartASL

Sign language builds up an important bridge between the d/Deaf and hard-of-hearing (DHH) and hearing people. Regrettably, most hearing people face challenges in comprehending sign language, necessitating sign language translation. However, state-of-the-art wearable-based techniques mainly concentrate on recognizing manual markers (e.g., hand gestures), while frequently overlooking non-manual markers, such as negative head shaking, question markers, and mouthing. This oversight results in the loss of substantial grammatical and semantic information in sign language. To address this limitation, we introduce SmartASL, a novel proof-of-concept system that can 1) recognize both manual and non-manual markers simultaneously using a combination of earbuds and a wrist-worn IMU, and 2) translate the recognized American Sign Language (ASL) glosses into spoken language. Our experiments demonstrate the SmartASL system's significant potential to accurately recognize the manual and non-manual markers in ASL, effectively bridging the communication gaps between ASL signers and hearing people using commercially available devices.