American Sign Language Signs Research Articles

An ERP investigation of perceptual vs motoric iconicity in sign production

The type of form-meaning mapping for iconic signs can vary. For perceptually-iconic signs there is a correspondence between visual features of a referent (e.g., the beak of a bird) and the form of the sign (e.g., extended thumb and index finger at the mouth for the American Sign Language (ASL) sign BIRD). For motorically-iconic signs there is a correspondence between how an object is held/manipulated and the form of the sign (e.g., the ASL sign FLUTE depicts how a flute is played). Previous studies have found that iconic signs are retrieved faster in picture-naming tasks, but type of iconicity has not been manipulated. We conducted an ERP study in which deaf signers and a control group of English speakers named pictures that targeted perceptually-iconic, motorically-iconic, or non-iconic ASL signs. For signers (unlike the control group), naming latencies varied by iconicity type: perceptually-iconic < motorically-iconic < non-iconic signs. A reduction in the N400 amplitude was only found for the perceptually-iconic signs, compared to both non-iconic and motorically-iconic signs. No modulations of N400 amplitudes were observed for the control group. We suggest that this pattern of results arises because pictures eliciting perceptually-iconic signs can more effectively prime lexical access due to greater alignment between features of the picture and the semantic and phonological features of the sign. We speculate that naming latencies are facilitated for motorically-iconic signs due to later processes (e.g., faster phonological encoding via cascading activation from semantic features). Overall, the results indicate that type of iconicity plays role in sign production when elicited by picture-naming tasks.

Multi-culture Sign Language Detection and Recognition Using Fine-tuned Convolutional Neural Network

The speech and hearing-impaired community use sign language as the primary means of communication. It is quite challenging for the general population to interpret or learn sign language completely. A sign language recognition system must be designed and developed to address this communication barrier. Most current sign language recognition systems rely on wearable sensors, keeping the recognition system unaffordable for most individuals. Moreover, the existing vision-based sign recognition frameworks do not consider all of the spatial and temporal information required for accurate recognition. A novel vision-based hybrid deep neural net methodology is proposed in this project for recognizing American Sign Language (ASL) and custom sign gestures. The proposed framework aims to establish a single framework for tracking and extracting multi-semantic properties, such as non-manual components and manual co-articulations. Furthermore, spatial feature extraction from the sign gestures is deployed using a Hybrid Deep Neural Network (HDNN) with atrous convolutions. The temporal and sequential feature extraction is carried out by employing attention-based HDNN. In addition, the distinguished abstract feature extraction is done using modified autoencoders. The discriminative feature extraction for differentiating the sign gestures from unwanted transition gestures is done by leveraging the hybrid attention module. The experimentation of the proposed model has been carried out on the novel multi-signer ASL and custom sign language dataset. The proposed sign language recognition framework with hybrid neural nets, specifically using HDNN, yields better results than other state-of-the-art frameworks. Additionally, a detection module is incorporated using Flask web, allowing manual input of images/signs for real-time recognition.

Interactive learning of natural sign language with radar

AbstractOver the past decade, there have been great advancements in radio frequency sensor technology for human–computer interaction applications, such as gesture recognition, and human activity recognition more broadly. While there is a significant amount of study on these topics, in most cases, experimental data are acquired in controlled settings by directing participants what motion to articulate. However, especially for communicative motions, such as sign language, such directed data sets do not accurately capture natural, in situ articulations. This results in a difference in the distribution of directed American Sign Language (ASL) versus natural ASL, which severely degrades natural sign language recognition in real‐world scenarios. To overcome these challenges and acquire more representative data for training deep models, the authors develop an interactive gaming environment, ChessSIGN, which records video and radar data of participants as they play the game without any external direction. The authors investigate various ways of generating synthetic samples from directed ASL data, but show that ultimately such data does not offer much improvement over just initialising using imagery from ImageNet. In contrast, an interactive learning paradigm is proposed by the authors in which model training is shown to improve as more and more natural ASL samples are acquired and augmented via synthetic samples generated from a physics‐aware generative adversarial network. The authors show that the proposed approach enables the recognition of natural ASL in a real‐world setting, achieving an accuracy of 69% for 29 ASL signs—a 60% improvement over conventional training with directed ASL data.

Turn-Taking Machinery in a Japanese Sign Language Triadic Conversation in an Online Environment

Abstract: This article explores turn-taking in Japanese Sign Language (JSL) by using Baker's (1977) framework. JSL as a language is wholly unrelated to American Sign Language (ASL), but because Baker and other discourse analysts have always been concerned mostly with ASL and European sign languages, it remains to be seen if Asian sign languages such as JSL have similar discourse mechanisms at work. Baker introduces three types of regulators that manage who has the floor in ASL conversations: initiation regulators, continuation regulators, and shift regulators. The sequential analysis, which looks closely at each case of regulatory behavior as they happen in chronological order, will be used. The video analyzed consists of three fluent deaf JSL users recorded on a virtual platform. While Baker's turn-taking system holds water for the most part, there are some crucial differences that deserve further examination. As the recording was done remotely rather than in person, a turn-taking mechanism not made explicit by Baker has been observed, and an addition to the sign language turn-taking process is put forward.

Better letter: iconicity in the manual alphabets of American Sign Language and Swedish Sign Language

Abstract While iconicity has sometimes been defined as meaning transparency, it is better defined as a subjective phenomenon bound to an individual’s perception and influenced by their previous language experience. In this article, I investigate the subjective nature of iconicity through an experiment in which 72 deaf, hard-of-hearing and hearing (signing and non-signing) participants rate the iconicity of individual letters of the American Sign Language (ASL) and Swedish Sign Language (STS) manual alphabets. It is shown that L1 signers of ASL and STS rate their own (L1) manual alphabet as more iconic than the foreign one. Hearing L2 signers of ASL and STS exhibit the same pattern as L1 signers, showing an iconic preference for their own (L2) manual alphabet. In comparison, hearing non-signers show no general iconic preference for either manual alphabet. Across all groups, some letters are consistently rated as more iconic in one sign language than the other, illustrating general iconic preferences. Overall, the results align with earlier findings from sign language linguistics that point to language experience affecting iconicity ratings and that one’s own signs are rated as more iconic than foreign signs with the same meaning, even if similar iconic mappings are used.

ASL champ!: a virtual reality game with deep-learning driven sign recognition

We developed an American Sign Language (ASL) learning platform in a Virtual Reality (VR) environment to facilitate immersive interaction and real-time feedback for ASL learners. We describe the first game to use an interactive teaching style in which users learn from a fluent signing avatar and the first implementation of ASL sign recognition using deep learning within the VR environment. Advanced motion-capture technology powers an expressive ASL teaching avatar within an immersive three-dimensional environment. The teacher demonstrates an ASL sign for an object, prompting the user to copy the sign. Upon the user’s signing, a third-party plugin executes the sign recognition process alongside a deep learning model. Depending on the accuracy of a user’s sign production, the avatar repeats the sign or introduces a new one. We gathered a 3D VR ASL dataset from fifteen diverse participants to power the sign recognition model. The proposed deep learning model’s training, validation, and test accuracy are 90.12%, 89.37%, and 86.66%, respectively. The functional prototype can teach sign language vocabulary and be successfully adapted as an interactive ASL learning platform in VR.

A helping hand

In this article, I examine responses to the taking of food items during snack time in an early childhood education classroom with deaf toddlers (18 months to three years old) who are native signers of American Sign Language (ASL). These children have grown up with exposure to ASL from deaf family members and are attending a classroom where all individuals use ASL. Through the combined analysis of ethnographic and interactional data, I argue that the teachers’ corporeal socialization of the deaf toddlers into a visual orientation leads to their development of a social and moral understanding of ownership rights within the classroom which is displayed in the children’s social awareness, social responsiveness and self-reliance in responding to food takings.

Multi-Modal Multi-Channel American Sign Language Recognition

In this paper, we propose a machine learning-based multi-stream framework to recognize American Sign Language (ASL) manual signs and nonmanual gestures (face and head movements) in real time from RGB-D videos. Our approach is based on 3D Convolutional Neural Networks (3D CNNs) by fusing the multi-modal features including hand gestures, facial expressions, and body poses from multiple channels (RGB, Depth, Motion, and Skeleton joints). To learn the overall temporal dynamics in a video, a proxy video is generated by selecting a subset of frames for each video which are then used to train the proposed 3D CNN model. We collected a new ASL dataset, ASL-100-RGBD, which contains 42 RGB-D videos captured by a Microsoft Kinect V2 camera. Each video consists of 100 ASL manual signs, along with RGB channel, Depth maps, Skeleton joints, Face features, and HD face. The dataset is fully annotated for each semantic region (i.e. the time duration of each sign that the human signer performs). Our proposed method achieves 92.88% accuracy for recognizing 100 ASL sign glosses in our newly collected ASL-100-RGBD dataset. The effectiveness of our framework for recognizing hand gestures from RGB-D videos is further demonstrated on a large-scale dataset, ChaLearn IsoGD, achieving the state-of-the-art results.

Poster Session I: Divided attention in Sign Language Recognition.

Previous work suggests that the brain has a limited ability to process multiple visual stimuli during divided attention: for example, people can recognize only one word at a time (White, Palmer, & Boynton, Psych Science 2018). Here, we examine whether American Sign Language (ASL) experience affects divided attention for stimuli: do signers & non-signers differ in their ability to process two signs at once? In a probe recognition paradigm, participants were presented with two letter signs, one or both of which were pre-cued as relevant (single- and dual-task conditions, respectively), and then responded whether a probe sign matched the cued sign(s). The dual-task deficit is the difference in performance between the single- and dual-task conditions and measures the cost of dividing attention. Preliminary data show that hearing non-signers and signers had a similar dual-task deficit (11.8% ± 2.2%, n = 5 vs. 11.4% ± 1.5%, n = 6), with no significant difference between the groups (t(7.27) = 0.18, p = 0.86). The magnitude of these divided attention effects is consistent with processing limits observed for object judgments (e.g. Popovkina, Palmer, Moore, & Boynton, JoV 2021). Thus, these preliminary results suggest that the attentional demands of ASL sign processing are similar in signers and non-signers.

L2 Learners’ Signed Language Processing Relates, in Part, to Perspective‐Taking Skills

AbstractThe comprehension of signed language requires linguistic and visual–spatial processing, such as perspective‐taking for correctly interpreting the layout of a spatial scene. However, little is known about how adult second‐language (L2) learners process visual–spatial constructions in a signed language that they are studying, including which angles of viewing are most challenging to process and whether there are relationships between perspective‐taking and the comprehension of non‐spatial (i.e., non‐scene based) constructions. We examine the performance of 95 intermediate signers of American Sign Language (ASL) on linguistic and non‐linguistic perspective‐taking tests. Half the participants completed a test of narrative comprehension that included visual–spatial scenes, and half took a test of signed phonological and morphophonological discrimination. Performance on linguistic perspective‐taking correlated moderately with performance on the narrative, but not with the discrimination test. These findings support the claim that perspective‐taking skills are yoked to some—but not all—aspects of signed language learning.

Attitudes and experiences regarding preventive strategies for the deaf population in Western New York.

People for whom English is a second language, such as the deaf population, often have unequal access to health information and low health literacy. In the context of a wider study on risk of tick-borne illness in deaf communities, we explored barriers, opportunities, and nuances to accessible health information and communication among deaf people. Semi-structured qualitative individual and group interviews were conducted with 40 deaf people in upstate New York, to explore factors associated with health literacy and health information accessibility. Interviews were conducted in American Sign Language (ASL) by a deaf researcher fluent in ASL. Data analysis included the translation of ASL signs into English words, systematic coding, and generation of themes. A total of 21 interview events (mean time per interview = 41 minutes) were conducted. Two main themes and multiple sub-themes emerged from the data: 1) Layers of obstacles faced by deaf people confirms (or reinforces) exclusion; and 2) preventive information is unavailable or inaccessible to deaf people. Sub- themes identified in the results were perceptions of the deaf community and deaf culture, complex layers of obstacles faced by deaf individuals, the digital divide, the culture of communication, awareness of tick and tick-borne disease (TBD) diseases, importance of using certified deaf interpreters (CDI), health information dissemination strategies and collaborations with the education system, and physical/virtual community engagement. The data suggested several challenges to health literacy in the deaf population, including healthcare and education inequalities and negative perceptions of deaf people by both deaf and hearing people. Improving health literacy in the deaf population requires more interpreters who themselves are deaf ("certified deaf interpreters"), provision of health information in ASL, and a greater engagement with the deaf population by education and healthcare systems.

Enhancing Sign Language Recognition through Fusion of CNN Models

This study introduces a pioneering hybrid model designed for the recognition of sign language, with a specific focus on American Sign Language (ASL) and Indian Sign Language (ISL). Departing from traditional machine learning methods, the model ingeniously blends hand-crafted techniques with deep learning approaches to surmount inherent limitations. Notably, the hybrid model achieves an exceptional accuracy rate of 96% for ASL and 97% for ISL, surpassing the typical 90-93% accuracy rates of previous models. This breakthrough underscores the efficacy of combining predefined features and rules with neural networks. What sets this hybrid model apart is its versatility in recognizing both ASL and ISL signs, addressing the global variations in sign languages. The elevated accuracy levels make it a practical and accessible tool for the hearing-impaired community. This has significant implications for real-world applications, particularly in education, healthcare, and various contexts where improved communication between hearing-impaired individuals and others is paramount. The study represents a noteworthy stride in sign language recognition, presenting a hybrid model that excels in accurately identifying ASL and ISL signs, thereby contributing to the advancement of communication and inclusivity.

A Deep Learning Approach for ASL Recognition and Text-to-Speech Synthesis using CNN

Abstract: Sign language is a visual language that is used by the deaf and hard-of-hearing community to communicate. However, sign language is not universally understood by non-signers, which can create communication barriers for the deaf and hard-ofhearing individuals. In this paper, we present a novel application for American Sign Language (ASL) to text to speech conversion using deep learning techniques. Our app aims to bridge the communication gap between hearing-impaired individuals who use ASL as their primary mode of communication and individuals who do not understand ASL. The app comprises three main components: a hand gesture recognition module that detects and recognizes ASL signs, a text-tospeech synthesis module that converts recognized ASL signs to text, and a speech synthesis module that converts the text to speech

American Sign Language Recognition and Generation : A CNN-based Approach

Although not a global language, sign language is an essential tool for the deaf community. Communication between these communities and hearing population is severely hampered by this, as human-based interpretation can be both costly and time-consuming. In this paper, we present a real-time American Sign Language (ASL) generation and recognition system that makes use of Convolutional Neural Networks and deep learning (CNNs). Despite differences in lighting, skin tones, and backdrops, our technology is capable of correctly identifying and generating ASL signs. We trained our model on a large dataset of ASL signs in order to obtain a high level of accuracy. Our findings show that, with accuracy rates of 98.53% and 98.84%, respectively, our system achieves high accuracy rates in both training and validation. Our approach uses the advantages of CNNs to accomplish quick and precise recognition of individual letters and words, making it particularly effective for sign fingerspelling recognition. We believe that our technology has the ability to transform communication between the hearing community and the deaf and hard-of-hearing communities by providing a dependable and cost-effective way of sign language interpretation. Our method could help people who use sign language communicate more easily and live better in a range of environments, including schools, hospitals, and public places.

SmartASL

Sign language builds up an important bridge between the d/Deaf and hard-of-hearing (DHH) and hearing people. Regrettably, most hearing people face challenges in comprehending sign language, necessitating sign language translation. However, state-of-the-art wearable-based techniques mainly concentrate on recognizing manual markers (e.g., hand gestures), while frequently overlooking non-manual markers, such as negative head shaking, question markers, and mouthing. This oversight results in the loss of substantial grammatical and semantic information in sign language. To address this limitation, we introduce SmartASL, a novel proof-of-concept system that can 1) recognize both manual and non-manual markers simultaneously using a combination of earbuds and a wrist-worn IMU, and 2) translate the recognized American Sign Language (ASL) glosses into spoken language. Our experiments demonstrate the SmartASL system's significant potential to accurately recognize the manual and non-manual markers in ASL, effectively bridging the communication gaps between ASL signers and hearing people using commercially available devices.

Impoverished language in early childhood affects the development of complex sentence structure.

The hypothesis that impoverished language experience affects complex sentence structure development around the end of early childhood was tested using a fully randomized, sentence-to-picture matching study in American Sign Language (ASL). The participants were ASL signers who had impoverished or typical access to language in early childhood. Deaf signers whose access to language was highly impoverished in early childhood (N=11) primarily comprehended structures consisting of a single verb and argument (Subject or Object), agreeing verbs, and the spatial relation or path of semantic classifiers. They showed difficulty comprehending more complex sentence structures involving dual lexical arguments or multiple verbs. As predicted, participants with typical language access in early childhood, deaf native signers (N=17) or hearing second-language learners (N=10), comprehended the range of 12 ASL sentence structures, independent of the subjective iconicity or frequency of the stimulus lexical items, or length of ASL experience and performance on non-verbal cognitive tasks. The results show that language experience in early childhood is necessary for the development of complex syntax. RESEARCH HIGHLIGHTS: Previous research with deaf signers suggests an inflection point around the end of early childhood for sentence structure development. Deaf signers who experienced impoverished language until the age of 9 or older comprehend several basic sentence structures but few complex structures. Language experience in early childhood is necessary for the development of complex sentence structure.

Overrepresentation of Whiteness Is in Sign Language As Well: A Commentary on “Undoing Competence: Coloniality, Homogeneity, and the Overrepresentation of Whiteness in Applied Linguistics”

Overrepresentation of Whiteness Is in Sign Language As Well: A Commentary on “Undoing Competence: Coloniality, Homogeneity, and the Overrepresentation of Whiteness in Applied Linguistics”

Visual attention for linguistic and non-linguistic body actions in non-signing and native signing children.

Evidence from adult studies of deaf signers supports the dissociation between neural systems involved in processing visual linguistic and non-linguistic body actions. The question of how and when this specialization arises is poorly understood. Visual attention to these forms is likely to change with age and be affected by prior language experience. The present study used eye-tracking methodology with infants and children as they freely viewed alternating video sequences of lexical American sign language (ASL) signs and non-linguistic body actions (self-directed grooming action and object-directed pantomime). In Experiment 1, we quantified fixation patterns using an area of interest (AOI) approach and calculated face preference index (FPI) values to assess the developmental differences between 6 and 11-month-old hearing infants. Both groups were from monolingual English-speaking homes with no prior exposure to sign language. Six-month-olds attended the signer’s face for grooming; but for mimes and signs, they were drawn to attend to the “articulatory space” where the hands and arms primarily fall. Eleven-month-olds, on the other hand, showed a similar attention to the face for all body action types. We interpret this to reflect an early visual language sensitivity that diminishes with age, just before the child’s first birthday. In Experiment 2, we contrasted 18 hearing monolingual English-speaking children (mean age of 4.8 years) vs. 13 hearing children of deaf adults (CODAs; mean age of 5.7 years) whose primary language at home was ASL. Native signing children had a significantly greater face attentional bias than non-signing children for ASL signs, but not for grooming and mimes. The differences in the visual attention patterns that are contingent on age (in infants) and language experience (in children) may be related to both linguistic specialization over time and the emerging awareness of communicative gestural acts.

The Effect of Iconicity on Weak Hand Drop in American Sign Language

The research community generally accepts that signed and spoken languages contain both iconicity and arbitrariness. Iconicity's impact on statistical distributions of motivated forms throughout signed language lexicons is clear (e.g. Occhino, 2016). However, there has been little work to determine whether the iconic links between form and meaning are relevant only to a sign's initial formation, or if these links are stored as part of lexical representations. In the present study, 40 Deaf signers of American Sign Language were asked to rate two-handed signs in their citation form and in one-handed (reduced) forms. Twelve signs were highly iconic. For each of these highly iconic sign, a less iconic but phonologically similar sign of the same grammatical category was also chosen. Signs were presented in carrier sentences and in isolation. Participants preferred one-handed forms of the highly iconic signs over one-handed forms of their phonolgogically similar but less iconic counterparts. Thus, iconicity impacted the application of a synchronic phonological process. This finding suggests that lexical representations retain iconic form-meaning links and that these links are accessible to the phonological grammar.

Embodied cognition

AbstractRecent work has shown that ASL (American Sign Language) signers not only articulate the language in the space in front of and around them, they interact with that space bodily, such that those interactions are frequently viewpointed. At a basic level, signers use their bodies to depict the actions of characters, either themselves or others, in narrative retelling. These viewpointed instances seem to reflect “embodied cognition”, in that our construal of reality is largely due to the nature of our bodies (Evans and Green, 2006) and “embodied language” such that the symbols we use to communicate are “grounded in recurring patterns of bodily experience” (Gibbs, 2017: 450). But what about speakers of a spoken language such as English? While we know that meaning and structure for any language, whether spoken or signed, affect and are affected by the embodied mind (note that the bulk of research on embodied language has been about spoken, not signed, language), we can learn much about embodied cognition and viewpointed space when spoken languages are treated as multimodal. Here, we compare signed ASL and spoken, multimodal English discourse to examine whether the two languages incorporate viewpointed space in similar or different ways.