American Sign Language Animations Research Articles

A Translation System That Converts English Text to American Sign Language Enhanced with Deep Learning Modules

A recent surge in interest to create translation systems inclusive of sign languages is engendered by not only the rapid development of various approaches in the field of machine translation, but also the increased awareness of the struggles of the deaf community to comprehend written English. This paper describes the working of SILANT (SIgn LANguage Translator), a machine translation system that converts English to American Sign Language (ASL) using the principles of Natural Language Processing (NLP) and Deep Learning. The translation of English text is based on transformational rules which generates an intermediate representation which in turn spawns appropriate ASL animations. Although this kind of rule-based translation is notorious for being an accurate yet narrow approach, in this system, we broaden the scope of the translation using a synonym network and paraphrasing module which implements deep learning algorithms. In doing so, we are able to achieve both the accuracy of a rule-based approach and the scale of a deep learning one.

Regression Analysis of Demographic and Technology-Experience Factors Influencing Acceptance of Sign Language Animation

Software for automating the creation of linguistically accurate and natural-looking animations of American Sign Language (ASL) could increase information accessibility for many people who are deaf. As compared to recording and updating videos of human ASL signers, technology for automatically producing animation from an easy-to-update script would make maintaining ASL content on websites more efficient. Most sign language animation researchers evaluate their systems by collecting subjective judgments and comprehension-question responses from deaf participants. Through a survey (N = 62) and multiple-regression analysis, we identified relationships between (a) demographic and technology-experience characteristics of participants and (b) the subjective and objective scores collected from them during the evaluation of sign language animation systems. These relationships were experimentally verified in a subsequent user study with 57 participants, which demonstrated that specific subpopulations have higher comprehension or subjective scores when viewing sign language animations in an evaluation study. This finding indicates that researchers should collect and report a set of specific characteristics about participants in any publications describing evaluation studies of their technology, a practice that is not yet currently standard among researchers working in this field. In addition to investigating this relationship between participant characteristics and study results, we have also released our survey questions in ASL and English that can be used to measure these participant characteristics, to encourage reporting of such data in future studies. Such reporting would enable researchers in the field to better interpret and compare results between studies with different participant pools.

An automated technique for real-time production of lifelike animations of American Sign Language

Generating sentences from a library of signs implemented through a sparse set of key frames derived from the segmental structure of a phonetic model of ASL has the advantage of flexibility and efficiency, but lacks the lifelike detail of motion capture. These difficulties are compounded when faced with real-time generation and display. This paper describes a technique for automatically adding realism without the expense of manually animating the requisite detail. The new technique layers transparently over and modifies the primary motions dictated by the segmental model and does so with very little computational cost, enabling real-time production and display. The paper also discusses avatar optimizations that can lower the rendering overhead in real-time displays.

Collecting and evaluating the CUNY ASL corpus for research on American Sign Language animation

While there is great potential for sign language animation generation software to improve the accessibility of information for deaf individuals with low written-language literacy, the understandability of current sign language animation systems is limited. Data-driven methodologies using annotated sign language corpora encoding detailed human movement have enabled some researchers to address several key linguistic challenges in ASL generation. This article motivates and describes our current research on collecting a motion-capture corpus of American Sign Language (ASL). As an evaluation of our motion-capture configuration, calibration, and recording protocol, we have conducted several rounds of evaluation studies with native ASL signers, and we have made use of our collected data to synthesize novel animations of ASL, which have also been evaluated in experimental studies with native signers.

Effect of Displaying Human Videos During an Evaluation Study of American Sign Language Animation

Many researchers internationally are studying how to synthesize computer animations of sign language; such animations have accessibility benefits for people who are deaf and have lower literacy in written languages. The field has not yet formed a consensus as to how to best conduct evaluations of the quality of sign language animations, and this article explores an important methodological issue for researchers conducting experimental studies with participants who are deaf. Traditionally, when evaluating an animation, some lower and upper baselines are shown for comparison during the study. For the upper baseline, some researchers use carefully produced animations, and others use videos of human signers. Specifically, this article investigates, in studies where signers view animations of sign language and are asked subjective and comprehension questions, whether participants differ in their subjective and comprehension responses when actual videos of human signers are shown during the study. Through three sets of experiments, we characterize how the Likert-scale subjective judgments of participants about sign language animations are negatively affected when they are also shown videos of human signers for comparison -- especially when displayed side-by-side. We also identify a small positive effect on the comprehension of sign language animations when studies also contain videos of human signers. Our results enable direct comparison of previously published evaluations of sign language animations that used different types of upper baselines -- video or animation. Our results also provide methodological guidance for researchers who are designing evaluation studies of sign language animation or designing experimental stimuli or questions for participants who are deaf.

Models of linguistic facial expressions for American Sign Language animation

Facial expressions are a necessary and important part of the grammar of American Sign Language (ASL) sentences. However, current computational linguistic software cannot produce a huge portion of ASL constructions that include facial expressions. Here we summarize our research motivations, proposed plan, and potential contributions in the accessibility field. Our research aim is to formulate signer-independent models of facial movements for various types of linguistically meaningful facial expressions in ASL animation. This project summary also includes a brief description of current progress and future research plans.

Data-Driven Synthesis of Spatially Inflected Verbs for American Sign Language Animation

We are studying techniques for producing realistic and understandable animations of American Sign Language (ASL); such animations have accessibility benefits for signers with lower levels of written language literacy. This article describes and evaluates a novel method for modeling and synthesizing ASL animations based on samples of ASL signs collected from native signers. We apply this technique to ASL inflecting verbs, common signs in which the location and orientation of the hands is influenced by the arrangement of locations in 3D space that represent entities under discussion. We train mathematical models of hand movement on animation data of signs produced by a native signer. In evaluation studies with native ASL signers, the verb animations synthesized from our model had similar subjective-rating and comprehension-question scores to animations produced by a human animator; they also achieved higher scores than baseline animations. Further, we examine a split modeling technique for accommodating certain verb signs with complex movement patterns, and we conduct an analysis of how robust our modeling techniques are to reductions in the size of their training data. The modeling techniques in this article are applicable to other types of ASL signs and to other sign languages used internationally. Our models’ parameterization of sign animations can increase the repertoire of generation systems and can partially automate the work of humans using sign language scripting systems.

Effect of spatial reference and verb inflection on the usability of sign language animations

Computer-generated animations of American Sign Language (ASL) can improve the accessibility of information, communication, and services for the significant number of deaf adults in the US with difficulty in reading English text. Unfortunately, there are several linguistic aspects of ASL that current automatic generation or translation systems cannot produce (or are time-consuming for human animators to create). To determine how important such phenomena are to user satisfaction and the comprehension of ASL animations, studies were conducted in which native ASL signers evaluated ASL animations with and without: establishment of spatial reference points around the virtual human signer representing entities under discussion, pointing pronoun signs, contrastive role shift, and spatial inflection of ASL verbs. It was found that adding these phenomena to ASL animations led to a significant improvement in user comprehension of the animations, thereby motivating future research on automating the generation of these animations.

The automation of spinal non-manual signals in American Sign Language

American Sign Language (ASL) is the first language of the Deaf in the United States. The most effective method for communication between the Deaf and Hearing is certified ASL interpreters, however interpreters are not always available. In this case, the use of a translator producing 3D animations of ASL could improve communication, however the characterization of ASL for automated 3D animation is still an open question. The goal of this research is to determine how and when non-manual signals are used in ASL sentences, and develop a methodology to portray them as 3D animations.

Participation of high school and undergraduate students who are deaf in research on American Sign Language animation

The Linguistic and Assistive Technologies Laboratory (the "LATLab") at Queens College of The City University of New York conducts research on computational linguistics and human computer interaction - with a focus on assistive technology and accessibility applications for people with disabilities. We are currently conducting a five-year study supported by a National Science Foundation CAREER Award (NSF Award #0746556). The goals of this research are to significantly improve the state of the art of animations of American Sign Language (ASL), which have important accessibility applications for people who are deaf. (Details of the scientific aims of this project are included below.) This article describes the educational outreach component of this project --- the participation of deaf high school students and undergraduate students in summer research experiences. Our goal in writing this article is to describe how the educational component of our project integrates with the scientific goals and to explain some of the details behind the summer research experiences we have organized. We hope that other SIGACCESS members who are interested in incorporating educational outreach efforts into their research may find this information useful when designing and proposing their own projects. There are several deaf students and researchers involved in our project. Each summer, two or three students from local deaf high schools or mainstream programs in the greater New York City area participate in three-month research experiences at the LATLab. In addition, we have also hosted a deaf undergraduate student (visiting from Gallaudet University) for a summer research experience, and a graduate-level deaf research assistant coordinates projects at the lab on an ongoing basis part-time throughout the academic year. We believe that the involvement of deaf ASL-signers in the research will help to train future researchers who have a community and cultural connection to the outcome of the work. Further, only users of ASL who have native level fluency are able to accomplish some of the aspects of the research: analyzing recordings of sign language we collect and conducting experimental sessions in which native ASL signers from the local community come to the lab to evaluate the ASL animations we produce. Our objective is to attract deaf students to careers in computer science by enabling deaf high school and undergraduate students to experience the research process. Students with disabilities pursue science careers at a lower than average rate (CEOSE, 2004), and our goal is to encourage the interest of these students in pursuing higher education and careers in the sciences, at a time when they may be making decisions about their future education. While these students have minimal programming or linguistics background, their ASL fluency can allow them to make meaningful contributions to the project. We want them to become aware of applications of computer science with direct relevance to the deaf community --- generating animations of American Sign Language to make information sources more accessible to deaf individuals who may have low literacy in written English.

Modeling animations of American Sign Language verbs through motion-capture of native ASL signers

Software to generate American Sign Language (ASL) automatically can provide benefits for deaf people with low English literacy. However, modern computational linguistic software cannot produce important aspects of ASL signs and verbs. Better models of spatially complex signs are needed. Our goals are: to create a linguistic resource of ASL signs via motion-capture data collection; to model the movement paths of inflecting/indicating verbs using machine learning and computational techniques; and to produce grammatical, natural looking and understandable animations of ASL. Our methods include linguistic annotation of the data and evaluation by native ASL signers. This summary also describes our research progress.

A Linguistically Motivated Model for Speed and Pausing in Animations of American Sign Language

Many deaf adults in the United States have difficulty reading written English text; computer animations of American Sign Language (ASL) can improve these individuals’ access to information, communication, and services. Planning and scripting the movements of a virtual character’s arms and body to perform a grammatically correct and understandable ASL sentence is a difficult task, and the timing subtleties of the animation can be particularly challenging. After examining the psycholinguistics literature on the speed and timing of ASL, we have designed software to calculate realistic timing of the movements in ASL animations. We have built algorithms to calculate the time-duration of signs and the location/length of pauses during an ASL animation. To determine whether our software can improve the quality of ASL animations, we conducted a study in which native ASL signers evaluated the ASL animations processed by our algorithms. We have found that: (1) adding linguistically motivated pauses and variations in sign-durations improved signers’ performance on a comprehension task and (2) these animations were rated as more understandable by ASL signers.

Evaluation of American Sign Language Generation by Native ASL Signers

There are many important factors in the design of evaluation studies for systems that generate animations of American Sign Language (ASL) sentences, and techniques for evaluating natural language generation of written texts are not easily adapted to ASL. When conducting user-based evaluations, several cultural and linguistic characteristics of members of the American Deaf community must be taken into account so as to ensure the accuracy of evaluations involving these users. This article describes an implementation and user-based evaluation (by native ASL signers) of a prototype ASL natural language generation system that produces sentences containing classifier predicates, which are frequent and complex spatial phenomena that previous ASL generators have not produced. Native signers preferred the system's output to Signed English animations -- scoring it higher in grammaticality, understandability, and naturalness of movement. They were also more successful at a comprehension task after viewing the system's classifier predicate animations.

SPATIAL, TEMPORAL, AND SEMANTIC MODELS FOR AMERICAN SIGN LANGUAGE GENERATION: IMPLICATIONS FOR GESTURE GENERATION

Software to generate animations of American Sign Language (ASL) has important accessibility benefits for the significant number of deaf adults with low levels of written language literacy. We have implemented a prototype software system to generate an important subset of ASL phenomena called "classifier predicates," complex and spatially descriptive types of sentences. The output of this prototype system has been evaluated by native ASL signers. Our generator includes several novel models of 3D space, spatial semantics, and temporal coordination motivated by linguistic properties of ASL. These classifier predicates have several similarities to iconic gestures that often co-occur with spoken language; these two phenomena will be compared. This article explores implications of the design of our system for research in multimodal gesture generation systems. A conceptual model of multimodal communication signals is introduced to show how computational linguistic research on ASL relates to the field of multimodal natural language processing.

Generating American Sign Language animation: overcoming misconceptions and technical challenges

Misconceptions about the English literacy rates of deaf Americans, the linguistic structure of American Sign Language (ASL), and the suitability of traditional machine translation (MT) technology to ASL have slowed the development of English-to-ASL MT systems for use in accessibility applications. This article traces the progress of a new English-to-ASL MT project targeted to translating texts important for literacy and user-interface applications. These texts include ASL phenomena called “classifier predicates.” Challenges in producing classifier predicates, novel solutions to these challenges, and applications of this technology to the design of user-interfaces accessible to deaf users will be discussed.

Representing coordination and non-coordination in American Sign Language animations

While strings and syntax trees are used by the Natural Language Processing community to represent the structure of spoken languages, these encodings are difficult to adapt to a signed language like American Sign Language (ASL). In particular, the multichannel nature of an ASL performance makes it difficult to encode in a linear single-channel string. This paper will introduce the Partition/Constitute (P/C) formalism, a new method of computationally representing a linguistic signal containing multiple channels. The formalism allows coordination and non-coordination relationships to be encoded between different portions of a signal. The P/C formalism will be compared to representations used in related research in gesture animation. The way in which P/C is used by this project to build an English-to-ASL machine translation system will also be discussed.