Native American Sign Language Signers Research Articles

The unexplored role of handshape similarity in processing numbers on the hands

With two simple experiments we investigate the overlooked influence of handshape similarity for processing numerical information conveyed on the hands. In most finger-counting sequences there is a tight relationship between the number of fingers raised and the numerical value represented. This creates a possible confound where numbers closer to each other are also represented by handshapes that are more similar. By using the American Sign Language (ASL) number signs we are able to dissociate between the two variables orthogonally. First, we test the effect of handshape similarity in a same/different judgment task in a group of hearing non-signers and then test the interference of handshape in a number judgment task in a group of native ASL signers. Our results show an effect of handshape similarity and its interaction with numerical value even in the group of native signers for whom these handshapes are linguistic symbols and not a learning tool for acquiring numerical concepts. Because prior studies have never considered handshape similarity, these results open new directions for understanding the relationship between finger-based counting, internal hand representations and numerical proficiency.

Implicit causality biases and thematic roles in American Sign Language

Implicit causality (IC) biases, the tendency of certain verbs to elicit re-mention of either the first-mentioned noun phrase (NP1) or the second-mentioned noun phrase (NP2) from the previous clause, are important in psycholinguistic research. Understanding IC verbs and the source of their biases in signed as well as spoken languages helps elucidate whether these phenomena are language general or specific to the spoken modality. As the first of its kind, this study investigates IC biases in American Sign Language (ASL) and provides IC bias norms for over 200 verbs, facilitating future psycholinguistic studies of ASL and comparisons of spoken versus signed languages. We investigated whether native ASL signers continued sentences with IC verbs (e.g., ASL equivalents of ‘Lisa annoys Maya because…’) by mentioning NP1 (i.e., Lisa) or NP2 (i.e., Maya). We found a tendency towards more NP2-biased verbs. Previous work has found that a verb’s thematic roles predict bias direction: stimulus-experiencer verbs (e.g., ‘annoy’), where the first argument is the stimulus (causing annoyance) and the second argument is the experiencer (experiencing annoyance), elicit more NP1 continuations. Verbs with experiencer-stimulus thematic roles (e.g., ‘love’) elicit more NP2 continuations. We probed whether the trend towards more NP2-biased verbs was related to an existing claim that stimulus-experiencer verbs do not exist in sign languages. We found that stimulus-experiencer structure, while permitted, is infrequent, impacting the IC bias distribution in ASL. Nevertheless, thematic roles predict IC bias in ASL, suggesting that the thematic role-IC bias relationship is stable across languages as well as modalities.

Design and Psychometric Evaluation of American Sign Language Translations of Usability Questionnaires

To promote greater inclusion of people who are Deaf and Hard of Hearing (DHH) in studies conducted by Human-Computer Interaction (HCI) researchers or professionals, we have undertaken a project to formally translate several standardized usability questionnaires from English to ASL. Many deaf adults in the U.S. have lower levels of English reading literacy, but there are currently no standardized usability questionnaires available in American Sign Language (ASL) for these users. A critical concern in conducting such a translation is to ensure that the meaning of the original question items has been preserved during translation, as well as other key psychometric properties of the instrument, including internal reliability, criterion validity, and construct validity. After identifying best-practices for such a translation and evaluation project, a bilingual team of domain experts (including native ASL signers who are members of the Deaf community) translated the System Usability Scale (SUS) and Net Promoter Score (NPS) instruments into ASL and then conducted back-translation evaluations to assess the faithfulness of the translation. The new ASL instruments were employed in usability tests with DHH participants, to assemble a dataset of response scores, in support of the psychometric validation. We are disseminating these translated instruments, as well as collected response values from DHH participants, to encourage greater participation in HCI studies among DHH users.

Detecting clauses and their dependencies in signed utterances: A syntactico-semantic approach

Investigating the syntactic structure of utterances with multiple predicates in sign languages requires a clear understanding of how many finite and infinitival clauses they contain and which syntactic dependencies exist between them. Since the sign language literature currently lacks a standardized methodology for identifying clause boundaries, this paper discusses syntactico-semantic diagnostics of clausehood and clause size and analyzes their applicability to American Sign Language (ASL) and German Sign Language (DGS). First, I discuss tests that distinguish coordinated clauses from dependent clause structures; specifically negation, A’-movement, and subject pronoun copy. Limitations of wh- and topic fronting as clausehood diagnostics are identified and a modified subject pronoun copy test is proposed. Determining whether a given utterance contains coordinated or dependent clauses is only half the battle, however; we also want to know the approximate “size” of the constituent an embedded predicate projects. The present study takes a first pass at filling this gap by introducing rightward wh-movement and confirming center-embedding as diagnostics that can discriminate between finite and infinitival clauses in signed languages. Based on acceptability judgments from 13 native signers of DGS and ASL, I show that wh-subjects can move across infinitival control complements and the secondary predicates of resultative constructions, but they cannot cross a finite complement clause. The diagnostic thus provides empirical evidence for the existence of various types of embedded clauses in signed languages that differ in their functional structure.

Adaptation of a vocabulary test from British Sign Language to American Sign Language

This study describes the adaptation process of a vocabulary knowledge test for British Sign Language (BSL) into American Sign Language (ASL) and presents results from the first round of pilot testing with 20 deaf native ASL signers. The web-based test assesses the strength of deaf children’s vocabulary knowledge by means of different mappings of phonological form and meaning of signs. The adaptation from BSL to ASL involved nine stages, which included forming a panel of deaf/hearing experts, developing a set of new items and revising/replacing items considered ineffective, and piloting the new version. Results provide new evidence in support of the use of this methodology for assessing signed languages, making a useful contribution toward the availability of tests to assess deaf children’s signed language skills.

Scalar implicatures in a signed language

This paper tests the calculation of scalar implicatures in American Sign Language (ASL) in one of the first experimental pragmatic studies in the manual/visual modality. Both native signers of ASL and native speakers of English participated in an automated Felicity Judgment Task to compare implicatures based on two traditional scales as well as “ad hoc” scales in their respective languages. Results show that native signers of ASL calculate scalar implicatures based on a prototypical scale <all, some> in ASL in the same pattern as native speakers of English, within the same experimental paradigm. There are similarly high rates of exact interpretations of numbers <three, two> in ASL as in English, despite the iconicity of the numerals in ASL. Finally, an ad hoc scale was tested showing fewer implicatures in English than on the conventionalized scales. In ASL, there was a trend toward increased implicatures on the ad hoc scale which made use of the unique ability of ASL to convey spatial information using the classifier system. Taken together, these results show that conventionalized scales in ASL have the same semantic/pragmatic scalar properties as in spoken languages, although in non-conventionalized scales the inclusion of additional information such as spatial location may affect pragmatic interpretation.

Collecting and evaluating the CUNY ASL corpus for research on American Sign Language animation

While there is great potential for sign language animation generation software to improve the accessibility of information for deaf individuals with low written-language literacy, the understandability of current sign language animation systems is limited. Data-driven methodologies using annotated sign language corpora encoding detailed human movement have enabled some researchers to address several key linguistic challenges in ASL generation. This article motivates and describes our current research on collecting a motion-capture corpus of American Sign Language (ASL). As an evaluation of our motion-capture configuration, calibration, and recording protocol, we have conducted several rounds of evaluation studies with native ASL signers, and we have made use of our collected data to synthesize novel animations of ASL, which have also been evaluated in experimental studies with native signers.

Acquiring Word Class Distinctions in American Sign Language: Evidence from Handshape

Handshape works differently in nouns versus a class of verbs in American Sign Language (ASL) and thus can serve as a cue to distinguish between these two word classes. Handshapes representing characteristics of the object itself (object handshapes) and handshapes representing how the object is handled (handling handshapes) appear in both nouns and a particular type of verb, classifier predicates, in ASL. When used as nouns, object and handling handshapes are phonemic––that is, they are specified in dictionary entries and do not vary with grammatical context. In contrast, when used as classifier predicates, object and handling handshapes do vary with grammatical context for both morphological and syntactic reasons. We ask here when young deaf children learning ASL acquire the word class distinction signaled by handshape. Specifically, we determined the age at which children systematically vary object versus handling handshapes as a function of grammatical context in classifier predicates but not in the nouns that accompany those predicates. We asked 4–6-year-old children, 7–10-year-old children, and adults, all of whom were native ASL signers, to describe a series of vignettes designed to elicit object and handling handshapes in both nouns and classifier predicates. We found that all of the children behaved like adults with respect to all nouns, systematically varying object and handling handshapes as a function of type of item and not grammatical context. The children also behaved like adults with respect to certain classifiers, systematically varying handshape type as a function of grammatical context for items whose nouns have handling handshapes. The children differed from adults in that they did not systematically vary handshape as a function of grammatical context for items whose nouns have object handshapes. These findings extend previous work by showing that children require developmental time to acquire the full morphological system underlying classifier predicates in sign language, just as children acquiring complex morphology in spoken languages do. In addition, we show for the first time that children acquiring ASL treat object and handling handshapes differently as a function of their status as nouns vs. classifier predicates, and thus display a distinction between these word classes as early as 4 years of age.

Data-Driven Synthesis of Spatially Inflected Verbs for American Sign Language Animation

We are studying techniques for producing realistic and understandable animations of American Sign Language (ASL); such animations have accessibility benefits for signers with lower levels of written language literacy. This article describes and evaluates a novel method for modeling and synthesizing ASL animations based on samples of ASL signs collected from native signers. We apply this technique to ASL inflecting verbs, common signs in which the location and orientation of the hands is influenced by the arrangement of locations in 3D space that represent entities under discussion. We train mathematical models of hand movement on animation data of signs produced by a native signer. In evaluation studies with native ASL signers, the verb animations synthesized from our model had similar subjective-rating and comprehension-question scores to animations produced by a human animator; they also achieved higher scores than baseline animations. Further, we examine a split modeling technique for accommodating certain verb signs with complex movement patterns, and we conduct an analysis of how robust our modeling techniques are to reductions in the size of their training data. The modeling techniques in this article are applicable to other types of ASL signs and to other sign languages used internationally. Our models’ parameterization of sign animations can increase the repertoire of generation systems and can partially automate the work of humans using sign language scripting systems.

Effect of spatial reference and verb inflection on the usability of sign language animations

Computer-generated animations of American Sign Language (ASL) can improve the accessibility of information, communication, and services for the significant number of deaf adults in the US with difficulty in reading English text. Unfortunately, there are several linguistic aspects of ASL that current automatic generation or translation systems cannot produce (or are time-consuming for human animators to create). To determine how important such phenomena are to user satisfaction and the comprehension of ASL animations, studies were conducted in which native ASL signers evaluated ASL animations with and without: establishment of spatial reference points around the virtual human signer representing entities under discussion, pointing pronoun signs, contrastive role shift, and spatial inflection of ASL verbs. It was found that adding these phenomena to ASL animations led to a significant improvement in user comprehension of the animations, thereby motivating future research on automating the generation of these animations.

Accurate and Accessible Motion-Capture Glove Calibration for Sign Language Data Collection

Motion-capture recordings of sign language are used in research on automatic recognition of sign language or generation of sign language animations, which have accessibility applications for deaf users with low levels of written-language literacy. Motion-capture gloves are used to record the wearer’s handshape. Unfortunately, they require a time-consuming and inexact calibration process each time they are worn. This article describes the design and evaluation of a new calibration protocol for motion-capture gloves, which is designed to make the process more efficient and to be accessible for participants who are deaf and use American Sign Language (ASL). The protocol was evaluated experimentally; deaf ASL signers wore the gloves, were calibrated (using the new protocol and using a calibration routine provided by the glove manufacturer), and were asked to perform sequences of ASL handshapes. Five native ASL signers rated the correctness and understandability of the collected handshape data. In an additional evaluation, ASL signers were asked to perform ASL stories while wearing the gloves and a motion-capture bodysuit (in some cases our new calibration protocol was used, in other cases, the standard protocol). Later, twelve native ASL signers watched animations produced from this motion-capture data and answered comprehension questions about the stories. In both evaluation studies, the new protocol received significantly higher scores than the standard calibration. The protocol has been made freely available online, and it includes directions for the researcher, images and videos of how participants move their hands during the process, and directions for participants (as ASL videos and English text).

Participation of high school and undergraduate students who are deaf in research on American Sign Language animation

The Linguistic and Assistive Technologies Laboratory (the "LATLab") at Queens College of The City University of New York conducts research on computational linguistics and human computer interaction - with a focus on assistive technology and accessibility applications for people with disabilities. We are currently conducting a five-year study supported by a National Science Foundation CAREER Award (NSF Award #0746556). The goals of this research are to significantly improve the state of the art of animations of American Sign Language (ASL), which have important accessibility applications for people who are deaf. (Details of the scientific aims of this project are included below.) This article describes the educational outreach component of this project --- the participation of deaf high school students and undergraduate students in summer research experiences. Our goal in writing this article is to describe how the educational component of our project integrates with the scientific goals and to explain some of the details behind the summer research experiences we have organized. We hope that other SIGACCESS members who are interested in incorporating educational outreach efforts into their research may find this information useful when designing and proposing their own projects. There are several deaf students and researchers involved in our project. Each summer, two or three students from local deaf high schools or mainstream programs in the greater New York City area participate in three-month research experiences at the LATLab. In addition, we have also hosted a deaf undergraduate student (visiting from Gallaudet University) for a summer research experience, and a graduate-level deaf research assistant coordinates projects at the lab on an ongoing basis part-time throughout the academic year. We believe that the involvement of deaf ASL-signers in the research will help to train future researchers who have a community and cultural connection to the outcome of the work. Further, only users of ASL who have native level fluency are able to accomplish some of the aspects of the research: analyzing recordings of sign language we collect and conducting experimental sessions in which native ASL signers from the local community come to the lab to evaluate the ASL animations we produce. Our objective is to attract deaf students to careers in computer science by enabling deaf high school and undergraduate students to experience the research process. Students with disabilities pursue science careers at a lower than average rate (CEOSE, 2004), and our goal is to encourage the interest of these students in pursuing higher education and careers in the sciences, at a time when they may be making decisions about their future education. While these students have minimal programming or linguistics background, their ASL fluency can allow them to make meaningful contributions to the project. We want them to become aware of applications of computer science with direct relevance to the deaf community --- generating animations of American Sign Language to make information sources more accessible to deaf individuals who may have low literacy in written English.

Modeling animations of American Sign Language verbs through motion-capture of native ASL signers

Software to generate American Sign Language (ASL) automatically can provide benefits for deaf people with low English literacy. However, modern computational linguistic software cannot produce important aspects of ASL signs and verbs. Better models of spatially complex signs are needed. Our goals are: to create a linguistic resource of ASL signs via motion-capture data collection; to model the movement paths of inflecting/indicating verbs using machine learning and computational techniques; and to produce grammatical, natural looking and understandable animations of ASL. Our methods include linguistic annotation of the data and evaluation by native ASL signers. This summary also describes our research progress.

A Linguistically Motivated Model for Speed and Pausing in Animations of American Sign Language

Many deaf adults in the United States have difficulty reading written English text; computer animations of American Sign Language (ASL) can improve these individuals’ access to information, communication, and services. Planning and scripting the movements of a virtual character’s arms and body to perform a grammatically correct and understandable ASL sentence is a difficult task, and the timing subtleties of the animation can be particularly challenging. After examining the psycholinguistics literature on the speed and timing of ASL, we have designed software to calculate realistic timing of the movements in ASL animations. We have built algorithms to calculate the time-duration of signs and the location/length of pauses during an ASL animation. To determine whether our software can improve the quality of ASL animations, we conducted a study in which native ASL signers evaluated the ASL animations processed by our algorithms. We have found that: (1) adding linguistically motivated pauses and variations in sign-durations improved signers’ performance on a comprehension task and (2) these animations were rated as more understandable by ASL signers.

Distinguishing the processing of gestures from signs in deaf individuals: An fMRI study

Manual gestures occur on a continuum from co-speech gesticulations to conventionalized emblems to language signs. Our goal in the present study was to understand the neural bases of the processing of gestures along such a continuum. We studied four types of gestures, varying along linguistic and semantic dimensions: linguistic and meaningful American Sign Language (ASL), non-meaningful pseudo-ASL, meaningful emblematic, and nonlinguistic, non-meaningful made-up gestures. Pre-lingually deaf, native signers of ASL participated in the fMRI study and performed two tasks while viewing videos of the gestures: a visuo-spatial (identity) discrimination task and a category discrimination task. We found that the categorization task activated left ventral middle and inferior frontal gyrus, among other regions, to a greater extent compared to the visual discrimination task, supporting the idea of semantic-level processing of the gestures. The reverse contrast resulted in enhanced activity of bilateral intraparietal sulcus, supporting the idea of featural-level processing (analogous to phonological-level processing of speech sounds) of the gestures. Regardless of the task, we found that brain activation patterns for the nonlinguistic, non-meaningful gestures were the most different compared to the ASL gestures. The activation patterns for the emblems were most similar to those of the ASL gestures and those of the pseudo-ASL were most similar to the nonlinguistic, non-meaningful gestures. The fMRI results provide partial support for the conceptualization of different gestures as belonging to a continuum and the variance in the fMRI results was best explained by differences in the processing of gestures along the semantic dimension.

The link between form and meaning in American Sign Language: Lexical processing effects.

Signed languages exploit iconicity (the transparent relationship between meaning and form) to a greater extent than spoken languages. where it is largely limited to onomatopoeia. In a picture-sign matching experiment measuring reaction times, the authors examined the potential advantage of iconicity both for 1st- and 2nd-language learners of American Sign Language (ASL). The results show that native ASL signers are faster to respond when a specific property iconically represented in a sign is made salient in the corresponding picture, thus providing evidence that a closer mapping between meaning and form can aid in lexical retrieval. While late 2nd-language learners appear to use iconicity as an aid to learning sign (R. Campbell, P. Martin, & T. White, 1992), they did not show the same facilitation effect as native ASL signers, suggesting that the task tapped into more automatic language processes. Overall, the findings suggest that completely arbitrary mappings between meaning and form may not be more advantageous in language and that, rather, arbitrariness may simply be an accident of modality.

Morphology of the Insula in Relation to Hearing Status and Sign Language Experience

We investigated whether auditory deprivation and/or sign language exposure during development alters the macroscopic neuroanatomy of the human insula. Volumetric analyses were based on MRI data from 25 congenitally deaf subjects who were native users of American Sign Language (ASL), 25 hearing subjects with no knowledge of ASL, and 16 hearing subjects who grew up in deaf families and were native ASL signers. Significant variation in insula volume was associated with both hearing status and sign language experience. Compared with both hearing groups, deaf subjects exhibited a significant increase in the amount of gray matter in the left posterior insular lobule, which we hypothesize may be related to the dependence on lip-reading and articulatory-based (rather than auditory-based) representations of speech for deaf individuals. Both deaf and hearing signers exhibited an increased volume of white matter in the right insula compared with hearing nonsigners. We hypothesize that the distinct morphology of the right insula for ASL signers may arise from enhanced connectivity resulting from an increased reliance on cross-modal sensory integration in sign language compared with spoken language.

Evaluation of American Sign Language Generation by Native ASL Signers

There are many important factors in the design of evaluation studies for systems that generate animations of American Sign Language (ASL) sentences, and techniques for evaluating natural language generation of written texts are not easily adapted to ASL. When conducting user-based evaluations, several cultural and linguistic characteristics of members of the American Deaf community must be taken into account so as to ensure the accuracy of evaluations involving these users. This article describes an implementation and user-based evaluation (by native ASL signers) of a prototype ASL natural language generation system that produces sentences containing classifier predicates, which are frequent and complex spatial phenomena that previous ASL generators have not produced. Native signers preferred the system's output to Signed English animations -- scoring it higher in grammaticality, understandability, and naturalness of movement. They were also more successful at a comprehension task after viewing the system's classifier predicate animations.

SPATIAL, TEMPORAL, AND SEMANTIC MODELS FOR AMERICAN SIGN LANGUAGE GENERATION: IMPLICATIONS FOR GESTURE GENERATION

Software to generate animations of American Sign Language (ASL) has important accessibility benefits for the significant number of deaf adults with low levels of written language literacy. We have implemented a prototype software system to generate an important subset of ASL phenomena called "classifier predicates," complex and spatially descriptive types of sentences. The output of this prototype system has been evaluated by native ASL signers. Our generator includes several novel models of 3D space, spatial semantics, and temporal coordination motivated by linguistic properties of ASL. These classifier predicates have several similarities to iconic gestures that often co-occur with spoken language; these two phenomena will be compared. This article explores implications of the design of our system for research in multimodal gesture generation systems. A conceptual model of multimodal communication signals is introduced to show how computational linguistic research on ASL relates to the field of multimodal natural language processing.

Five Nonmanual Modifiers That Mitigate Requests and Rejections in American Sign Language

A notable difference between signed and spoken languages is the use of nonmanual linguistic signals that co-occur with the production of signs. These nonmanual signals involve primarily the face and upper torso and are an important feature of American Sign Language (ASL). They include grammatical markers that indicate syntactic categories such as yes-no/questions and wh-word questions, as well as nonmanual markers (NMMs) that function as adverbs and adjectives. The article describes the ways in which native ASL signers use five nonmanual markers to alter requests and rejections in different discourse contexts. It also argues for a linear ordering of these NMMs based on the degree to which each mitigates requests and rejections and concludes with a discussion of implications for linguistic studies, ASL instruction, and ASL/English interpretation.