Hearing Non-signers Research Articles

Production and Comprehension of Prosodic Markers in Sign Language Imperatives.

In signed and spoken language sentences, imperative mood and the corresponding speech acts such as for instance, command, permission or advice, can be distinguished by morphosyntactic structures, but also solely by prosodic cues, which are the focus of this paper. These cues can express paralinguistic mental states or grammatical meaning, and we show that in American Sign Language (ASL), they also exhibit the function, scope, and alignment of prosodic, linguistic elements of sign languages. The production and comprehension of prosodic facial expressions and temporal patterns therefore can shed light on how cues are grammaticalized in sign languages. They can also be informative about the formal semantic and pragmatic properties of imperative types not only in ASL, but also more broadly. This paper includes three studies: one of production (Study 1) and two of comprehension (Studies 2 and 3). In Study 1, six prosodic cues are analyzed in production: temporal cues of sign and hold duration, and non-manual cues including tilts of the head, head nods, widening of the eyes, and presence of mouthings. Results of Study 1 show that neutral sentences and commands are well distinguished from each other and from other imperative speech acts via these prosodic cues alone; there is more limited differentiation among explanation, permission, and advice. The comprehension of these five speech acts is investigated in Deaf ASL signers in Study 2, and in three additional groups in Study 3: Deaf signers of German Sign Language (DGS), hearing non-signers from the United States, and hearing non-signers from Germany. Results of Studies 2 and 3 show that the ASL group performs significantly better than the other 3 groups and that all groups perform above chance for all meaning types in comprehension. Language-specific knowledge, therefore, has a significant effect on identifying imperatives based on targeted cues. Command has the most cues associated with it and is the most accurately identified imperative type across groups—indicating, we suggest, its special status as the strongest imperative in terms of addressing the speaker's goals. Our findings support the view that the cues are accessible in their content across groups, but that their language-particular combinatorial possibilities and distribution within sentences provide an advantage to ASL signers in comprehension and attest to their prosodic status.

Working Memory for Linguistic and Non-linguistic Manual Gestures: Evidence, Theory, and Application.

Linguistic manual gestures are the basis of sign languages used by deaf individuals. Working memory and language processing are intimately connected and thus when language is gesture-based, it is important to understand related working memory mechanisms. This article reviews work on working memory for linguistic and non-linguistic manual gestures and discusses theoretical and applied implications. Empirical evidence shows that there are effects of load and stimulus degradation on working memory for manual gestures. These effects are similar to those found for working memory for speech-based language. Further, there are effects of pre-existing linguistic representation that are partially similar across language modalities. But above all, deaf signers score higher than hearing non-signers on an n-back task with sign-based stimuli, irrespective of their semantic and phonological content, but not with non-linguistic manual actions. This pattern may be partially explained by recent findings relating to cross-modal plasticity in deaf individuals. It suggests that in linguistic gesture-based working memory, semantic aspects may outweigh phonological aspects when processing takes place under challenging conditions. The close association between working memory and language development should be taken into account in understanding and alleviating the challenges faced by deaf children growing up with cochlear implants as well as other clinical populations.

Predicting sign learning in hearing adults: The role of perceptual-motor (and phonological?) processes

ABSTRACTThe present study aimed to identify predictors of one aspect of sign language acquisition, sign learning, in hearing nonsigners. Candidate predictors were selected based on the theory that the observed relationship between phonological short-term memory and L2 lexical learning is due in part to common perceptual-motor processes. Hearing nonsigning adults completed a sign learning task, three assessments of short-term memory for movements (movement STM; two of which used sign-like stimuli), and two visuospatial STM tasks. The final sample included 103 adults, ranging between 18 and 33 years of age. All predictors were moderately to strongly correlated with the sign learning task and to each other. A series of regression analyses revealed that both movement and visuospatial STM uniquely contributed to the prediction of sign learning. These results suggest that perceptual-motor processes play a significant role in sign learning and raise questions about the role of phonological processing.

Sign-speaking: The structure of simultaneous bimodal utterances

Abstract We present data from a bimodal trilingual situation involving Indian Sign Language (ISL), Hindi and English. Signers are co-using these languages while in group conversations with deaf people and hearing non-signers. The data show that in this context, English is an embedded language that does not impact on the grammar of the utterances, while both ISL and Hindi structures are realised throughout. The data show mismatches between the simultaneously expressed ISL and Hindi, such that semantic content and/or syntactic structures are different in both languages, yet are produced at the same time. The data also include instances of different propositions expressed simultaneously in the two languages. This under-documented behaviour is called “sign-speaking” here, and we explore its implications for theories of multilingualism, code-switching, and bilingual language production.

Face Recognition is Shaped by the Use of Sign Language

Previous research has suggested that early deaf signers differ in face processing. Which aspects of face processing are changed and the role that sign language may have played in that change are however unclear. Here, we compared face categorization (human/non-human) and human face recognition performance in early profoundly deaf signers, hearing signers, and hearing non-signers. In the face categorization task, the three groups performed similarly in term of both response time and accuracy. However, in the face recognition task, signers (both deaf and hearing) were slower than hearing non-signers to accurately recognize faces, but had a higher accuracy rate. We conclude that sign language experience, but not deafness, drives a speed-accuracy trade-off in face recognition (but not face categorization). This suggests strategic differences in the processing of facial identity for individuals who use a sign language, regardless of their hearing status.

Gesture-based customer interactions: deaf and hearing Mumbaikars’ multimodal and metrolingual practices

ABSTRACTThe article furthers the study of urban multilingual (i.e. metrolingual) practices, in particular the study of customer interactions, by a focus on the use of gestures in these practices. The article focuses on fluent deaf signers and hearing non-signers in Mumbai who use gestures to communicate with each other, often combined with mouthing, speaking and/or writing in different languages. The data were gathered through linguistic ethnography in markets, shops, food joints and public transport in Mumbai. Within gesture-based interactions, people with sensorial asymmetries (i.e. deaf vs. hearing) combined the visual-gestural modality and certain features of the auditory-oral modality, and/or switched between modalities. Interlocutors thus orient towards the ongoing interaction and negotiate the constraints and possibilities imposed not only by different modalities but also by different sensorial access to these modalities.

Why space is not one-dimensional: Location may be categorical and imagistic.

In our commentary, we raise concerns with the idea that location should be considered a gestural component of sign languages. We argue that psycholinguistic studies provide evidence for location as a "categorical" element of signs. More generally, we propose that the use of space in sign languages comes in many flavours and may be both categorical and imagistic. In their target article, Goldin-Meadow & Brentari (G-M&B) discuss several observations suggesting that the use of space is imagistic and may not form part of the categorical properties of sign languages. Specifically, they point out that (1) the number of locations toward which agreeing verbs can be directed is not part of a discrete set, (2) event descriptions by users of different sign languages and hearing nonsigners exhibit marked similarities in the use of space, and (3) location as a phonological parameter is not categorically perceived by native signers. It should be noted that G-M&B acknowledge that categorical properties of location and movement may simply not have been captured yet because the proper investigative tools are not yet readily available.

Fingerspelled and Printed Words Are Recoded into a Speech-based Code in Short-term Memory.

We conducted three immediate serial recall experiments that manipulated type of stimulus presentation (printed or fingerspelled words) and word similarity (speech-based or manual). Matched deaf American Sign Languagesigners and hearing non-signers participated (mean reading age=14-15 years). Speech-based similarity effects were found for both stimulus types indicating that deaf signers recoded both printed and fingerspelled words into a speech-based phonological code. A manual similarity effect was not observed for printed words indicating that print was not recoded into fingerspelling (FS). A manual similarity effect was observed for fingerspelled words when similarity was based on joint angles rather than on handshape compactness. However, a follow-up experiment suggested that the manual similarity effect was due to perceptual confusion at encoding. Overall, these findings suggest that FS is strongly linked to English phonology for deaf adult signers who are relatively skilled readers. This link between fingerspelled words and English phonology allows for the use of a more efficient speech-based code for retaining fingerspelled words in short-term memory and may strengthen the representation of English vocabulary.

ASL-LEX: A lexical database of American Sign Language.

ASL-LEX is a lexical database that catalogues information about nearly 1,000 signs in American Sign Language (ASL). It includes the following information: subjective frequency ratings from 25-31 deaf signers, iconicity ratings from 21-37 hearing non-signers, videoclip duration, sign length (onset and offset), grammatical class, and whether the sign is initialized, a fingerspelled loan sign, or a compound. Information about English translations is available for a subset of signs (e.g., alternate translations, translation consistency). In addition, phonological properties (sign type, selected fingers, flexion, major and minor location, and movement) were coded and used to generate sub-lexical frequency and neighborhood density estimates. ASL-LEX is intended for use by researchers, educators, and students who are interested in the properties of the ASL lexicon. An interactive website where the database can be browsed and downloaded is available at http://asl-lex.org .

Measuring conventionalization in the manual modality

AbstractGestures produced by users of spoken languages differ from signs produced by users of sign languages in that gestures are more typically ad hoc and idiosyncratic, while signs are more typically conventionalized and shared within a language community. To measure how gestures may change over time as a result of the process of conventionalization, we used a social coordination game to elicit repeated silent gestures from hearing non-signers, and used Microsoft Kinect to unobtrusively track the movement of their bodies as they gestured. Our approach follows from a tradition of laboratory experiments designed to study language evolution and draws upon insights from sign language research on language emergence. Working with silent gesture, we were able to simulate and quantify hallmarks of conventionalization that have been described for sign languages, in the laboratory. With Kinect, we measured a reduction in the size of the articulatory space and a decrease in the distance traveled by the articulators, while communicative success increased between participants over time. This approach opens the door for more direct future comparisons between ad hoc gestures produced in the lab and natural sign languages in the world.

Preexisting semantic representation improves working memory performance in the visuospatial domain.

Working memory (WM) for spoken language improves when the to-be-remembered items correspond to preexisting representations in long-term memory. We investigated whether this effect generalizes to the visuospatial domain by administering a visual n-back WM task to deaf signers and hearing signers, as well as to hearing nonsigners. Four different kinds of stimuli were presented: British Sign Language (BSL; familiar to the signers), Swedish Sign Language (SSL; unfamiliar), nonsigns, and nonlinguistic manual actions. The hearing signers performed better with BSL than with SSL, demonstrating a facilitatory effect of preexisting semantic representation. The deaf signers also performed better with BSL than with SSL, but only when WM load was high. No effect of preexisting phonological representation was detected. The deaf signers performed better than the hearing nonsigners with all sign-based materials, but this effect did not generalize to nonlinguistic manual actions. We argue that deaf signers, who are highly reliant on visual information for communication, develop expertise in processing sign-based items, even when those items do not have preexisting semantic or phonological representations. Preexisting semantic representation, however, enhances the quality of the gesture-based representations temporarily maintained in WM by this group, thereby releasing WM resources to deal with increased load. Hearing signers, on the other hand, may make strategic use of their speech-based representations for mnemonic purposes. The overall pattern of results is in line with flexible-resource models of WM.

Effects of Hearing Status and Sign Language Use on Working Memory

Deaf individuals have been found to score lower than hearing individuals across a variety of memory tasks involving both verbal and nonverbal stimuli, particularly those requiring retention of serial order. Deaf individuals who are native signers, meanwhile, have been found to score higher on visual-spatial memory tasks than on verbal-sequential tasks and higher on some visual-spatial tasks than hearing nonsigners. However, hearing status and preferred language modality (signed or spoken) frequently are confounded in such studies. That situation is resolved in the present study by including deaf students who use spoken language and sign language interpreting students (hearing signers) as well as deaf signers and hearing nonsigners. Three complex memory span tasks revealed overall advantages for hearing signers and nonsigners over both deaf signers and deaf nonsigners on 2 tasks involving memory for verbal stimuli (letters). There were no differences among the groups on the task involving visual-spatial stimuli. The results are consistent with and extend recent findings concerning the effects of hearing status and language on memory and are discussed in terms of language modality, hearing status, and cognitive abilities among deaf and hearing individuals.

Connections between Fingerspelling and Print: The Impact of Working Memory and Temporal Dynamics on Lexical Activation

Recently there has been a renewed interest in characterizing the role of fingerspelling for deaf readers. The present study takes a step back and creates a theoretical foundation for investigating similarities between fingerspelling and print decoding in hearing signers. In this way, we can probe the constraints of temporal processing and memory on L1 orthography and the processing of L2 fingerspelling. Using a cross-modal priming paradigm, the role of orthography and phonology in print and fingerspelling word recognition was investigated. Results indicate significant inhibition in target retrieval when the prime was fingerspelled but not when it was presented in print. It was hypothesized that inhibition was due to either recoding or prime temporal dynamics. Hearing nonsigners were tested with serially or simultaneously presented print to determine the role of recoding and temporal dynamics. The results suggest that: (1) difficulties in processing fingerspelling for L2 learners might arise from recoding back into an L1 orthographic representation; (2) working memory abilities may reduce inhibition caused by recoding in L2 learners; (3) serial presentation of an orthographic code, either manual or visual, reduces priming effects; and (4) letter position differences provide evidence of depletion of activation over time.

Monitoring Different Phonological Parameters of Sign Language Engages the Same Cortical Language Network but Distinctive Perceptual Ones.

The study of signed languages allows the dissociation of sensorimotor and cognitive neural components of the language signal. Here we investigated the neurocognitive processes underlying the monitoring of two phonological parameters of sign languages: handshape and location. Our goal was to determine if brain regions processing sensorimotor characteristics of different phonological parameters of sign languages were also involved in phonological processing, with their activity being modulated by the linguistic content of manual actions. We conducted an fMRI experiment using manual actions varying in phonological structure and semantics: (1) signs of a familiar sign language (British Sign Language), (2) signs of an unfamiliar sign language (Swedish Sign Language), and (3) invented nonsigns that violate the phonological rules of British Sign Language and Swedish Sign Language or consist of nonoccurring combinations of phonological parameters. Three groups of participants were tested: deaf native signers, deaf nonsigners, and hearing nonsigners. Results show that the linguistic processing of different phonological parameters of sign language is independent of the sensorimotor characteristics of the language signal. Handshape and location were processed by different perceptual and task-related brain networks but recruited the same language areas. The semantic content of the stimuli did not influence this process, but phonological structure did, with nonsigns being associated with longer RTs and stronger activations in an action observation network in all participants and in the supramarginal gyrus exclusively in deaf signers. These results suggest higher processing demands for stimuli that contravene the phonological rules of a signed language, independently of previous knowledge of signed languages. We suggest that the phonological characteristics of a language may arise as a consequence of more efficient neural processing for its perception and production.

Bimodal bilingualism as multisensory training?: Evidence for improved audiovisual speech perception after sign language exposure

The aim of the present study was to characterize effects of learning a sign language on the processing of a spoken language. Specifically, audiovisual phoneme comprehension was assessed before and after 13 weeks of sign language exposure. L2 ASL learners performed this task in the fMRI scanner. Results indicated that L2 American Sign Language (ASL) learners’ behavioral classification of the speech sounds improved with time compared to hearing nonsigners. Results indicated increased activation in the supramarginal gyrus (SMG) after sign language exposure, which suggests concomitant increased phonological processing of speech. A multiple regression analysis indicated that learner’s rating on co-sign speech use and lipreading ability was correlated with SMG activation. This pattern of results indicates that the increased use of mouthing and possibly lipreading during sign language acquisition may concurrently improve audiovisual speech processing in budding hearing bimodal bilinguals.

Language lateralization of hearing native signers: A functional transcranial Doppler sonography (fTCD) study of speech and sign production

Neuroimaging studies suggest greater involvement of the left parietal lobe in sign language compared to speech production. This stronger activation might be linked to the specific demands of sign encoding and proprioceptive monitoring.In Experiment 1 we investigate hemispheric lateralization during sign and speech generation in hearing native users of English and British Sign Language (BSL). Participants exhibited stronger lateralization during BSL than English production. In Experiment 2 we investigated whether this increased lateralization index could be due exclusively to the higher motoric demands of sign production. Sign naïve participants performed a phonological fluency task in English and a non-sign repetition task. Participants were left lateralized in the phonological fluency task but there was no consistent pattern of lateralization for the non-sign repetition in these hearing non-signers.The current data demonstrate stronger left hemisphere lateralization for producing signs than speech, which was not primarily driven by motoric articulatory demands.

Input processing at first exposure to a sign language

There is growing interest in learners’ cognitive capacities to process a second language (L2) at first exposure to the target language. Evidence suggests that L2 learners are capable of processing novel words by exploiting phonological information from their first language (L1). Hearing adult learners of a sign language, however, cannot fall back on their L1 to process novel signs because the modality differences between speech (aural–oral) and sign (visual-manual) do not allow for direct cross-linguistic influence. Sign language learners might use alternative strategies to process input expressed in the manual channel. Learners may rely on iconicity, the direct relationship between a sign and its referent. Evidence up to now has shown that iconicity facilitates learning in non-signers, but it is unclear whether it also facilitates sign production. In order to fill this gap, the present study investigated how iconicity influenced articulation of the phonological components of signs. In Study 1, hearing non-signers viewed a set of iconic and arbitrary signs along with their English translations and repeated the signs as accurately as possible immediately after. The results show that participants imitated iconic signs significantly less accurately than arbitrary signs. In Study 2, a second group of hearing non-signers imitated the same set of signs but without the accompanying English translations. The same lower accuracy for iconic signs was observed. We argue that learners rely on iconicity to process manual input because it brings familiarity to the target (sign) language. However, this reliance comes at a cost as it leads to a more superficial processing of the signs’ full phonetic form. The present findings add to our understanding of learners’ cognitive capacities at first exposure to a signed L2, and raises new theoretical questions in the field of second language acquisition.

Perceptual categorization of handling handshapes in British Sign Language

abstractSign languages like British Sign Language (BSL) include partially lexicalized constructions depicting object handling or manipulation – handling constructions. Object sizes gradiently vary, yet it is unclear whether handling handshapes depict handled objects categorically or gradiently. This study investigates the influence of sign language experience on perception of handling handshapes. Deaf signers and hearing non-signers completed perceptual handshape identification and discrimination tasks. We examined whether deaf BSL signers perceived handshape continua categorically or continuously compared with hearing non-signers, and whether reaction times were modulated by linguistic representations. The results revealed similar binary categorization of dynamically presented handling handshapes as deaf and hearing perceivers displayed higher discrimination accuracy on category boundaries, and lower, but above chance, within-category discrimination, suggesting that perceptual categorization was not uniquely mediated by linguistic experience. However, RTs revealed critical differences between groups in processing times; deaf signers’ RTs reflected stronger category bias and increased sensitivity to boundaries, suggesting underlying linguistic representations. Further, handshape variability within categories influenced deaf signers’ discrimination RTs in a manner that suggested graded category organization, with handshape prototype grounding the category. These findings provide an insight into the internal organization of handling handshapes and highlight the complex relationship between sign language, cognition, and gesture.

Neural correlates of fingerspelling, text, and sign processing in deaf American Sign Language–English bilinguals

We used functional magnetic resonance imaging to identify the neural regions that support comprehension of fingerspelled words, printed words, and American Sign Language (ASL) signs in deaf ASL–English bilinguals. Participants made semantic judgements (concrete–abstract?) to each lexical type, and hearing non-signers served as controls. All three lexical types engaged a left frontotemporal circuit associated with lexical semantic processing. Both printed and fingerspelled words activated the visual word form area for deaf signers only. Fingerspelled words were more left lateralised than signs, paralleling the difference between reading and listening for spoken language. Greater activation in left supramarginal gyrus was observed for signs compared to fingerspelled words, supporting its role in processing sign-specific phonology. Fingerspelling ability was negatively correlated with activation in left occipital cortex, while ASL ability was negatively correlated with activation in right angular gyrus. Overall, the results reveal both overlapping and distinct neural regions for comprehension of signs, text, and fingerspelling.

Production and comprehension show divergent constituent order preferences: Evidence from elicited pantomime.

All natural languages develop devices to communicate who did what to whom. Elicited pantomime provides one model for studying this process, by providing a window into how humans (hearing non-signers) behave in a natural communicative modality (silent gesture) without established conventions from a grammar. Most studies in this paradigm focus on production, although they sometimes make assumptions about how comprehenders would likely behave. Here, we directly assess how naïve speakers of English (Experiments 1 & 2), Korean (Experiment 1), and Turkish (Experiment 2) comprehend pantomimed descriptions of transitive events, which are either semantically reversible (Experiments 1 & 2) or not (Experiment 2). Contrary to previous assumptions, we find no evidence that Person-Person-Action sequences are ambiguous to comprehenders, who simply adopt an agent-first parsing heuristic for all constituent orders. We do find that Person-Action-Person sequences yield the most consistent interpretations, even in native speakers of SOV languages. The full range of behavior in both production and comprehension provides counter-evidence to the notion that producers' utterances are motivated by the needs of comprehenders. Instead, we argue that production and comprehension are subject to different sets of cognitive pressures, and that the dynamic interaction between these competing pressures can help explain synchronic and diachronic constituent order phenomena in natural human languages, both signed and spoken.