Bimodal Language Research Articles

A developmental model of audio-visual attention (MAVA) for bimodal language learning in infants and robots

A social individual needs to effectively manage the amount of complex information in his or her environment relative to his or her own purpose to obtain relevant information. This paper presents a neural architecture aiming to reproduce attention mechanisms (alerting/orienting/selecting) that are efficient in humans during audiovisual tasks in robots. We evaluated the system based on its ability to identify relevant sources of information on faces of subjects emitting vowels. We propose a developmental model of audio-visual attention (MAVA) combining Hebbian learning and a competition between saliency maps based on visual movement and audio energy. MAVA effectively combines bottom-up and top-down information to orient the system toward pertinent areas. The system has several advantages, including online and autonomous learning abilities, low computation time and robustness to environmental noise. MAVA outperforms other artificial models for detecting speech sources under various noise conditions.

Testing the bilingual advantage for executive function: insights from hearing children who are native signers

ABSTRACT Bimodal bilingualism involves the use of a sign language and a spoken language, and offers a unique opportunity to explore the cognitive effects of growing up bilingual. The aim of this study was to investigate the relationship between bimodal bilingualism and executive function (EF) in hearing children who are native users of a sign language. We studied three groups of children: bimodal bilinguals (users of a sign language and a spoken language), unimodal bilinguals (users of two spoken languages), and monolinguals, performing three cognitive tasks to measure different components of EF: visuospatial working memory (Odd One Out task), cognitive flexibility (Children’s Colored Trails Test) and conflict resolution (Simon task). All groups of children obtained similar scores on the Odd One Out task and the Children’s Colored Trails Test. Bimodal bilinguals displayed distinct patterns of conflict resolution in comparison to monolingual children: bimodal bilinguals had better overall accuracy and slower overall reaction time in the Simon task. The subsequent analysis did not find a straightforward trade-off between speed and accuracy. These results suggest that bimodal language experience and/or visual-spatial language usage may explain the small bimodal bilingual advantage in the Simon task that we observed in hearing children who are native signers.

The different brain areas occupied for integrating information of hierarchical linguistic units: a study based on EEG and TMS.

Human language units are hierarchical, and reading acquisition involves integrating multisensory information (typically from auditory and visual modalities) to access meaning. However, it is unclear how the brain processes and integrates language information at different linguistic units (words, phrases, and sentences) provided simultaneously in auditory and visual modalities. To address the issue, we presented participants with sequences of short Chinese sentences through auditory, visual, or combined audio-visual modalities while electroencephalographic responses were recorded. With a frequency tagging approach, we analyzed the neural representations of basic linguistic units (i.e. characters/monosyllabic words) and higher-level linguistic structures (i.e. phrases and sentences) across the 3 modalities separately. We found that audio-visual integration occurs in all linguistic units, and the brain areas involved in the integration varied across different linguistic levels. In particular, the integration of sentences activated the local left prefrontal area. Therefore, we used continuous theta-burst stimulation to verify that the left prefrontal cortex plays a vital role in the audio-visual integration of sentence information. Our findings suggest the advantage of bimodal language comprehension at hierarchical stages in language-related information processing and provide evidence for the causal role of the left prefrontal regions in processing information of audio-visual sentences.

Logics of (In)sane and (Un)reliable Beliefs

Abstract Inspired by an interesting quotation from the literature, we propose four modalities, called ‘sane belief’, ‘insane belief’, ‘reliable belief’ and ‘unreliable belief’, and introduce logics with each operator as the modal primitive. We show that the four modalities constitute a square of opposition, which indicates some interesting relationships among them. We compare the relative expressivity of these logics and other related logics, including a logic of false beliefs from the literature. The four main logics are all less expressive than the standard modal logic over various model classes, and the logics of sane and insane beliefs are, respectively, equally expressive as the logics of unreliable and reliable beliefs on any class of models. The logics of reliable and unreliable beliefs are then combined into a bimodal logic, which turns out to be equally expressive as the standard modal logic. Despite this, we cannot obtain a complete axiomatization of the minimal bimodal logic, by simply translating the axioms and rules of the minimal modal logic $\textbf{K}$ into the bimodal language. We then introduce a schematic modality which unifies reliable and unreliable beliefs and axiomatize it over the class of all frames and also the class of serial frames. This line of research is finally extended to unify sane and insane beliefs and some axiomatizations are given.

Modality-specific effects in bilingual language perception

ABSTRACTLanguage switching typically leads to language switch costs. Previous studies demonstrated that production-based language switching often results in reduced costs when the languages are assigned to distinct modalities (i.e. bimodal language switching) compared to switching within a modality (i.e. unimodal switching). In Experiment 1–3, we compared unimodal to bimodal switching in language perception. In Experiment 4, unimodal switching was compared to bivocal switching (i.e. languages were assigned to different voices). Experiment 1–3 revealed significantly higher switch costs for bimodal than for unimodal switching, indicating fundamental differences to language production studies. While bimodal switching leads to no (additional) costs but – under specific conditions – even to advantage in production, it increases switch costs in language perception. Moreover, Experiment 4 indicates that this is more a modality-specific rather than a general perception-based effect. Different voices instead of different modalities evoked no similar increase in switch costs.

Modality effects in language switching: Evidence for a bimodal advantage

In language switching, it is assumed that in order to produce a response in one language, the other language must be inhibited. In unimodal (spoken-spoken) language switching, the fact that the languages share the same primary output channel (the mouth) means that only one language can be produced at a time. In bimodal (spoken-signed) language switching, however, it is possible to produce both languages simultaneously. In our study, we examined modality effects in language switching using multilingual subjects (speaking German, English, and German Sign Language). Focusing on German vocal responses, since they are directly compatible across conditions, we found shorter reaction times, lower error rates, and smaller switch costs in bimodal vs. unimodal switching. This result suggests that there are different inhibitory mechanisms at work in unimodal and bimodal language switching. We propose that lexical inhibition is involved in unimodal switching, whereas output channel inhibition is involved in bimodal switching.

Building a single proposition from imagistic and categorical components.

Bimodal bilingual language provides further evidence for the viewpoint advocated by Goldin-Meadow & Brentari (G-M&B) that sign, speech, and gesture work together to create a single proposition, illustrating the potential in each set of articulators for both imagistic and categorical components. Recent advances in formal semantics provide a framework for incorporating both imagistic and categorical components into a single compositional system.

Bimodal language switching: New insights from signing and typing

Abstract Language switching typically refers to unimodal switching between two spoken languages. In bimodal language switching, one language is produced vocally and the other language is produced manually (e.g., a sign language). We compared unimodal language switching with two different kinds of bimodal language switching for non-signers. In Experiment 1a and 1b participants switched between speaking and newly learned signs. Participants of Experiment 2 switched between two originally spoken languages in a bimodal way: One language was produced vocally and the other language was produced manually (by typing). Interestingly, switch costs were reduced for bimodal compared to unimodal switching in Experiment 1a and 1b but not in Experiment 2. This speaks for different language control mechanisms. We assume less costly output channel inhibition for switching between speaking and signing (Experiment 1a and 1b) and more costly lexical inhibition for switching between speaking and typing (Experiment 2).

Spoken Language Activation Alters Subsequent Sign Language Activation in L2 Learners of American Sign Language.

A large body of literature has characterized unimodal monolingual and bilingual lexicons and how neighborhood density affects lexical access; however there have been relatively fewer studies that generalize these findings to bimodal (M2) second language (L2) learners of sign languages. The goal of the current study was to investigate parallel language activation in M2L2 learners of sign language and to characterize the influence of spoken language and sign language neighborhood density on the activation of ASL signs. A priming paradigm was used in which the neighbors of the sign target were activated with a spoken English word and compared the activation of the targets in sparse and dense neighborhoods. Neighborhood density effects in auditory primed lexical decision task were then compared to previous reports of native deaf signers who were only processing sign language. Results indicated reversed neighborhood density effects in M2L2 learners relative to those in deaf signers such that there were inhibitory effects of handshape density and facilitatory effects of location density. Additionally, increased inhibition for signs in dense handshape neighborhoods was greater for high proficiency L2 learners. These findings support recent models of the hearing bimodal bilingual lexicon, which posit lateral links between spoken language and sign language lexical representations.

Language-switch costs and dual-response costs in bimodal bilingual language production

In communication, different forms of language combinations are possible for bimodal bilinguals, who use a spoken and a signed language. They can either switch from one language to another (language switching) or produce a word and a sign simultaneously (language blending). The present study examines language control mechanisms in language switching and simultaneous bimodal language production, comparing single-response (German or German Sign Language) and dual-response trials (Blend of the German word and the German Sign Language sign). There were three pure blocks, one for each Target-response (German, German Sign Language, Blend), as well as mixed blocks, in which participants switched between all three Target-responses. We observed language mixing costs, switch costs and dual-response costs. Further, the data pattern showed a specific dual-response advantage for switching into a Blend (i.e., a dual-response trial), indicating the specific nature of a blended response in bimodal bilingual language production.

Simultaneous perception of a spoken and a signed language: The brain basis of ASL-English code-blends

Code-blends (simultaneous words and signs) are a unique characteristic of bimodal bilingual communication. Using fMRI, we investigated code-blend comprehension in hearing native ASL-English bilinguals who made a semantic decision (edible?) about signs, audiovisual words, and semantically equivalent code-blends. English and ASL recruited a similar fronto-temporal network with expected modality differences: stronger activation for English in auditory regions of bilateral superior temporal cortex, and stronger activation for ASL in bilateral occipitotemporal visual regions and left parietal cortex. Code-blend comprehension elicited activity in a combination of these regions, and no cognitive control regions were additionally recruited. Furthermore, code-blends elicited reduced activation relative to ASL presented alone in bilateral prefrontal and visual extrastriate cortices, and relative to English alone in auditory association cortex. Consistent with behavioral facilitation observed during semantic decisions, the findings suggest that redundant semantic content induces more efficient neural processing in language and sensory regions during bimodal language integration.

Bimodal bilingual language development of hearing children of deaf parents

Adopting a bimodal bilingual language acquisition model, this qualitative case study is the first in Germany to investigate the spoken and sign language development of hearing children of deaf adults (codas). The spoken language competence of six codas within the age range of 3;10 to 6;4 is assessed by a series of standardised tests (SETK 3-5, TROG-D, PDSS). Additionally, a language assessment tool designed specifically for hearing bilingual children named HAVAS 5 is employed to measure both the children’s spoken and sign language abilities. A heterogeneous picture emerges: all codas show abilities equal to those of monomodal monolingual children in their age range with regard to German sentence structure and verb inflection and an extensive receptive vocabulary, but deviations from the age norm are observed for the production of verbs and prepositions. Furthermore, three codas show below average T-values in some grammatical subtests. Overall, the findings suggest that at least some codas may acquire both of their languages (i.e. spoken and signed German) simultaneously. However, our study also indicates that the spoken language development of some codas as well as the acquisition context of this minority group strongly resembles a form of successive language acquisition that is known from children who acquire German as an early second language in a migration background.

The Impact of Bimodal Bilingual Parental Input on the Communication and Language Development of a Young Deaf Child

This study explores the impact of bimodal bilingual parental input on the communication and language development of a young deaf child. The participants in this case study were a severe-to-profoundly deaf boy and his hearing parents, who were enrolled in a bilingual (English and Australian Sign Language) home-based early intervention programme. The parents typified the challenges faced by hearing parents of deaf children seeking to provide a bimodal language environment for their deaf child. Data were collected over 20 months when the boy was aged between 23 and 42 months. The parents' level of Auslan proficiency, bilingual practice, and sensitivity to their son's communication needs were measured. Findings indicated that although communication and language outcomes were delayed compared with hearing peers, English and Auslan vocabulary growth over time was strongly correlated with the parents' sensitivity to his communication needs. Findings also indicated that there were several child-led changes in language modality preferences throughout the study.

The Ease of Language Understanding (ELU) model: theoretical, empirical, and clinical advances

Working memory is important for online language processing during conversation. We use it to maintain relevant information, to inhibit or ignore irrelevant information, and to attend to conversation selectively. Working memory helps us to keep track of and actively participate in conversation, including taking turns and following the gist. This paper examines the Ease of Language Understanding model (i.e., the ELU model, Rönnberg, 2003; Rönnberg et al., 2008) in light of new behavioral and neural findings concerning the role of working memory capacity (WMC) in uni-modal and bimodal language processing. The new ELU model is a meaning prediction system that depends on phonological and semantic interactions in rapid implicit and slower explicit processing mechanisms that both depend on WMC albeit in different ways. It is based on findings that address the relationship between WMC and (a) early attention processes in listening to speech, (b) signal processing in hearing aids and its effects on short-term memory, (c) inhibition of speech maskers and its effect on episodic long-term memory, (d) the effects of hearing impairment on episodic and semantic long-term memory, and finally, (e) listening effort. New predictions and clinical implications are outlined. Comparisons with other WMC and speech perception models are made.

The Iterative Conception of Set

The use of tensed language and the metaphor of set ‘formation’ found in informal descriptions of the iterative conception of set are seldom taken at all seriously. Both are eliminated in the nonmodal stage theories that formalise this account. To avoid the paradoxes, such accounts deny the Maximality thesis, the compelling thesis that any sets can form a set. This paper seeks to save the Maximality thesis by taking the tense more seriously than has been customary (although not literally). A modal stage theory, \({\textsf{MST}}\), is developed in a bimodal language, governed by a tenselike logic. Such a language permits a very natural axiomatisation of the iterative conception, which upholds the Maximality thesis. It is argued that the modal approach is consonant with mathematical practice and a plausible metaphysics of sets and shown that \({\textsf{MST}}\) interprets a natural extension of Zermelo set theory less the axiom of Infinity and, when extended with a further axiom concerning the extent of the hierarchy, interprets Zermelo–Fraenkel set theory.

Hearing, Mother Father Deaf: Hearing People in Deaf Families (review)

Hearing, Mother Father Deaf: Hearing People in Deaf Families, ed. Michele Bishop and Sherry L. Hicks (Washington, D. C: Gallaudet University Press, 2008, 340 pp., $85.00, ISBN 978-1563683978). THIS SEMINAL book provides an in-depth look at language use drawn from several studies on hearing bimodal bilinguals (hearing individuals with Deaf parents, also known as Codas). This volume is significant not only because of the research but also because the editors selected a range of international studies that provide variation in the languages and issues discussed. The chapters in this volume will appeal to researchers, educators, linguists, those who identify as Codas, and those who have deaf parents but do not identify with the Coda label. Hearing, Mother Father Deaf is divided into three parts: bimodal bilingualism in adults; Koda (kids of deaf parents); and coda identity and experience. The organization of the book flows nicely: Content in one chapter provides information that is useful for understanding later chapters. The studies range from controlled experiments to reports of personal experiences or anecdotal evidence. In their introduction, Bishop and Hicks provide an extensive review of issues relevant to the Deaf community and that affect Codas. Following the introduction, the chapters in the first section are Bimodal Bilingualism (Emmorey, Borinstein, Thompson, and Gollan), The Face of Bimodal Bilingualism: ASL Grammatical Markers Are Produced When Bilinguals Speak to English Monolinguals (Pyers and Emmorey), and Coda Talk: Bimodal Discourse among Hearing, Native Signers (Bishop and Hicks). These chapters focus on adult ASLEnglish hearing bimodal bilinguals and examine a specific type of bilingualism in which both languages can be produced at the same time due to the bimodal nature of the languages. In part II the focus shifts to younger children of Deaf parents, Kodas. These chapters explore issues related to the languages used by Kodas in three countries, NGT-Dutch, ASL-English, and DGS-German. The chapter titles are Bimodal Language Acquisition in Kodas (Van den Bogaerde and Baker); Eyes over Ears: The Development of Visual Strategies by Hearing Children of Deaf Parents (Mather and Andrews), and Sociolinguistic Aspects of the Communication between Hearing Children and Deaf Parents (Wilhelm) . Part III has only one chapter, Brazilian Codas: Libras and Portuguese in Contact Zones, by Muller de Quadros and Masutti. The three chapters in part IV share insights into Coda identity and experience. The chapters are Exploring Linguistic and Cultural Identity: My Personal Experience (Napier); Virgilean Guides: Esthetic Subjects in Coda Autobiographies (Ataman); and Characteristics of the Coda Experience in 21st-century Contemporary Culture (Adams). The articles in this section are about hearing individuals with deaf parents who use BSL, Auslan, DGS, and/or ASL. In the remainder of this review, I provide an overview of the information reported in each chapter but encourage readers to read each chapter in its entirety. Although a few sentences in the book are confusing due to typing errors and editing, the studies presented in this volume provide information and insight into a group that has much to offer toward a better understanding of bilingualism and biculturalism, first and second language acquisition, how the brain works with languages, language attitudes, and how some people navigate between the hearing and the deaf worlds. Part I. Bimodal Bilingualism in Adults By studying adult bimodal bilinguals, the researchers in this section provide insight into code blending and language suppression. In the first chapter, Emmorey et al. propose a model of hearing bimodal/ bilingual language production. They report on a study in which they examined conversations and elicited narratives for code blends and code switches in order to determine the characteristics of each. The authors discuss code blending, code switching, and language suppression or language selection. …

Semantical Characterizations for Irreflexive and Generalized Modal Languages

This paper deals with two main topics: One is a semantical investigation for a bimodal language with a modal operator \blacksquare associated with the intersection of the accessibility relation R and the inequality ≠. The other is a generalization of some of the former results to general extended languages with modal operators. First, for our language L\sb{\square\blacksquare}, we prove that Segerberg's theorem (equivalence between finite frame property and finite model property) fails and establish both van Benthem-style and Goldblatt-Thomason-style characterizations. We extract the notion of \blacksquare-realizer (a generalization of bulldozing) as an essence from the proofs of our results. Second, we generalize the notion of \blacksquare-realizer and prove quite general versions of these semantical characterization results. The known and previously unknown characterization results for almost all of the languages extended with modal operators already proposed will be immediate corollaries.

非反射的様相をもつ二様相論理

This paper proposes a bimodal logic with an additional modality (called the irreflexive modality), which corresponds semantically to the intersection of the accessibility relation and the inequality. First, we show that we can define, within this framework, several properties that are undefinable in the unimodal language; irreflexivity is one of such properties. Second, with respect to the frame expressivity, we compare our language with the unimodal language and another bimodal language with the difference operator that is studied by de Rijke. Finally, we give a Hilbert-style axiomatization of our logic and prove that certain familiar modal systems, such as S4 and S5, enjoy Kripke completeness in our language.

Bimodaal Codewisselen

In this article, the authors combine the insights from two linguistic sub disciplines: language contact and sign language linguistics. After a short introduction about some possible outcomes of language contact the choice for code switching and an analysis along the lines of Myers-Scottons Matrix Language model is made. First, the characteristics of bimodal language contact are discussed, and then the dataset is presented: a bimodal corpus by a deaf mother and her hearing son Jonas at the ages of approximately 3;0 and 6;0. Analyses show that a large part of the utterances is code blended, this means simultaneously signed and spoken. Even after a closer reanalysis it turns out to be difficult in many instances to decide what the matrix language is, although in many instances it is NGT.

« Everywhere » and « here »

ABSTRACT The paper studies propositional logics in a bimodal language, in which the first modality is interpreted as the local truth, and the second as the universal truth. The logic S4UC is introduced, which is finitely axiomatizable, has the f.m.p. and is determined by every connected separable metric space.