Non-linguistic Cues Research Articles

Coding Emotion in Computer-Mediated Communication : The Example of YouTube Comments

Computer-mediated communication (CMC) is essentially text-based. This is generally said to result in a “paucity of paralinguistic and non-linguistic cues” (Bieswanger, 2013 : 468), but it is also considered to be the main reason for the development of other characteristic non-verbal modalities, such as emoticons and emoji. How are, then, emotions expressed in such a specific context ? Are verbal modalities as limited as they are said to be ? Do non-verbal modalities complete or replace them ? This paper will try to answer those questions using discourse analysis tools applied to a sample of comments on a YouTube video.

When speaker identity is unavoidable: Neural processing of speaker identity cues in natural speech

Speech sound acoustic properties vary largely across speakers and accents. When perceiving speech, adult listeners normally disregard non-linguistic variation caused by speaker or accent differences, in order to comprehend the linguistic message, e.g. to correctly identify a speech sound or a word. Here we tested whether the process of normalizing speaker and accent differences, facilitating the recognition of linguistic information, is found at the level of neural processing, and whether it is modulated by the listeners’ native language. In a multi-deviant oddball paradigm, native and nonnative speakers of Dutch were exposed to naturally-produced Dutch vowels varying in speaker, sex, accent, and phoneme identity. Unexpectedly, the analysis of mismatch negativity (MMN) amplitudes elicited by each type of change shows a large degree of early perceptual sensitivity to non-linguistic cues. This finding on perception of naturally-produced stimuli contrasts with previous studies examining the perception of synthetic stimuli wherein adult listeners automatically disregard acoustic cues to speaker identity. The present finding bears relevance to speech normalization theories, suggesting that at an unattended level of processing, listeners are indeed sensitive to changes in fundamental frequency in natural speech tokens.

Effects of attentional and lexical priming on syntactic production in aphasia: eye-tracking while speaking

Effects of attentional and lexical priming on syntactic production in aphasia: eye-tracking while speaking

Children Increase Their Sensitivity to a Speaker's Nonlinguistic Cues Following a Communicative Breakdown.

Bilingual children regularly face communicative challenges when speakers switch languages. To cope with such challenges, children may attempt to discern a speaker's communicative intent, thereby heightening their sensitivity to nonverbal communicative cues. Two studies examined whether such communication breakdowns increase sensitivity to nonverbal cues. English-speaking monolingual (n = 64) and bilingual (n = 54) 3- to 4-year-olds heard instructions in either English only or English mixed with a foreign language. Later, children played a hiding game that relied on nonverbal cues. Hearing a foreign language switch improved both monolingual and bilingual children's use of these cues. Moreover, bilinguals with more prior code-switching exposure outperformed those with less prior code-switching exposure. Children's short-term strategies to repair communication breakdowns may evolve into a more generalizable set of skills.

Pedagogical Cues to an Artist’s Intention in Young Children’s Understanding of Drawings

Three studies investigated the effects of pedagogical cues to an artist’s referential intention on 2- and 2.5-year-old children’s understanding of drawings in a matching task without verbal labels support. Results showed that pedagogical cues, the combination of the artist’s eye gaze while she was creating the drawings (nonlinguistic cues), and verbal descriptions about her graphic actions (linguistic cues) enabled 2-year-olds to match highly realistic line drawings with referents. However, 2-year-olds’ performance was not influenced to an equal degree by nonlinguistic and linguistic cues; verbal scripts appeared to be the critical aspect of pedagogical demonstration even with predrawn pictures. By contrast, at 2.5 years of age, children inferred the artist’s intention when comprehending drawings in the absence of pedagogical cues. This research illustrates the potential power of pedagogical demonstration to communicate referential intentions in the pictorial symbol domain.

Interlocutor identity affects language activation in bilinguals

In bilingual communities, individuals often communicate in one of their languages only, and they adjust to the linguistic background of different interlocutors with ease. What facilitates such efficiency? We investigated whether bilinguals’ language activation is supported by non-linguistic cues (e.g., interlocutor identity). First, in an audio–visual task, early (proficient) and late (less proficient) Basque–Spanish bilinguals were familiarized with six novel interlocutors who spoke either Spanish, Basque, or both languages. Then, participants completed an audio–visual lexical decision task, in which the interlocutors produced test items in Spanish or Basque. Early, but not late, bilinguals’ speed of processing decreased when the language that the interlocutors spoke during familiarization matched the language they spoke at test, relative to test trials when the interlocutors changed languages. Overall, results suggest that proficient and/or early bilinguals benefit from an association between language and interlocutor during (or even before) language comprehension, because they are able to predict the context-appropriate language based on non-linguistic cues, such as interlocutor context.

Judgments of social awkwardness from brief exposure to children with and without high-functioning autism.

We form first impressions of many traits based on very short interactions. This study examines whether typical adults judge children with high-functioning autism to be more socially awkward than their typically developing peers based on very brief exposure to still images, audio-visual, video-only, or audio-only information. We used video and audio recordings of children with and without high-functioning autism captured during a story-retelling task. Typically developing adults were presented with 1 s and 3 s clips of these children, as well as still images, and asked to judge whether the person in the clip was socially awkward. Our findings show that participants who are naïve to diagnostic differences between the children in the clips judged children with high-functioning autism to be socially awkward at a significantly higher rate than their typically developing peers. These results remain consistent for exposures as short as 1 s to visual and/or auditory information, as well as for still images. These data suggest that typical adults use subtle nonverbal and non-linguistic cues produced by children with high-functioning autism to form rapid judgments of social awkwardness with the potential for significant repercussions in social interactions.

Artist’s referential intention in early understanding of drawings as symbols / La intencionalidad referencial del dibujante: su papel en la comprensión temprana de dibujos como símbolos

In order to examine the influence of artist’s referential intention on early understanding of drawings, two studies were conducted using a matching task. Study 1 explored whether 24-month-old children’s understanding of drawings can be facilitated when an adult draws an object while looking at it carefully (non-linguistic cues) and makes her symbolic intention more explicit to the children through verbal descriptions about her drawing actions (linguistic cues). Study 2 examines if at 30 months of age children are able to solve the task with pre-drawn drawings, without being told the artist’s intention. The results show that the convergence of non-linguistic and linguistic cues enables children to use drawings as symbols at 24 months; only six months later, children spontaneously relate the drawings and their referents. These results are discussed analysing young children’s comprehension of the relationships between drawing, referent, artist and observer.

Two faces, two languages: An fMRI study of bilingual picture naming

This fMRI study explores how nonlinguistic cues modulate lexical activation in the bilingual brain. We examined the influence of face race on bilingual language production in a picture-naming paradigm. Chinese–English bilinguals were presented with pictures of objects and images of faces (Asian or Caucasian). Participants named the picture in their first or second language (Chinese or English) in separate blocks. Face race and naming language were either congruent (e.g., naming in Chinese when seeing an Asian face) or incongruent (e.g., naming in English when seeing an Asian face). Our results revealed that face cues facilitate naming when the socio-cultural identity of the face is congruent with the naming language. The congruence effects are reflected as effective integration of lexical and facial cues in key brain regions including IFG, MFG, ACC, and caudate. Implications of the findings in light of theories of language processing and cultural priming are discussed.

Talker specificity and the perception of grammatical variation

Language processing has been shown to be affected by both recent exposure to linguistic structures (priming effects) and non-linguistic, indexical information (talker effects). Three experiments explored two hypotheses: that sociolinguistically variable subject–verb agreement constructions are susceptible to grammatical priming, and that comprehension of these constructions is susceptible to talker-specificity effects. Experiment 1 tested for grammatical priming effects on the interpretation of two subject–verb agreement constructions, there's+NPSG/PL and NPSG/PL+don't. In a forced choice task, participants chose which of two images represented the subject noun that was in a sentence; participants read an unambiguous prime sentence and then heard an ambiguous target sentence. Experiment 1 found a priming effect: nonstandard primes increased participants’ perception of nonstandard agreement in target sentences. Experiment 2 tested whether this priming could be modulated by non-linguistic cues by manipulating whether the prime and target sentence were spoken by the same talker or by different talkers, cued by voice and photograph. Experiment 2 did not find a talker specificity effect. In an alternative design using identical prime and target sentences, Experiment 3 tested for talker-specific lexical repetition of subject nouns across primes and targets separated by intervening trials. Experiment 3 found a talker effect: participants were better at matching a subject noun in the target block to the subject noun in the prime block when the talker was identical than when the talker differed. These results are discussed in terms of competing linguistic and non-linguistic cues during language processing, and the activation of sociolinguistic representations.

Modelling the coevolution of joint attention and language

Joint attention (JA) is important to many social, communicative activities, including language, and humans exhibit a considerably high level of JA compared with non-human primates. We propose a coevolutionary hypothesis to explain this degree-difference in JA: once JA started to aid linguistic comprehension, along with language evolution, communicative success (CS) during cultural transmission could enhance the levels of JA among language users. We illustrate this hypothesis via a multi-agent computational model, where JA boils down to a genetically transmitted ability to obtain non-linguistic cues aiding comprehension. The simulation results and statistical analysis show that: (i) the level of JA is correlated with the understandability of the emergent language; and (ii) CS can boost an initially low level of JA and 'ratchet' it up to a stable high level. This coevolutionary perspective helps explain the degree-difference in many language-related competences between humans and non-human primates, and reflects the importance of biological evolution, individual learning and cultural transmission to language evolution.

Language Augmented Prediction

Language Augmented Prediction

Decoding intention: A neuroergonomic perspective

Decoding the intentions of other people based on non-linguistic cues such as their body movement is a major requirement of many jobs. Whether it is maintaining security at an airport or negotiating with locals in a foreign country, there is a need to maximize the effectiveness of training or real-time performance in this decoding process. This review considers the potential utility of neuroergonomic solutions, and in particular, of electroencephalographic (EEG) methods for augmenting action understanding. Focus is given to body movements and hand-object interactions, where there is a rapid growth in relevant science. The interpretation of EEG-based signals is reinforced by a consideration of functional magnetic resonance imaging experiments demonstrating underlying brain mechanisms that support goal oriented action. While no EEG method is currently implemented as a practical application for enhancing the understanding of unspoken intentions, there are a number of promising approaches that merit further development.

Abercrombie Unveiled: A Theoretical and Empirical Analysis of Trademark Distinctiveness

The word mark taxonomy established in Abercrombie is a longstanding tenet of doctrinal orthodoxy. Lawmakers have so oft restated the Abercrombie classes and so consistently rehearsed the theoretical grounds that undergird them that it no longer occurs to us to question their premises. But there are good reasons - both pragmatic and theoretical ones - to question the Abercrombie system. At the practical level, it should be noted that the perceived objectivity and predictability of the Abercrombie classification system are a mirage. The law is incessantly and understandably searching for fixed lodestars to guide and cabin judicial and administrative decision-making. Abercrombie offers the appearance of such a lodestar at the threshold stage of evaluating distinctiveness - in the form of a five-level classification system that channels the availability, timing, and degree of protection. Yet although some of those classifications turn on bright-line, objective criteria, that cannot be said of the classification that matters most - of the line between the suggestive and the merely descriptive. Courts and commentators have long lamented the subjectivity and arbitrariness of sorting word marks between these two classifications, and that pragmatic problem gives substantial grounds for an agnostic skepticism toward the Abercrombie orthodoxy. In terms of theory, law is ripe for a thorough reevaluation in the light of the tools employed in the scholarly study of consumer psychology and behavior. The law's reflexive restatement of the grounds for its treatment of merely word marks - that they are unlikely to be perceived by consumers as indicators of source - is a glaring reminder that the law in this field turns on questions well beyond the competence of those trained in the art of legal analysis and interpretation. Where (as here) the law depends on answers to questions that are the focus of an entire field of scholarly study, we cannot afford to proceed in arrogant ignorance of the insights offered by such a field of expertise. In this case, the field of consumer psychology has much to offer the law of distinctiveness. Perceptual schema theory provides ample grounds for doubting the law's myopic focus on the semantic meaning of a descriptive word mark. A consumer who encounters a descriptive word mark in its trademark context may well perceive it as a source indicator on the basis of the mark's non-linguistic cues or indicators of meaning. Those non-linguistic indicators will be provided in large part by the consumer's perceptual models and expectations of a typical brand or source indicator in commercial context. If the descriptive word mark is presented in a spatial layout, size, and placement that matches the consumer's general schematic mental model of what a brand looks like, the word may be perceived as a source indicator even if its semantic meaning is merely descriptive. Our theoretical model provides grounds for skepticism of Abercrombie; our empirical study completely undermines it. In other words, perceptual schema theory offers a reason to wonder whether and to what extent non-linguistic signs may overwhelm the linguistic signs credited by the law-to consider the possibility that even merely word marks might nonetheless be perceived as source indicators when the non-linguistic markers that accompany their use overshadow their semantic meaning. But empirical analysis can - and does - provide concrete answers to the question of the relative impact of the linguistic and non-linguistic indicators of distinctiveness. Our empirical studies show that Abercrombie rests on an erroneous assumption about the predominant impact of semantic word meaning. So long as the word is used in an average use context, a word mark's semantic meaning is shown to be overshadowed by the non-linguistic, contextual markers that establish its distinctiveness as a source indicator.

Putting Social Context into Text: The Semiotics of E‐mail Interaction

E‐mail excludes the multiple nonlinguistic cues and gestures that facilitate face‐to‐face communication. How, then, should interaction in a text‐based context be understood? The authors analyze the problems and solutions experienced by a research panel that communicated over e‐mail and face‐to‐face for 18 months, evaluating both kinds of exchanges alongside survey and interview data. Semiotic and linguistic theory is used to expose essential properties associated with the successful communication of meaning in each context. The authors find that e‐mail requires the cultivation of new techniques for specifically conveying the “pragmatic information” that connects the meaning of words to their users. Such information is assigned in e‐mail through the use of what are termed emphatic, referential, and characterizing semiotic tactics. These tactics are also evident in sustained online interactions studied by other researchers. This theoretical vocabulary represents an alternative to the dominant sociological c...

Automatic acoustic synthesis of human-like laughter

A technique to synthesize laughter based on time-domain behavior of real instances of human laughter is presented. In the speech synthesis community, interest in improving the expressive quality of synthetic speech has grown considerably. While the focus has been on the linguistic aspects, such as precise control of speech intonation to achieve desired expressiveness, inclusion of nonlinguistic cues could further enhance the expressive quality of synthetic speech. Laughter is one such cue used for communicating, say, a happy or amusing context. It can be generated in many varieties and qualities: from a short exhalation to a long full-blown episode. Laughter is modeled at two levels, the overall episode level and at the local call level. The first attempts to capture the overall temporal behavior in a parametric model based on the equations that govern the simple harmonic motion of a mass-spring system is presented. By changing a set of easily available parameters, the authors are able to synthesize a variety of laughter. At the call level, the authors relied on a standard linear prediction based analysis-synthesis model. Results of subjective tests to assess the acceptability and naturalness of the synthetic laughter relative to real human laughter samples are presented.

Reading emotion cues: social communication difficulties in pediatric populations.

Speech-language pathologists frequently address social communication difficulties in children with diverse linguistic profiles. Of consequence to effective management of social communication skills is that some children with language disorders may also have difficulty understanding emotional cues. The ability to recognize and comprehend the emotional meaning of messages is accomplished through integration of linguistic cues (e.g., what the speaker says), nonlinguistic cues (e.g., the speaker's facial expressions), and situational cues (e.g., predicting how the speaker is likely to feel about the particular topic). This article explores children's comprehension of emotion as expressed through facial expressions and situational cues. First, development of emotion comprehension in children with normal development is summarized. This is followed by a brief review of studies investigating emotion comprehension in clinical populations. Suggestions for assessment and intervention of children's emotion comprehension skills are presented.

The perceptual foundations of categorization in infancy

In bilingual communities, individuals often communicate in one of their languages only, and they adjust to the linguistic background of different interlocutors with ease. What facilitates such efficiency? We investigated whether bilinguals’ language activation is supported by non-linguistic cues (e.g., interlocutor identity). First, in an audio–visual task, early (proficient) and late (less proficient) Basque–Spanish bilinguals were familiarized with six novel interlocutors who spoke either Spanish, Basque, or both languages. Then, participants completed an audio–visual lexical decision task, in which the interlocutors produced test items in Spanish or Basque. Early, but not late, bilinguals’ speed of processing decreased when the language that the interlocutors spoke during familiarization matched the language they spoke at test, relative to test trials when the interlocutors changed languages. Overall, results suggest that proficient and/or early bilinguals benefit from an association between language and interlocutor during (or even before) language comprehension, because they are able to predict the context-appropriate language based on non-linguistic cues, such as interlocutor context.

Language Acquisition in Two Modalities: The Role of Nonlinguistic Cues in Linguistic Mastery

Acquisition of both American Sign Language (ASL) and spoken English by a hearing child of deaf parents was studied to determine whether two unique properties of the signed language would facilitate acquisition of grammar. Neither the transparency of reference in person deictics (signed personal pronouns) nor the isomorphism of signs with communicative gesture found in the nonmanual ASL marker “neg” facilitated acquisition, suggesting that children’s acquisition of grammar is relatively unaffected by the kinds of nonlinguistic cues studied here.

Comments on Patricia Carrell's "Cohesion Is Not Coherence". A Reader Reacts

Carrell (1982) questions Halliday and Hasan's cohesion theory (1976), calling it incomplete and warning of dangers if it were to be applied to second language teaching. Basing her views on schema theory, she sees coherence as a sort of implicit bond between the writer/reader's pre-knowledge and the text itself. Non-linguistic cues, as well as linguistic ones, exist in the memory of the reader/writer and determine subsequent coherence in a text, whether produced or processed. For her, coherence and cohesive elements in a text are effects, natural ones, arising out of the innate grammaticality of the writer/reader. Conversely, she says that Halliday and Hasan ignore these kinds of notions and envision a causal relationship between cohesion and coherence, perhaps even viewing the terms as synonymous. As a result, teachers may teach cohesive ties as predictors of coherence.