Speaker Gaze Research Articles

Speaker gaze increases information coupling between infant and adult brains

When infants and adults communicate, they exchange social signals of availability and communicative intention such as eye gaze. Previous research indicates that when communication is successful, close temporal dependencies arise between adult speakers' and listeners' neural activity. However, it is not known whether similar neural contingencies exist within adult-infant dyads. Here, we used dual-electroencephalography to assess whether direct gaze increases neural coupling between adults and infants during screen-based and live interactions. In experiment 1 (n = 17), infants viewed videos of an adult who was singing nursery rhymes with (i) direct gaze (looking forward), (ii) indirect gaze (head and eyes averted by 20°), or (iii) direct-oblique gaze (head averted but eyes orientated forward). In experiment 2 (n = 19), infants viewed the same adult in a live context, singing with direct or indirect gaze. Gaze-related changes in adult-infant neural network connectivity were measured using partial directed coherence. Across both experiments, the adult had a significant (Granger) causal influence on infants' neural activity, which was stronger during direct and direct-oblique gaze relative to indirect gaze. During live interactions, infants also influenced the adult more during direct than indirect gaze. Further, infants vocalized more frequently during live direct gaze, and individual infants who vocalized longer also elicited stronger synchronization from the adult. These results demonstrate that direct gaze strengthens bidirectional adult-infant neural connectivity during communication. Thus, ostensive social signals could act to bring brains into mutual temporal alignment, creating a joint-networked state that is structured to facilitate information transfer during early communication and learning.

Direct Speaker Gaze Promotes Trust in Truth-Ambiguous Statements.

A speaker’s gaze behaviour can provide perceivers with a multitude of cues which are relevant for communication, thus constituting an important non-verbal interaction channel. The present study investigated whether direct eye gaze of a speaker affects the likelihood of listeners believing truth-ambiguous statements. Participants were presented with videos in which a speaker produced such statements with either direct or averted gaze. The statements were selected through a rating study to ensure that participants were unlikely to know a-priori whether they were true or not (e.g., “sniffer dogs cannot smell the difference between identical twins”). Participants indicated in a forced-choice task whether or not they believed each statement. We found that participants were more likely to believe statements by a speaker looking at them directly, compared to a speaker with averted gaze. Moreover, when participants disagreed with a statement, they were slower to do so when the statement was uttered with direct (compared to averted) gaze, suggesting that the process of rejecting a statement as untrue may be inhibited when that statement is accompanied by direct gaze.

Editorial: Understanding Social Signals: How Do We Recognize the Intentions of Others?

Editorial: Understanding Social Signals: How Do We Recognize the Intentions of Others?

What you see is what you do: on the relationship between gaze and gesture in multimodal alignment

abstractInteractive language use inherently involves a process of coordination, which often leads to matching behaviour between interlocutors in different semiotic channels. We study this process of interactive alignment from a multimodal perspective: using data from head-mounted eye-trackers in a corpus of face-to-face conversations, we measure which effect gaze fixations by speakers (on their own gestures, condition 1) and fixations by interlocutors (on the gestures by those speakers, condition 2) have on subsequent gesture production by those interlocutors. The results show there is a significant effect of interlocutor gaze (condition 2), but not of speaker gaze (condition 1) on the amount of gestural alignment, with an interaction between the conditions.

Effect of speaker gaze on word learning in fragile X syndrome: a comparison with nonsyndromic autism spectrum disorder.

This study examined use of a speaker's direction of gaze during word learning by boys with fragile X syndrome (FXS), boys with nonsyndromic autism spectrum disorder (ASD), and typically developing (TD) boys. A fast-mapping task with follow-in and discrepant labeling conditions was administered. We expected that the use of speaker gaze would lead to participants selecting as the referent of the novel label the object to which they attended in follow-in trials and the object to which the examiner attended in the discrepant labeling trials. Participants were school-aged boys with FXS (n=18) or ASD (n=18) matched on age, intelligence quotient, and nonverbal cognition and younger TD boys (n=18) matched on nonverbal cognition. All groups performed above chance in both conditions, although the TD boys performed closest to the expected pattern. Boys with FXS performed better during follow-in than in discrepant label trials, whereas TD boys and boys with ASD did equally well in both trial types. The type of trial administered first influenced subsequent responding. Error patterns also distinguished the groups. The ability to utilize a speaker's gaze during word learning is not as well developed in boys with FXS or nonsyndromic ASD as in TD boys of the same developmental level.

The influence of speaker gaze on listener comprehension: Contrasting visual versus intentional accounts

Previous research has shown that listeners follow speaker gaze to mentioned objects in a shared environment to ground referring expressions, both for human and robot speakers. What is less clear is whether the benefit of speaker gaze is due to the inference of referential intentions (Staudte and Crocker, 2011) or simply the (reflexive) shifts in visual attention. That is, is gaze special in how it affects simultaneous utterance comprehension? In four eye-tracking studies we directly contrast speech-aligned speaker gaze of a virtual agent with a non-gaze visual cue (arrow). Our findings show that both cues similarly direct listeners’ attention and that listeners can benefit in utterance comprehension from both cues. Only when they are similarly precise, however, does this equality extend to incongruent cueing sequences: that is, even when the cue sequence does not match the concurrent sequence of spoken referents can listeners benefit from gaze as well as arrows. The results suggest that listeners are able to learn a counter-predictive mapping of both cues to the sequence of referents. Thus, gaze and arrows can in principle be applied with equal flexibility and efficiency during language comprehension.

Word Learning with Multiple Cues in Children with Autistic Spectrum Disorder: Evidence from Eye Movements

Several previous studies consistently reported that children with Autistic Spectrum Disorder(ASD) had impairments in referential word learning when they are provided with only social eye gaze cues. More recent studies, however, suggested that children with ASD could use a speaker's gaze direction to learn words when there were abundant additional perceptual cues enhancing the salience of the referent object. But their findings could not exclude the possibility that children in these studies just relied on perceptual cues to guide their word learning without noticing any social cues. In addition, it is also unknown that how ASD children in late childhood should respond when social cues conflict with perceptual cues. In order to examine the relative roles of social cues on word learning task compared to perceptual cues, the study recorded the eye movements of 18 verbally able children with ASD(M ages = 134.83 months) and their typical developing peers matched on receptive language and non-verbal intelligence(M ages = 71.61 months) while these two groups were completing word learning tasks under the following conditions: when gaze cues were no provided(No Gaze condition), when gaze cues were consistent with perceptual cues(Consistent Gaze condition), and when gaze cues were inconsistent with perceptual cues(Inconsistent Gaze condition). We found that, like typically developing(TD) children, children with ASD would map the novel word to the interesting object in the No Gaze condition and in the Consistent Gaze condition and the performance in the Consistent Gaze condition was better than that in the No Gaze condition. We also found that the ASD children would overlook perceptual cues to follow the speaker's attention focus to pick up the boring object in the Inconsistent Gaze condition. Eye movement data results showed that: 1) the correct rate of first gaze following(the first saccades from face to target objects/ the first saccades from face to target and non-target objects) and the frequency of gaze following(saccades from face to target objects) in ASD children were lower than TD children; 2) the proportion of duration on eye region(fixation to eye region/ fixation to face region) in ASD children was higher than TD children; 3)there were no differences between two groups in the proportion of fixation to interesting object(fixation to interesting objects/ fixation to interesting and boring objects) and the proportion of gaze following(saccades from face to target objects/ saccades from face to target and non-target objects). The results illustrated that although social cues facilitated word learning in autism in overlapping cues condition and children with ASD also weighted social cues more heavily than perceptual cues, the style of capturing social cues in ASD children were different from TD children. They could not flexibly distribute attention resource according to the change in social scene, captured social information with the analytic way of feature process, and lacked the sensitivity of social information and the understanding of social meaning of social information.

Preparing to be punched: Prediction may not always require inference of intentions

Pickering & Garrod's (P&G's) framework assumes an efference copy based on the interlocutor's intentions. Yet, elaborate attribution of intentions may not always be necessary for online prediction. Instead, contextual cues such as speaker gaze can provide similar information with a lower demand on processing resources.

Formulation in clinical interviews

Formulation, according to Garfinkel and Sacks (1986 [1969]), refers to speakers' ways of talking about the current interaction. This article explores how formulations are used in clinical assessment interviews as a way of providing evidence of the respondent's capacities that are currently assessed. The videotaped data are drawn from the clinical interviews of unemployed adults and older persons. The data are analyzed using conversation analysis (CA). The article shows how formulation is achieved through vocalized and/or embodied actions (e.g., nodding index finger pointing), in conjunction with the speaker's gaze directed to the recipient. It argues that these formulations enable access to the shared epistemic domain of the current interaction. This domain is an interactional achievement and, as a resource, it allows the participants to designate viewpoints regarding the respondent's social competence. In sum, the paper demonstrates how participants can show an explicit orientation to the interactional substrate of interview.

Brief Report: Pointing Cues Facilitate Word Learning in Children with Autism Spectrum Disorder

Children with autism spectrum disorder (ASD) reportedly have difficulty associating novel words to an object via the speaker's gaze. It has also been suggested that their performance is related to their gaze duration on the object and improves when the object moves and becomes more salient. However, there is a possibility that they have only relied on the object's movement and have not referenced the speaker's cue (i.e. gaze direction). The current study with children with ASD and typically developing children aged 6-11years demonstrated that adding another speaker's cue (i.e. pointing) improves the performance of children with ASD. This suggests that additional speaker's cues may help referential word learning in children with ASD.

Can Speaker Gaze Modulate Syntactic Structuring and Thematic Role Assignment during Spoken Sentence Comprehension?

During comprehension, a listener can rapidly follow a frontally seated speaker’s gaze to an object before its mention, a behavior which can shorten latencies in speeded sentence verification. However, the robustness of gaze-following, its interaction with core comprehension processes such as syntactic structuring, and the persistence of its effects are unclear. In two “visual-world” eye-tracking experiments participants watched a video of a speaker, seated at an angle, describing transitive (non-depicted) actions between two of three Second Life characters on a computer screen. Sentences were in German and had either subjectNP1-verb-objectNP2 or objectNP1-verb-subjectNP2 structure; the speaker either shifted gaze to the NP2 character or was obscured. Several seconds later, participants verified either the sentence referents or their role relations. When participants had seen the speaker’s gaze shift, they anticipated the NP2 character before its mention and earlier than when the speaker was obscured. This effect was more pronounced for SVO than OVS sentences in both tasks. Interactions of speaker gaze and sentence structure were more pervasive in role-relations verification: participants verified the role relations faster for SVO than OVS sentences, and faster when they had seen the speaker shift gaze than when the speaker was obscured. When sentence and template role-relations matched, gaze-following even eliminated the SVO-OVS response-time differences. Thus, gaze-following is robust even when the speaker is seated at an angle to the listener; it varies depending on the syntactic structure and thematic role relations conveyed by a sentence; and its effects can extend to delayed post-sentence comprehension processes. These results suggest that speaker gaze effects contribute pervasively to visual attention and comprehension processes and should thus be accommodated by accounts of situated language comprehension.

Questioning and responding in Italian

Questions are design problems for both the questioner and the addressee. They must be produced as recognizable objects and must be comprehended by taking into account the context in which they occur and the local situated interests of the participants. This paper investigates how people do ‘questioning’ and ‘responding’ in Italian ordinary conversations. I focus on the features of both questions and responses. I first discuss formal linguistic features that are peculiar to questions in terms of intonation contours ( e.g. final rise), morphology ( e.g. tags and question words) and syntax ( e.g. inversion). I then show additional features that characterize their actual implementation in conversation such as their minimality (often the subject or the verb is only implied) and the usual occurrence of speaker gaze towards the recipient during questions. I then look at which social actions ( e.g. requests for information, requests for confirmation) the different question types implement and which responses are regularly produced in return. The data shows that previous descriptions of “interrogative markings” are neither adequate nor sufficient to comprehend the actual use of questions in natural conversation.

Use of Speaker's Gaze and Syntax in Verb Learning

Speaker eye gaze and gesture are known to help child and adult listeners establish communicative alignment and learn object labels. Here we consider how learners use these cues, along with linguistic information, to acquire abstract relational verbs. Test items were perspective verb pairs (e.g., chase/flee, win/lose), which pose a special problem for observational accounts of word learning because their situational contexts overlap very closely; the learner must infer the speaker's chosen perspective on the event. Two cues to the speaker's perspective on a depicted event were compared and combined: (a) the speaker's eye gaze to an event participant (e.g., looking at the Chaser vs. looking at the Flee-er) and (b) the speaker's linguistic choice of which event participant occupies Subject position in his utterance. Participants (3-, 4-, and 5-year-olds) were eye-tracked as they watched a series of videos of a man describing drawings of perspective events (e.g., a rabbit chasing an elephant). The speaker looked at one of the two characters and then uttered either an utterance that was referentially uninformative (He's mooping him) or informative (The rabbit's mooping the elephant/The elephant's mooping the rabbit) because of the syntactic positioning of the nouns. Eye-tracking results showed that all participants regardless of age followed the speaker's gaze in both uninformative and informative contexts. However, verb-meaning choices were responsive to speaker's gaze direction only in the linguistically uninformative condition. In the presence of a linguistically informative context, effects of speaker gaze on meaning were minimal for the youngest children to nonexistent for the older populations. Thus children, like adults, can use multiple cues to inform verb-meaning choice but rapidly learn that the syntactic positioning of referring expressions is an especially informative source of evidence for these decisions.

Preschoolers' Perspective Taking in Word Learning

When learning new words, do children use a speaker's eye gaze because it reveals referential intent? We conducted two experiments that addressed this question. In Experiment 1, the experimenter left while two novel objects were placed where the child could see both, but the experimenter would be able to see only one. The experimenter returned, looked directly at the mutually visible object, and said either, "There's the [novel word]!" or "Where's the [novel word]?" Two- through 4-year-olds selected the target of the speaker's gaze more often on there trials than on where trials, although only the older children identified the referent correctly at above-chance levels on trials of both types. In Experiment 2, the experimenter placed a novel object where only the child could see it and left while the second object was similarly hidden. When she returned and asked, "Where's the [novel word]?" 2- through 4-year-olds chose the second object at above-chance levels. Preschoolers do not blindly follow gaze, but consider the linguistic and pragmatic context when learning a new word.

Coordinating Gesture, Talk, and Gaze in Reenactments

In this article, I examine the coordination of talk, gaze, and gesture in the production of reenactments in conversation. Reenactments involve re-presentations or depictions and are thus distinct from tellings, which are primarily descriptive. A basic question I address concerns how recipients are able to parse a larger telling into those parts of it that narrate or tell about the events being described and those that reenact them. Analysis of several instances suggests that speaker gaze plays a crucial role in this respect. I discuss the relation between reenactments and direct quotation (Holt, 2000) and demonstration (Clark & Gerrig, 1990) as well as the significance of the analysis for current understanding of multimodality in interaction.

Listener Responses as a Collaborative Process: The Role of Gaze

The authors examined precisely when and how listeners insert their responses into a speaker's narrative. A collaborative theory would predict a relationship between the speaker's acts and the listener's responses, and the authors proposed that speaker gaze coordinated this collaboration. The listener typically looks more at the speaker than the reverse, but at key points while speaking the speaker seeks a response by looking at the listener, creating a brief period of mutual gaze called here a gaze window. The listener was very likely to respond with “mhm,” a nod, or other reaction during this period, after which the speaker quickly looked away and continued speaking. This model was tested with 9 dyads in which 1 person was telling a close-call story to the other. The results confirmed the model for each dyad, demonstrating both collaboration in dialogue at the microlevel and a high degree of integration and coordination of audible and visible acts, in this case, speech and gaze.

Do Children with Autism Use the Speaker's Direction of Gaze Strategy to Crack the Code of Language?

Normal toddlers infer the referent of a novel word by consulting the speaker's direction of gaze. That is, they use the Speaker's Direction of Gaze (SDG) strategy. This is a far more powerful strategy than the alternative, the Listener's Direction of Gaze (LDG) strategy. In Study 1 we tested if children with autism, who have well-documented impairments in joint attention, used the SDG or the LDG strategy to learn a novel word for a novel object. Results showed that although 70.6% of children with mental handicap passed the test by making the correct mapping between a novel word and a novel object, via the SDG strategy, only 29.4% of children with autism did so. Instead, their reliance on the LDG strategy led to mapping errors. In Study 2 a group of normal children, whose chronological age (24 months old) was equated with the verbal mental age of the 2 clinical groups in Study 1, was tested using a similar procedure. Results showed that 79% of this normal group passed the test by making the correct mapping between a novel word and a novel object using the SDG strategy. Taken together, the results from both studies suggest that children with autism are relatively insensitive to a speaker's gaze direction as an index of the speaker's intention to refer. This result is consistent with previous findings showing that children with autism are relatively "blind" to the mentalistic significance of the eyes. Discussion centers on how the absence of an SDG strategy might disrupt specific aspects of language development in autism.

Ostensive naming in face-to-face interaction: The role of speaker gaze

Ostensive naming in face-to-face interaction: The role of speaker gaze

Social Anxiety and Speaker Gaze in a Persuasive Atmosphere

Twenty females scoring low and 20 females scoring high on a pretested measure of social anxiety participated in an experiment to investigate the role of social anxiety in a persuasive situation. Results supported the hypothesis that social anxiety mediates the tendency for subjects to direct their gazes at an agreeing or disagreeing confederate when expressing their views on a controversial topic. As expected, socially anxious Ss spent less time gazing at the other participants than did socially secure Ss. Moreover, socially anxious Ss spent significantly less time gazing toward a disagreeing confederate than did socially secure subjects. On the other hand, no significant differences were found between socially anxious and secure Ss′ gazes at an agreeing confederate. These results are interpreted from a social control perspective.

Planning units in spontaneous speech: some evidence from hesitation in speech and speaker gaze direction in conversation

Planning units in spontaneous speech: some evidence from hesitation in speech and speaker gaze direction in conversation