Referential Gaze Research Articles

The effects of referential gaze in spoken language comprehension: Human speaker vs. virtual agent listener gaze

IntroductionFour studies addressed effects of human speaker gaze vs. virtual agent listener gaze on eye movements during spoken sentence comprehension.MethodParticipants saw videos in which a static scene depicting three characters was presented on a screen. Eye movements were recorded as participants listened to German subject-verb-object (SVO) sentences describing an interaction between two of these characters. Participants' task was to verify whether the sentence matched a schematic depiction of the event. Two critical factors were manipulated across all four experiments: (1) whether the human speaker—uttering the sentence—was visible, and (2) whether the agent listener was present. Moreover, in Experiments 2 and 4, the target second noun phrase (NP2) was made inaudible, and in Experiments 3 and 4, the gaze time course of the agent listener was altered: it looked at the NP2 referent about 400 ms before the speaker did. These manipulations served to increase the value of the speaker's and listener's gaze cues for correctly anticipating the NP2 referent.ResultsHuman speaker gaze led to increased fixations of the NP2 referent in all experiments, but primarily after the onset of its mention. Only in Experiment 3 did participants reliably anticipate the NP2 referent, in this case making use of both the human speaker's and the virtual agent listener's gaze. In all other cases, virtual agent listener gaze had no effect on visual anticipation of the NP2 referent, even when it was the exclusive cue.DiscussionSuch information on the use of gaze cues can refine theoretical models of situated language processing and help to develop virtual agents that act as competent communication partners in conversations with human interlocutors.

Alpha and theta brain activity in 9-month-old infants during a live referential gaze paradigm.

The ability to establish a connection between the direction of the other's gaze and the object that is observed has important implications in the development of social cognition and learning. In this study, we analyzed alpha and theta band oscillations in one group of 9-month-old infants by implementing a face-to-face live paradigm, which presented the infants with a triadic social interaction with a real human being. We compared neural activations in two experimental conditions: Congruent and Incongruent gaze shift following the appearance of an object. In the Incongruent object-gaze shift condition, we observed an increase of the theta power in comparison with the Congruent condition. We also found an enhancement of the alpha activity during the Congruent versus the Incongruent object-gaze condition. These findings confirm the involvement of the theta and alpha band activity in the detection of the gaze of others when it shifts toward a referential target. We consider that the theta band modulation could be associated with the processing of unexpected events. Furthermore, the increase of the alpha band activity during the Congruent object-gaze condition seems to be in agreement with prior findings on the mechanisms of internally controlled attention that emerge before the first year of life. The implementation of a live paradigm elicited a partially different oscillatory pattern in comparison with non-live standard paradigms, supporting the importance of an ecological set-up reproducing real-life conditions to study the development of social cognition.

Age-Related Differences in the Perception of Robotic Referential Gaze in Human-Robot Interaction

There is an increased interest in using social robots to assist older adults during their daily life activities. As social robots are designed to interact with older users, it becomes relevant to study these interactions under the lens of social cognition. Gaze following, the social ability to infer where other people are looking at, deteriorates with older age. Therefore, the referential gaze from robots might not be an effective social cue to indicate spatial locations to older users. In this study, we explored the performance of older adults, middle-aged adults, and younger controls in a task assisted by the referential gaze of a Pepper robot. We examined age-related differences in task performance, and in self-reported social perception of the robot. Our main findings show that referential gaze from a robot benefited task performance, although the magnitude of this facilitation was lower for older participants. Moreover, perceived anthropomorphism of the robot varied less as a result of its referential gaze in older adults. This research supports that social robots, even if limited in their gazing capabilities, can be effectively perceived as social entities. Additionally, this research suggests that robotic social cues, usually validated with young participants, might be less optimal signs for older adults.

Temperamental patterns and infant joint attention in typical development.

Joint attention refers to the ability to share attention between two individuals and a third object. These skills include referential gaze and gestures such as giving and showing, and pointing. We distinguish the RJA (Response to Joint Attention) when a behaviour is initiated by the parent and the child follows an indicated object with his eye gaze, and the IJA (Initiative to Joint Attention) when the child begins a sharing behaviour aimed at satisfaction of his needs. Some researchers have highlighted how temperament plays a role in the development of JA, in particular the style of temperament and emotional reactivity. In this study, we investigate infant’s temperamental elements at 18-24 month and how it could relate to IJA and RJA performance. Study participants were recruited among 32 nurseries, for a total of 248 children. In order to assess the temperamental factors and joint attention capabilities, nest operators were required to perform two questionnaires, respectively to evaluate the temperament and JA.

Children's Social Information Seeking is Sensitive to Referential Ambiguity.

We examined children's spontaneous information seeking in response to referential ambiguity. Children ages 2-5 (n=160) identified the referents of familiar and novel labels. We manipulated ambiguity by changing the number of objects present and their familiarity (Experiments 1 and 2), and the availability of referential gaze (Experiment 2). In both experiments, children looked to the face of the experimenter more often while responding, specifically when the referent was ambiguous. In Experiment 2, 3- to 4-year olds also demonstrated sensitivity to graded referential evidence. These results suggest that social information seeking is an active learning behavior that could contribute to language acquisition in early childhood.

The understanding of congruent and incongruent referential gaze in 17-month-old infants: an eye-tracking study comparing human and robot

Several studies have shown that the human gaze, but not the robot gaze, has significant effects on infant social cognition and facilitate social engagement. The present study investigates early understanding of the referential nature of gaze by comparing—through the eye-tracking technique—infants’ response to human and robot’s gaze. Data were acquired on thirty-two 17-month-old infants, watching four video clips, where either a human or a humanoid robot performed an action on a target. The agent’s gaze was either turned to the target (congruent) or opposite to it (incongruent). The results generally showed that, independent of the agent, the infants attended longer at the face area compared to the hand and target. Additionally, the effect of referential gaze on infants’ attention to the target was greater when infants watched the human compared to the robot’s action. These results suggest the presence, in infants, of two distinct levels of gaze-following mechanisms: one recognizing the other as a potential interactive partner, the second recognizing partner's agency. In this study, infants recognized the robot as a potential interactive partner, whereas ascribed agency more readily to the human, thus suggesting that the process of generalizability of gazing behaviour to non-humans is not immediate.

Aporias e apostas do representável: vazios e vestígios da memória em Os dias com ele (Maria Clara Escobar, 2013)

O irrepresentável é uma matriz teórica recorrente nas análises do documentário subjetivo contemporâneo. Oriunda dos estudos do Holocausto, esta abordagem contém o trauma pessoal e histórico em seu núcleo, fundamentando várias considerações sobre o cinema documental realizado por descendentes de militantes vítimas das ditaduras do Cone Sul. O objetivo aqui é promover uma discussão acerca do representável a partir da análise de Os dias com ele (Maria Clara Escobar, 2013), documentário no qual a filha enfrenta as resistências do pai em relatar as torturas sofridas durante a ditadura brasileira (1964-1985). Os impasses da memória se manifestam com força nesse filme, em diálogos tensos e enquadramentos de um vazio eloquente. Contudo, junto com o reconhecimento das aporias da representação, é possível fazer outras apostas interpretativas. Se os vazios são plenos de significados, os vestígios do passado inscritos nas imagens também merecem atenção. A hipótese é que, no lugar do foco exclusivo nas aporias do irrepresentável, uma interpretação historiográfica dessa obra deve levar em conta, dialeticamente, os vazios e os vestígios da memória. Reivindica-se, assim, uma visada referencial que é não só epistemologicamente possível, mas politicamente urgente face à ofensiva negacionista que ganha cada vez mais espaço no Brasil pós-2018.

Eye'll Help You Out! How the Gaze Cue Reduces the Cognitive Load Required for Reference Processing.

Referential gaze has been shown to benefit language processing in situated communication in terms of shifting visual attention and leading to shorter reaction times on subsequent tasks. The present study simultaneously assessed both visual attention and, importantly, the immediate cognitive load induced at different stages of sentence processing. We aimed to examine the dynamics of combining visual and linguistic information in creating anticipation for a specific object and the effect this has on language processing. We report evidence from three visual‐world eye‐tracking experiments, showing that referential gaze leads to a shift in visual attention toward the cued object, which consequently lowers the effort required for processing the linguistic reference. Importantly, perceiving and following the gaze cue did not prove costly in terms of cognitive effort, unless the cued object did not fit the verb selectional preferences.

A network analytic approach to gaze coordination during a collaborative task

A critical component of collaborative learning is the establishment of intersubjectivity, or the construction of mutual understanding. Collaborators coordinate their understanding with one another across various modes of communication, including speech, gesture, posture, and gaze. Given the dynamic, interdependent, and complex nature of coordination, this study sought to develop and test a method for constructing detailed and nuanced models of coordinated referential gaze patterns. In the study, 13 dyads participated in a simple collaborative task. We used dual mobile eye tracking to record each participant's gaze behavior, and we used epistemic network analysis (ENA) to model the gazes of both conversational participants synchronously. In the model, the nodes in the network represent gaze targets for each participant, and the connections between nodes indicate the likelihood of gaze coordination. Our analyses indicate: (a) properties and patterns of how gaze coordination unfolds throughout an interaction sequence; and (b) differences in gaze coordination patterns for interaction sequences that lead to breakdowns and repairs. In addition to contributing to the growing body of knowledge on the coordination of gaze behaviors in collaborative activities, this work suggests that ENA enables more effective modeling of gaze coordination.

Object permanence and the development of attention capacity in preterm and term infants: an eye-tracking study

The relationship between premature birth and early cognitive function as measured by eye-tracking data remains unexplored. The aim of this study was to determine the effect of prematurity on the development of object permanence and attention capacity using eye-tracking measures. We prospectively studied very low birth weight (VLBW < 1500 g) preterm infants who were admitted to the Neonatal Intensive Care Unit of Hanyang University College of Medicine, Seoul, Korea and visited a follow-up clinic. Using eye-tracking measures, object permanence was assessed in 15 VLBW preterm and 10 term infants at a corrected age of 6–10 months, and attention capacity was measured in 26 VLBW preterm and 18 term children who were age-matched for the corrected age of 6–10 or18 months. No differences were found in chronologic age (corrected age for prematurity), sex, or maternal education between the study groups. The VLBW preterm infants had lower scores than term infants on eye-tracking measures of object permanence than the term infants did at 6–10 months (P = 0.042). The VLBW preterm infants had a shorter referential gaze than the term infants did at 6–10 months (P = 0.038); moreover, the length of referential gaze of the VLBW preterm infants was significantly lower at 6–10 months than at 18 months (P = 0.047), possibly indicating a delayed trajectory of attention development. The VLBW preterm infants have different attention capacities and object permanence developmental markers than term infants at the corrected age of 6–10 months.

Social Attention, Joint Attention and Sustained Attention in Autism Spectrum Disorder and Williams Syndrome: Convergences and Divergences.

There is limited knowledge on shared and syndrome-specific attentional profiles in autism spectrum disorder (ASD) and Williams syndrome (WS). Using eye-tracking, we examined attentional profiles of 35 preschoolers with ASD, 22 preschoolers with WS and 20 typically developing children across social and non-social dimensions of attention. Children with ASD and those with WS presented with overlapping deficits in spontaneous visual engagement with the target of others' attention and in sustained attention. Children with ASD showed syndrome-specific abnormalities in monitoring and following a person's referential gaze, as well as a lack of preferential attention to social stimuli. Children with ASD and WS present with shared as well as syndrome-specific abnormalities across social and non-social dimensions of attention.

Look together: analyzing gaze coordination with epistemic network analysis.

When conversing and collaborating in everyday situations, people naturally and interactively align their behaviors with each other across various communication channels, including speech, gesture, posture, and gaze. Having access to a partner's referential gaze behavior has been shown to be particularly important in achieving collaborative outcomes, but the process in which people's gaze behaviors unfold over the course of an interaction and become tightly coordinated is not well understood. In this paper, we present work to develop a deeper and more nuanced understanding of coordinated referential gaze in collaborating dyads. We recruited 13 dyads to participate in a collaborative sandwich-making task and used dual mobile eye tracking to synchronously record each participant's gaze behavior. We used a relatively new analysis technique—epistemic network analysis—to jointly model the gaze behaviors of both conversational participants. In this analysis, network nodes represent gaze targets for each participant, and edge strengths convey the likelihood of simultaneous gaze to the connected target nodes during a given time-slice. We divided collaborative task sequences into discrete phases to examine how the networks of shared gaze evolved over longer time windows. We conducted three separate analyses of the data to reveal (1) properties and patterns of how gaze coordination unfolds throughout an interaction sequence, (2) optimal time lags of gaze alignment within a dyad at different phases of the interaction, and (3) differences in gaze coordination patterns for interaction sequences that lead to breakdowns and repairs. In addition to contributing to the growing body of knowledge on the coordination of gaze behaviors in joint activities, this work has implications for the design of future technologies that engage in situated interactions with human users.

Referential gaze and word learning in adults with autism.

While typically developing children can use referential gaze to guide their word learning, those with autism spectrum disorder are often described to have problems with that. However, some researchers assume that the ability to follow gaze to select the correct referent can develop in autism later compared to typically developing individuals. To test this assumption, we compared the performance of adults with and without autism on a word learning task while recording their gaze behavior using an eye tracker. Results showed that both groups mostly chose the correct referent, but less so for the autism spectrum disorder group when the distractor's saliency was increased, suggesting that the ability to learn novel words by referring to gaze develops in autism spectrum disorder, but not fully, relative to their typically developing peers.

Langage adressé au bébé et exploration visuelle chez le bébé de 4,5 mois : mise en évidence d’un effet de genre

Background & aimInfant directed speech (IDS) is a simplified form of language known to capture infant visual attention. IDS is an early form of communication to which infants are particularly sensitive beginning at birth and play an essential part in the acquisition of language and in the development of social cognition. The present research aims at determining whether an infant as young as 4.5months explores different face zones of a person talking to her when the form of speech employed is either IDS or adult directed speech (ADS). We wish to bring more insight in the relation between the way an adult talks to an infant and the infant's visual attention, such a relation being a central tenet to understanding early interactions. MethodsEighteen 4.5month-old infants were presented videos of a woman talking either IDS or ADS, each video played with and without soundtrack. The total time focused respectively on eye areas and mouth area was collected with an eye-tracking system (Tobii 2150). Results and conclusionResults showed evidence of more visual exploration when the soundtrack was played as opposed to silence while exploration focused more on the eye area. Moreover, interaction effects were observed involving a gender effect. These results raise the question of the importance of infant's gaze and attention directed to the face, and specifically to the eyes of the partner. Indeed, eyes of a talking face seem of great importance at 4.5months of age, i.e., long before an infant is known to be able to interpret a referential gaze. Our results also add to the gender effect differences found in early interactions between infant and social partners. A better understanding of what infant attention focuses on in a communication-oriented situation, including infant gender impact, would not only help to detect early parent-infant communication distortions but add to a more adapted and efficient clinical follow-up.

Infants understand the referential nature of human gaze but not robot gaze

Infants can acquire much information by following the gaze direction of others. This type of social learning is underpinned by the ability to understand the relationship between gaze direction and a referent object (i.e., the referential nature of gaze). However, it is unknown whether human gaze is a privileged cue for information that infants use. Comparing human gaze with nonhuman (robot) gaze, we investigated whether infants’ understanding of the referential nature of looking is restricted to human gaze. In the current study, we developed a novel task that measured by eye-tracking infants’ anticipation of an object from observing an agent’s gaze shift. Results revealed that although 10- and 12-month-olds followed the gaze direction of both a human and a robot, only 12-month-olds predicted the appearance of objects from referential gaze information when the agent was the human. Such a prediction for objects reflects an understanding of referential gaze. Our study demonstrates that by 12months of age, infants hold referential expectations specifically from the gaze shift of humans. These specific expectations from human gaze may enable infants to acquire various information that others convey in social learning and social interaction.

Looking back at ‘looking back’: operationalising referential gaze for dingoes in an unsolvable task

This paper examined the performance of dingoes (Canis dingo) on the rope-pulling task, previously used by Miklósi et al. (Curr Biol 13:763-766, 2003) to highlight a key distinction in the problem-solving behaviour of wolves compared to dogs when in the company of humans. That is, when dogs were confronted with an unsolvable task, following a solvable version of the task they looked back or gazed at the human, whereas, wolves did not. We replicated the rope-pulling task using 12 sanctuary-housed dingoes and used the Miklósi et al. (Curr Biol 13:763-766, 2003) definition of looking back behaviour to analyse the data. However, at least three different types of look backs were observed in our study. We, then developed a more accurate operational definition of looking back behaviour that was task specific and reanalysed the data. We found that the operational definition employed greatly influences the results, with vague definitions potentially overestimating the prevalence of looking back behaviour. Thus, caution must be taken when interpreting the results of studies utilising looking back as behaviour linked to assistance seeking during problem solving. We present a more stringent definition and make suggestions for future research.

Static gaze direction detection in children with autism: a developmental perspective

Background and objectives Children gain a considerable amount of information about their environment from the direction of the gaze: the eyes often point to the object of someone’s attention and help children to identify a shared topic during communication and social learning (Csibra & Gergely, 2006). While the developmental pattern of referential gaze processing has been well investigated in typically developing children, far less is known about gaze processing in respect to children with autism. Some observational studies suggest that accurate gaze direction judgment is problematic for individuals with Autism (Gepner et al., 1996; Webster & Potter, 2008; Goldberg et al., 2008; Riby, Doherty-Sneddon, & Bruce, 2008). However, other studies show that gaze cueing appears to be relatively intact in ASDs (Chawarska et al., 2003; Ristic et al., 2005; Sweetenham et al., 2003). Very recently, Freeth, Ropar, Chapman, Mitchell (2010) and Freeth, Chapman, Ropar, & Mitchell (2010) found that high functioning adolescents with ASDs selectively attended to people’s faces like TDs controls and spontaneously followed gaze direction. These very recent results lead to the question of whether gaze direction detection is impaired early in ASDs and becomes regular functioning later in adolescence or early adulthood. This would suggest whether early gaze following deficits might be not only the result of a specific deficit but also the consequence of a developmental delay.

Investigating joint attention mechanisms through spoken human–robot interaction

Referential gaze during situated language production and comprehension is tightly coupled with the unfolding speech stream ( Griffin, 2001; Meyer, Sleiderink, & Levelt, 1998; Tanenhaus, Spivey-Knowlton, Eberhard, & Sedivy, 1995). In a shared environment, utterance comprehension may further be facilitated when the listener can exploit the speaker’s focus of (visual) attention to anticipate, ground, and disambiguate spoken references. To investigate the dynamics of such gaze-following and its influence on utterance comprehension in a controlled manner, we use a human–robot interaction setting. Specifically, we hypothesize that referential gaze is interpreted as a cue to the speaker’s referential intentions which facilitates or disrupts reference resolution. Moreover, the use of a dynamic and yet extremely controlled gaze cue enables us to shed light on the simultaneous and incremental integration of the unfolding speech and gaze movement. We report evidence from two eye-tracking experiments in which participants saw videos of a robot looking at and describing objects in a scene. The results reveal a quantified benefit-disruption spectrum of gaze on utterance comprehension and, further, show that gaze is used, even during the initial movement phase, to restrict the spatial domain of potential referents. These findings more broadly suggest that people treat artificial agents similar to human agents and, thus, validate such a setting for further explorations of joint attention mechanisms.

Influence of Eye Gaze on Spoken Word Processing: An ERP Study With Infants

Eye gaze is an important communicative signal, both as mutual eye contact and as referential gaze to objects. To examine whether attention to speech versus nonspeech stimuli in 4- to 5-month-olds (n=15) varies as a function of eye gaze, event-related brain potentials were used. Faces with mutual or averted gaze were presented in combination with forward- or backward-spoken words. Infants rapidly processed gaze and spoken words in combination. A late Slow Wave suggests an interaction of the 2 factors, separating backward-spoken word+direct gaze from all other conditions. An additional experiment (n=15) extended the results to referential gaze. The current findings suggest that interactions between visual and auditory cues are present early in infancy.

Do children with ASD use referential gaze to learn the name of an object? An eye-tracking study

Children with autism spectrum disorder (ASD) are reported to have difficulty in learning novel word–object associations in case of discrepancy between objects in the speaker's focus and their focus (the discrepant condition). Two eye-tracking experiments investigated this difficulty by controlling and recording children's gaze fixation. In Experiment 1's discrepant condition, typically developing (TD) children (age: 6–11 years) mapped the novel word to the novel object in the speaker's focus more frequently than children with ASD (age: 6–11 years). Additionally, the former looked at the object in the speaker's focus longer than the object in their own focus, while the latter looked at these objects for the same duration. In Experiment 2, the saliency of the object in the speaker's focus was enhanced and children with ASD (age: 6–12 years) mapped the word to the object in the speaker's focus as well as TD children (age: 6–12 years). Moreover, they looked at the object in the speaker's focus longer than the object in their focus. Comparison between experiments also confirmed improvement in performance, suggesting that the duration of gaze at an object in the speaker's focus is related to difficulty in referential word learning in children with ASD.