Eye Gaze Information Research Articles

Face perception and social cognitive development in early autism: A prospective longitudinal study from 3 months to 7 years of age.

Autism spectrum condition (ASC) is characterized by atypical attention to eyes and faces, but the onset and impact of these atypicalities remain unclear. This prospective longitudinal study examined face perception in infants who develop ASC (N = 22, female = 5, 100% White) compared with typically developing infants (N = 131, female = 65, 55.6% White), tracking social-cognitive and ASC development through age seven. Reduced interest in direct gaze and eyes during infancy correlated with atypical development of adaptive behavior at age four and theory of mind at age seven. Principal component analyses revealed less integrated processing of facial features and eye-gaze information in ASC infants, potentially impacting their childhood social functioning. These findings highlight the intertwined nature of social-cognitive development and ASC.

ShouldAR: Detecting Shoulder Surfing Attacks Using Multimodal Eye Tracking and Augmented Reality

Shoulder surfing attacks (SSAs) are a type of observation attack designed to illicitly gather sensitive data from "over the shoulder" of victims. This attack can be directed at mobile devices, desktop screens, Personal Identification Number (PIN) pads at an Automated Teller Machine (ATM), or written text. Existing solutions are generally focused on authentication techniques (e.g., logins) and are limited to specific attack scenarios (e.g., mobile devices or PIN Pads). We present ShotjldAR, a mobile and usable system to detect SSAs using multimodal eye gaze information (i.e., from both the potential attacker and victim). ShouldAR uses an augmented reality headset as a platform to incorporate user eye gaze tracking, rear-facing image collection and eye gaze analysis, and user notification of potential attacks. In a 24-participant study, we show that the prototype is capable of detecting 87.28% of SSAs against both physical and digital targets, a two-fold improvement on the baseline solution using a rear-facing mirror, a widely used solution to the SSA problem. The ShouldAR approach provides an AR-based, active SSA defense that applies to both digital and physical information entry in sensitive environments.

Peripheral Eye Tracking for Measurement of Military Pilot Performance

Abstract Flying plays a crucial role in establishing both tactical and strategic superiority through direct engagement, facilitating logistics movement, and providing other essential supports. However, a concerning trend has been observed regarding pilot flight time, which has decreased from approximately 300 hours per year to 230 hours per year. This decline could potentially impact flight safety and training, particularly in situations requiring rapid decision-making. One of the critical factors affecting rapid decision-making is a pilot's ability to closely monitor the aircraft's systems and flight path. The objective of this paper is to review and synthesize a series of studies that investigated the impact of peripheral gaze behavior on flying performance. Pilots can process peripheral eye gaze information more quickly and effectively, and peripheral eye movements could be used to activate controls faster than typical motor reflexes. Mathematical models of visual attention related to peripheral eye tracking have proven to be valuable predictive tools for human-machine interfaces and can lay the groundwork for improved training of pilots. Understanding gaze behavior offers valuable insights into the human-machine interface in modern cockpits. Keywords: Peripheral eye tracking, Pilot decision making, Human-machine teaming.

Eye-tracking correlates of response to joint attention in preschool children with autism spectrum disorder

BackgroundA number of differences in joint attention behaviour between children with autism spectrum disorder (ASD) and typically developing (TD) individuals have previously been documented.MethodWe use eye-tracking technology to assess response to joint attention (RJA) behaviours in 77 children aged 31 to 73 months. We conducted a repeated-measures analysis of variance to identify differences between groups. In addition, we analysed correlations between eye-tracking and clinical measures using Spearman’s correlation.ResultsThe children diagnosed with ASD were less likely to follow gaze compared to TD children. Children with ASD were less accurate at gaze following when only eye gaze information was available, compared to when eye gaze with head movement was observed. Higher accuracy gaze-following profiles were associated with better early cognition and more adaptive behaviours in children with ASD. Less accurate gaze-following profiles were associated with more severe ASD symptomatology.ConclusionThere are differences in RJA behaviours between ASD and TD preschool children. Several eye-tracking measures of RJA behaviours in preschool children were found to be associated with clinical measures for ASD diagnosis. This study also highlights the construct validity of using eye-tracking measures as potential biomarkers in the assessment and diagnosis of ASD in preschool children.

Estimation of Confidence in the Dialogue based on Eye Gaze and Head Movement Information

In human-robot interaction, human mental states in dialogue have attracted attention to human-friendly robots that support educational use. Although estimating mental states using speech and visual information has been conducted, it is still challenging to estimate mental states more precisely in the educational scene. In this paper, we proposed a method to estimate human mental state based on participants’ eye gaze and head movement information. Estimated participants’ confidence levels in their answers to the miscellaneous knowledge question as a human mental state. The participants’ non-verbal information, such as eye gaze and head movements during dialog with a robot, were collected in our experiment using an eye-tracking device. Then we collect participants’ confidence levels and analyze the relationship between human mental state and non-verbal information. Furthermore, we also applied a machine learning technique to estimate participants’ confidence levels from extracted features of gaze and head movement information. As a result, the performance of a machine learning technique using gaze and head movements information achieved over 80 % accuracy in estimating confidence levels. Our research provides insight into developing a human-friendly robot considering human mental states in the dialogue.

Looking deep in the eyes: Investigating interpretation methods for neural models on reading tasks using human eye-movement behaviour

This paper provides the first broad overview of the relation between different interpretation methods and human eye-movement behaviour across different tasks and architectures. The interpretation methods of neural networks provide the information the machine considers important, while the human eye-gaze has been believed to be a proxy of the human cognitive process. Thus, comparing them explains machine behaviour in terms of human behaviour, leading to improvement in machine performance through minimising their difference. We consider three types of natural language processing (NLP) tasks: sentiment analysis, relation classification and question answering, and four interpretation methods based on: simple gradient, integrated gradient, input-perturbation and attention, and three architectures: LSTM, CNN and Transformer. We leverage two corpora annotated with eye-gaze information: the Zuco dataset and the MQA-RC dataset. This research sets up two research questions. First, we investigate whether the saliency (importance) of input-words conform with those from human eye-gaze features. To this end, we compute a saliency distance (SD) between input words (by an interpretation method) and an eye-gaze feature. SD is defined as the KL-divergence between the saliency distribution over input words and an eye-gaze feature. We found that the SD scores vary depending on the combinations of tasks, interpretation methods and architectures. Second, we investigate whether the models with good saliency conformity to human eye-gaze behaviour have better prediction performances. To this end, we propose a novel evaluation device called “SD-performance curve” (SDPC) which represents the cumulative model performance against the SD scores. SDPC enables us to analyse the underlying phenomena that were overlooked using only the macroscopic metrics, such as average SD scores and rank correlations, that are typically used in the past studies. We observe that the impact of good saliency conformity between humans and machines on task performance varies among the combinations of tasks, interpretation methods and architectures. Our findings should be considered when introducing eye-gaze information for model training to improve the model performance.

Investigating Search Among Physical and Virtual Objects Under Different Lighting Conditions.

By situating computer-generated content in the physical world, mobile augmented reality (AR) can support many tasks that involve effective search and inspection of physical environments. Currently, there is limited information regarding the viability of using AR in realistic wide-area outdoor environments and how AR experiences affect human behavior in these environments. Here, we conducted a wide-area outdoor AR user study ($n=48$) using a commercially available AR headset (Microsoft Hololens 2) to compare (1) user interactions with physical and virtual objects in the environment (2) the effects of different lighting conditions on user behavior and AR experience and (3) the impact of varying cognitive load on AR task performance. Participants engaged in a treasure hunt task where they searched for and classified virtual target items (green "gems") in an augmented outdoor courtyard scene populated with physical and virtual objects. Cognitive load was manipulated so that in half the search trials users were required to monitor an audio stream and respond to specific target sounds. Walking paths, head orientation and eye gaze information were measured, and users were queried about their memory of encountered objects and provided feedback on the experience. Key findings included (1) Participants self-reported significantly lower comfort in the ambient natural light condition, with virtual objects more visible and participants more likely to walk into physical objects at night; (2) recall for physical objects was worse than for virtual objects, (3) participants discovered more gems hidden behind virtual objects than physical objects, implying higher attention on virtual objects and (4) dual-tasking modified search behavior. These results suggest there are important technical, perceptual and cognitive factors that must be considered if the full potential of "anywhere and anytime mobile AR" is to be realized.

Improving disease classification performance and explainability of deep learning models in radiology with heatmap generators.

As deep learning is widely used in the radiology field, the explainability of Artificial Intelligence (AI) models is becoming increasingly essential to gain clinicians' trust when using the models for diagnosis. In this research, three experiment sets were conducted with a U-Net architecture to improve the disease classification performance while enhancing the heatmaps corresponding to the model's focus through incorporating heatmap generators during training. All experiments used the dataset that contained chest radiographs, associated labels from one of the three conditions ["normal", "congestive heart failure (CHF)", and "pneumonia"], and numerical information regarding a radiologist's eye-gaze coordinates on the images. The paper that introduced this dataset developed a U-Net model, which was treated as the baseline model for this research, to show how the eye-gaze data can be used in multi-modal training for explainability improvement and disease classification. To compare the classification performances among this research's three experiment sets and the baseline model, the 95% confidence intervals (CI) of the area under the receiver operating characteristic curve (AUC) were measured. The best method achieved an AUC of 0.913 with a 95% CI of [0.860, 0.966]. "Pneumonia" and "CHF" classes, which the baseline model struggled the most to classify, had the greatest improvements, resulting in AUCs of 0.859 with a 95% CI of [0.732, 0.957] and 0.962 with a 95% CI of [0.933, 0.989], respectively. The decoder of the U-Net for the best-performing proposed method generated heatmaps that highlight the determining image parts in model classifications. These predicted heatmaps, which can be used for the explainability of the model, also improved to align well with the radiologist's eye-gaze data. Hence, this work showed that incorporating heatmap generators and eye-gaze information into training can simultaneously improve disease classification and provide explainable visuals that align well with how the radiologist viewed the chest radiographs when making diagnosis.

Dynamic Warnings

People often unnecessarily disclose identity information online and puts their privacy at risk. Computer warnings mitigate needless identity disclosure. People, however, often click the OK button without reading warning messages. We utilize eye gaze information to provide dynamic warnings. The dynamic warnings are designed to display just-in-time and then fade out after users read them. They are shown right next to the location where users look. We built a restaurant reservation app to evaluate our dynamic warning system. We conducted an experiment with follow-up surveys. The results showed that our dynamic warnings reduced unnecessary identity disclosure and that they were around 5 times more effective than a Windows warning with a close button. We also found that the longer users’ eyes registered on dynamic warning, the less likely users were to provide their identify information in the dynamic warning condition experiment.

Direct eye gaze enhances the ventriloquism effect

The “ventriloquism effect” describes an illusory phenomenon where the perceived location of an auditory stimulus is pulled toward the location of a visual stimulus. Ventriloquists use this phenomenon to create an illusion where an inanimate puppet is perceived to speak. Ventriloquists use the expression and suppression of their own and the puppet’s mouth movements as well the direction of their respective eye gaze to maximize the illusion. While the puppet’s often exaggerated mouth movements have been demonstrated to enhance the ventriloquism effect, the contribution of direct eye gaze remains unknown. In Experiment 1, participants viewed an image of a person’s face while hearing a temporally synchronous recording of a voice originating from different locations on the azimuthal plane. The eyes of the facial stimuli were either looking directly at participants or were closed. Participants were more likely to misperceive the location of a range of voice locations as coming from a central position when the eye gaze of the facial stimuli were directed toward them. Thus, direct gaze enhances the ventriloquist effect by attracting participants’ perception of the voice locations toward the location of the face. In an exploratory analysis, we furthermore found no evidence for an other-race effect between White vs Asian listeners. In Experiment 2, we replicated the effect of direct eye gaze on the ventriloquism effect, also showing that faces per se attract perceived sound locations compared with audio-only sound localization. Showing a modulation of the ventriloquism effect by socially-salient eye gaze information thus adds to previous findings reporting top-down influences on this effect.

Human gaze-aware attentive object detection for ambient intelligence

Understanding human behavior and the surrounding environment is essential for realizing ambient intelligence (AmI), for which eye gaze and object information are reliable cues. In this study, the authors propose a novel human gaze-aware attentive object detection framework as an elemental technology for AmI. The proposed framework detects users’ attentive objects and shows more precise and robust performance against object-scale variations. A novel Adaptive-3D-Region-of-Interest (Ada-3D-RoI) scheme is designed as a front-end module, and scalable detection network structures are proposed to maximize cost-efficiency. The experiments show that the detection rate is improved up to 97.6% on small objects (14.1% on average), and it is selectively tunable with a tradeoff between accuracy and computational complexity. In addition, the qualitative results demonstrate that the proposed framework detects a user’s single object-of-interest only, even when the target object is occluded or extremely small. Complementary matters for follow-up study are presented as suggestions to extend the results of the proposed framework to further practical AmI applications. This study will help develop advanced AmI applications that demand a higher-level understanding of scene context and human behavior such as human–robot symbiosis, remote-/autonomous control, and augmented/mixed reality.

Selective eye-gaze augmentation to enhance imitation learning in Atari games

This paper presents the selective use of eye-gaze information in learning human actions in Atari games. Extensive evidence suggests that our eye movements convey a wealth of information about the direction of our attention and mental states and encode the information necessary to complete a task. Based on this evidence, we hypothesize that selective use of eye-gaze, as a clue for attention direction, will enhance the learning from demonstration. For this purpose, we propose a selective eye-gaze augmentation (SEA) network that learns when to use the eye-gaze information. The proposed network architecture consists of three sub-networks: gaze prediction, gating, and action prediction network. Using the prior 4 game frames, a gaze map is predicted by the gaze prediction network, which is used for augmenting the input frame. The gating network will determine whether the predicted gaze map should be used in learning and is fed to the final network to predict the action at the current frame. To validate this approach, we use publicly available Atari Human Eye-Tracking And Demonstration (Atari-HEAD) dataset consists of 20 Atari games with 28 million human demonstrations and 328 million eye-gazes (over game frames) collected from four subjects. We demonstrate the efficacy of selective eye-gaze augmentation compared to the state-of-the-art Attention Guided Imitation Learning (AGIL) and Behavior Cloning (BC). The results indicate that the selective augmentation approach (the SEA network) performs significantly better than the AGIL and BC. Moreover, to demonstrate the significance of selective use of gaze through the gating network, we compare our approach with the random selection of the gaze. Even in this case, the SEA network performs significantly better, validating the advantage of selectively using the gaze in demonstration learning.

GG-Net: Gaze Guided Network for Self-driving Cars

Imitation learning is used massively in autonomous driving for training networks to predict steering commands from frames using annotated data collected by an expert driver. Believing that the frames taken from a front-facing camera are completely mimicking the driver’s eyes raises the question of how eyes and the complex human vision system attention mechanisms perceive the scene. This paper proposes the idea of incorporating eye gaze information with the frames into an end-to-end deep neural network in the lane-following task. The proposed novel architecture, GG-Net, is composed of a spatial transformer network (STN), and a multitask network to predict steering angle as well as the gaze map for the input frame. The experimental results of this architecture show a great improvement in steering angle prediction accuracy of 36% over the baseline with inference time of 0.015 seconds per frame (66 fps) using NVIDIA K80 GPU enabling the proposed model to operate in real-time. We argue that incorporating gaze maps enhances the model generalization capability to the unseen environments. Additionally, a novel course-steering angle conversion algorithm with a complementing mathematical proof is proposed.

Word learning in monolingual and bilingual children: The influence of speaker eye-gaze.

The current study examined the impact of a speaker's gaze on novel-word learning in 4-5-year old monolingual (N = 23) and bilingual children (N = 24). Children were taught novel words when the speaker looked at the object both times while labeling it (consistent) and when the speaker looked at the object only the first time (inconsistent). During teaching, bilingual children differentiated between the target object (that matched the label) and non-target object (that did not match the label) earlier than the monolingual children on trials without eye-gaze information. However, during testing, monolingual children showed more robust retention of novel words than bilingual children in both conditions. Findings suggest that bilingualism shapes children's attention to eye-gaze during word learning, but that, ultimately, there is no bilingual advantage for utilizing this cue in the service of word retention.

Does decreased visual attention to faces underlie difficulties interpreting eye gaze cues in autism?

BackgroundShifts in eye gaze communicate social information that allows people to respond to another’s behavior, interpret motivations driving behavior, and anticipate subsequent behavior. Understanding the social communicative nature of gaze shifts requires the ability to link eye movements and mental state information about objects in the world. Autism spectrum disorder (ASD) is characterized by atypical sensitivity to eye gaze cues, which impacts social communication and relationships. We evaluated whether reduced visual attention to faces explains this difficulty in ASD.MethodsWe employed eye-tracking technology to measure visual attention to faces and gazed-at objects in a 4-alternative forced choice paradigm in adolescents with ASD and typically developing (TD) adolescents. Participants determined the target object that an actor was looking at in ecologically rich scenes. We controlled for group differences in task engagement and data quality.ResultsIn the Gaze Following task, adolescents with ASD were relatively impaired (Cohen’s d = 0.63) in the ability to identify the target object. In contrast to predictions, both groups exhibited comparable fixation durations to faces and target objects. Among both groups, individuals who looked longer at the target objects, but not faces, performed better in the task. Finally, among the ASD group, parent SSIS-Social Skills ratings were positively associated with performance on the Gaze Following task. In the Gaze Perception task, there was a similar pattern of results, which provides internal replication of the findings that visual attention to faces is not related to difficulty interpreting eye gaze cues. Together, these findings indicate that adolescents with ASD are capable of following gaze, but have difficulty linking gaze shifts with mental state information.LimitationsAdditional work is necessary to determine whether these findings generalize to individuals across the full autism spectrum. New paradigms that manipulate component processes of eye gaze processing need to be tested to confirm these interpretations.ConclusionsReduced visual attention to faces does not appear to contribute to atypical processing of eye gaze cues among adolescents with ASD. Instead, the difficulty for individuals with ASD is related to understanding the social communicative aspects of eye gaze information, which may not be extracted from visual cues alone.

Gaze direction biases emotion categorisation in schizophrenia

The successful integration of eye gaze direction and emotion cues from faces is important not only for co-ordinated interactions, but also for the detection of social signals alerting us to threat posed by a conspecific, or elsewhere in our immediate environment. It is now well-established that people with schizophrenia experience aberrant eye gaze and facial emotion processing. These social-cognitive differences might contribute to the maintenance of socially-themed delusions which are characterised by the hyper-attribution of threatening intentions to others. However, no study has directly examined whether the mechanisms which govern the integration of eye gaze and emotion information diverge in schizophrenia, and more importantly, whether this reflects a fundamental ‘bottom-up’ perceptual deficit or a ‘top-down’ cognitive bias. Fifteen outpatients diagnosed with schizophrenia and 21 healthy age- and IQ-matched controls performed an emotion categorisation task (anger/fear) on morphed facial expressions of anger or fear, displaying either direct or averted gaze. Results in both controls and patients replicated the previous finding that combinations of anger with direct gaze, and fear with averted gaze – which signal a relevant threat to the observer – benefited from more accurate emotion recognition than alternate gaze-emotion combinations. Bayesian model selection revealed that for patients this effect was mediated by a shift in decision bias towards emotions which signal self-relevant threat, rather than a change in sensitivity as observed in controls. These results critically highlight a different cognitive mechanism governing gaze and face-cued emotion integration in schizophrenia, which has a top-down influence on the evaluation of perceptual input.

Eye-gaze information input based on pupillary response to visual stimulus with luminance modulation.

This study develops an information-input interface in which a visual stimulus targeted by a user’s eye gaze is identified based on the pupillary light reflex to periodic luminance modulations of the object. Experiment 1 examines how pupil size changes in response to periodic luminance modulation of visual stimuli, and the results are used to develop an algorithm for information input. Experiment 2a examines the effectiveness of interfaces with two objects. The results demonstrate that 98% accurate identification of the gaze targeted object is possible if the luminance modulation frequencies of two objects differ by at least 0.12 Hz. Experiment 2b examines the accuracy of a gaze directed information input method based on a keyboard configuration with twelve responses. The results reveal that keyboard input is possible with an average accuracy of 85% for luminance modulation frequencies from 0.75 to 2.75 Hz. The proposed pupillometry based information-input interface offers several advantages, such as low burden on users, minimal invasiveness, no need for training or experience, high theoretical validity, and no need for calibration. Thus, the pupillometry method presented herein has advantages for practical use without requiring the eye’s position to be calibrated. Additionally, this method has a potential for the design of interfaces that allow patients with severely limited motor function to communicate with others.

Evaluating fMRI-Based Estimation of Eye Gaze During Naturalistic Viewing.

The collection of eye gaze information during functional magnetic resonance imaging (fMRI) is important for monitoring variations in attention and task compliance, particularly for naturalistic viewing paradigms (e.g., movies). However, the complexity and setup requirements of current in-scanner eye tracking solutions can preclude many researchers from accessing such information. Predictive eye estimation regression (PEER) is a previously developed support vector regression-based method for retrospectively estimating eye gaze from the fMRI signal in the eye's orbit using a 1.5-min calibration scan. Here, we provide confirmatory validation of the PEER method's ability to infer eye gaze on a TR-by-TR basis during movie viewing, using simultaneously acquired eye tracking data in five individuals (median angular deviation < 2°). Then, we examine variations in the predictive validity of PEER models across individuals in a subset of data (n= 448) from the Child Mind Institute Healthy Brain Network Biobank, identifying head motion as a primary determinant. Finally, we accurately classify which of the two movies is being watched based on the predicted eye gaze patterns (area under the curve = 0.90 ± 0.02) and map the neural correlates of eye movements derived from PEER. PEER is a freely available and easy-to-use tool for determining eye fixations during naturalistic viewing.

Eye Contact Is a Two-Way Street: Arousal Is Elicited by the Sending and Receiving of Eye Gaze Information.

Research shows that arousal is significantly enhanced while participants make eye contact with a live person compared to viewing a picture of direct or averted gaze. Recent research has pointed toward the potential for social interaction as a possible driving force behind the arousal enhancement. That is, eye gaze is not only a signal perceived but also a signal sent out in order to communicate with others. This study aimed to test this by having dyads engage in eye contact and averted gaze naturally, while wearing sunglasses, and while blindfolded; such that the gaze signals were clear, degraded, and blocked, respectively. Autonomic nervous system arousal was measured via skin conductance response and level. The results showed that dyads exhibited the highest degree of arousal (increased skin conductance) while making eye contact (send/receive) compared to send-only or receive-only gaze trials; however, this was only the case if eye contact was clear. Once gaze information became degraded (by sunglasses or blindfold), arousal significantly decreased and was no longer modulated by the sending and receiving of gaze. Therefore, the arousal enhancement observed during eye contact is not only caused by receiving gaze signals (the focus of previous research) and should be more accurately attributed to the subtle interplay between sending and receiving gaze signals.

Software Architecture for Automating Cognitive Science Eye-Tracking Data Analysis and Object Annotation

The advancement of wearable eye-tracking technology enables cognitive researchers to capture vast amounts of eye gaze information while participants are completing specific tasks without restrictions on their movement. However, while eye trackers can overlay a gaze indicator on the scene video, identifying the specific objects being looked at and analyzing the resulting dataset are accomplished mostly by manual annotation. This method is a cost-prohibitive and time-consuming approach that is prone to human error. Such analytic difficulty limits researchers’ ability to data mine the information efficiently, ultimately restricting the number of scenarios that can feasibly be conducted within budget. Here, the first fully automated solution for eye-tracking data analysis is presented, which eliminates the need for manual annotation. The proposed software architecture, gaze to object classification (GoC), processes the gaze-overlaid video from commercially available wearable eye trackers, recognizes and classifies the specific object a user is focusing on and calculates the gaze duration time. GoC utilizes an image cross-correlation method to locate the gaze indicator and an image similarity measurement to support faster processing. The presented system has been successfully adopted by cognitive psychologists. GoC's exceptional performance in analyzing a case study spanning over 50 h of mobile eye-tracking is presented. The accuracy and a cost-analysis comparison between GoC and state-of-the-art manual annotation software are provided. GoC has game-changing potential for increasing the ecological validity of using eye-tracking technology in cognitive research.