Eye Gaze Data Research Articles

Acceptability and Feasibility of Portable Eye-Tracking Technology within a Children's Dynamic Sport Context: An Exploratory Study with Boys Who Play Grassroots Football.

Teaching practices are moving from decontextualised to more representative curricula. Although this is argued to be a positive step, low motor competence is a continual issue in primary-aged school children. One methodological approach to investigate ways to improve motor competence, eye tracking, is moving to more representative tasks. So far, eye-tracking research using static activities has demonstrated a positive association between motor competence and earlier fixation and longer duration. However, this research has been constrained to laboratory settings and tasks, or discrete activities (e.g., throw and catch). This study seeks to understand how to conduct more representative eye-tracking research in primary school-aged children. To this end, thirteen 10-11-year-old children were fitted with an eye-tracker during a typical football coaching session. Children were asked acceptability-based questions, and eye-gaze data were captured to illustrate what children attended to under a representative dynamic football-based activity. Based on the voices of children and captured eye-gaze data, six practical implications for research in this population are proposed: (1) conduct eye-tracking research indoors (where possible); (2) ensure long hair or fringes are secured so as not to obscure line of sight; (3) run the same activity to increase comparability across children wearing the eye-tracker; (4) use a properly fitted backpack (if a backpack is to be used); (5) assure children about the capability and hardiness of the eye-tracker, as they do not need to change the way they move; (6) explain there may be some discomfort with the nose clip, head strap, and battery weight and ensure that children wish to continue.

Early cognitive assessment in premature infants: the discriminatory value of eye-tracking vs. Bayley Scales.

The testing of visuocognitive development in preterm infants shows strong interactions between perinatal characteristics and cognition, learning and overall neurodevelopment evolution. The assessment of anticipatory gaze data of object-location bindings via eye-tracking can predict the neurodevelopment of preterm infants at the age of 3 years; little is known, however, about the early cognitive function and its assessment methods during the first year of life. The current study presents data from a novel assessment tool, a Delayed Match Retrieval (DMR) paradigm via eye-tracking was used to measure visual working memory (VWM) and attention skills. The eye-tracking task that was designed to measure infants' ability to actively localize objects and to make online predictions of object-location bindings. 63 infants participated in the study, 39 preterm infants and 24 healthy full term infants - at a corrected age of 8-9 months for premature infants and similar chronological age for full term infants. Infants were also administered the Bayley Scales of Infant and Toddler Development. The analysis of the Bayley scores showed no significant difference between the two groups while the eye-tracking data showed a significant group effect on all measurements. Moreover, preterm infants' VWM performance was significantly lower than full term's. Birth weight affected the gaze time on all Areas Of Interest (AOIs), overall VWM performance and the scores at the Cognitive Bayley subscale. Furthermore, preterm infants with fetal growth restriction (FGR) showed significant performance effects in the eye-tracking measurements but not on their Bayley scores verifying the high discriminatory value of the eye gaze data. Visual working memory and attention as measured via eye-tracking is a non-intrusive, painless, short duration procedure (approx. 4-min) was found to be a significant tool for identifying prematurity and FGR effects on the development of cognition during the first year of life. Bayley Scales alone may not pick up these deficits. Identifying tools for early neurodevelopmental assessments and cognitive function is important in order to enable earlier support and intervention in the vulnerable group of premature infants, given the associations between foundational executive functional skills and later cognitive and academic ability.

On current limitations of online eye-tracking to study the visual processing of source code

Context:Eye-tracking is an increasingly popular instrument to study how programmers process and comprehend source code. While most studies are conducted in controlled environments with lab-grade hardware, it would be desirable to simplify and scale participation in experiments for users sitting remotely, leveraging home equipment. Objective:This study investigates the possibility of performing eye-tracking studies remotely using open-source algorithms and consumer-grade webcams. It establishes the technology’s current limitations and evaluates the quality of the data collected by it. We conclude by recommending ways forward to address the shortcomings and make remote code-reading studies in support of eye-tracking feasible in the future. Method:We gathered eye-gaze data remotely from 40 participants performing a code reading experiment on a purpose-built web application. The utilized eye-tracker worked client-side and used ridge regression to generate x- and y-coordinates in real-time predicting the participants’ on-screen gaze points without the need to collect and save video footage. We processed and analysed the collected data according to common practices for isolating eye-movement events and deriving metrics used in software engineering eye-tracking studies. In response to the lack of an algorithm explicitly developed for detecting oculomotor fixation events in low-frequency webcam data, we also introduced a dispersion threshold algorithm for that purpose. The quality of the collected data was subsequently assessed to determine the adequacy and validity of the methodology for eye-tracking. Results:The collected data was found to be of varying quality despite extensive calibration and graphical user guidance. We present our results highlighting both the negative and positive observations from which the community hopefully can learn. Both accuracy and precision were low and ultimately deemed insufficient for drawing valid conclusions in a high-precision empirical study. We nonetheless contribute to identifying critical limitations to be addressed in future research. Apart from the overall challenge of vastly diverse equipment, setup, and configuration, we found two main problems with the current webcam eye-tracking technology. The first was the absence of a validated algorithm to isolate fixations in low-frequency data, compromising the assurance of the accuracy of the data derived from it. The second problem was the lack of algorithmic support for head movements when predicting gaze location. Unsupervised participants do not always keep their heads still, even if instructed to do so. Consequently, we frequently observed spatial shifts that corrupted many collected datasets. Three encouraging observations resulted from the study. Even when shifted, gaze points were consistently dispersed in patterns resembling both the shape and size of the stimuli without extreme deviations. We could also distinguish recognizable reading patterns. Linearity was significantly different when participants were reading source code compared to natural text, and we could detect the expected left-to-right and top-to-bottom reading directions for participants reading natural text snippets. Conclusion:The accuracy and precision levels were not sufficient for a word-by-word analysis of code reading but could be adequate for a broader, coarse-grained precision study. Additionally we identified two main issues compromising the collected data validity and contributed a fixation detection algorithm to approach one of these issues. With suitable solutions to the identified issues, remote eye-tracking studies with webcams on code reading could eventually be feasible.

EyeXNet: Enhancing Abnormality Detection and Diagnosis via Eye-Tracking and X-ray Fusion

Integrating eye gaze data with chest X-ray images in deep learning (DL) has led to contradictory conclusions in the literature. Some authors assert that eye gaze data can enhance prediction accuracy, while others consider eye tracking irrelevant for predictive tasks. We argue that this disagreement lies in how researchers process eye-tracking data as most remain agnostic to the human component and apply the data directly to DL models without proper preprocessing. We present EyeXNet, a multimodal DL architecture that combines images and radiologists’ fixation masks to predict abnormality locations in chest X-rays. We focus on fixation maps during reporting moments as radiologists are more likely to focus on regions with abnormalities and provide more targeted regions to the predictive models. Our analysis compares radiologist fixations in both silent and reporting moments, revealing that more targeted and focused fixations occur during reporting. Our results show that integrating the fixation masks in a multimodal DL architecture outperformed the baseline model in five out of eight experiments regarding average Recall and six out of eight regarding average Precision. Incorporating fixation masks representing radiologists’ classification patterns in a multimodal DL architecture benefits lesion detection in chest X-ray (CXR) images, particularly when there is a strong correlation between fixation masks and generated proposal regions. This highlights the potential of leveraging fixation masks to enhance multimodal DL architectures for CXR image analysis. This work represents a first step towards human-centered DL, moving away from traditional data-driven and human-agnostic approaches.

Optimising virtual object position for efficient eye-gaze interaction in Hololens2

ABSTRACT Our study explored eye-tracking technology in the Hololens2 HMD. We assessed the effectiveness of eye-gaze interactions in a 3D environment, particularly in text entry applications. Existing recommendations for the placement and size of virtual objects are often followed without empirical validation. Therefore, we evaluated text entry target selection rates within the manufacturer’s specified optimal gaze interaction zone. We measured the spatial accuracy and precision of eye gaze data and optimised target positions for enhanced text entry performance. By establishing confidence ellipses covering 95% of gaze points per target, we derived an Area of Interest (AOI) recalibration function. Applying a receiver operating characteristic-based method, we quantified the recalibrated tracker’s performance at various AOIs. Our results indicate an optimal recalibrated AOI radius is 0.036 m, and the ideal object-plane distance from the eye-plane is 2.25 m. This recalibrated specification allows to efficiently interact with Hololens2 via eye movement, achieving target selection rates exceeding 90%.

Gaze-based human intention prediction in the hybrid foraging search task

An agent’s ability to predict a human’s intention can facilitate the effectiveness of a human-agent team. The aim of this study was to explore the feasibility of predicting human intention in the hybrid foraging search task using eye gaze data and machine learning. In the hybrid foraging search paradigm, the human performs searches in patches and decides to stay and exploit a patch or to leave and explore another patch. Researchers have modeled the “patch-leaving problem” on the collective level using the marginal value theorem. However, few have attempted to predict the exact moment when the human will leave. In the current study, 40 participants performed the hybrid foraging search task while eye gaze data were collected with an eye-tracker. The leaving intention was associated with larger pupil size, shorter average fixation duration, larger number of fixations, longer saccade amplitude, and faster saccade velocity compared to the searching intention. A cross-subject machine learning model with an artificial neural network algorithm was able to predict whether a participant would leave the current patch with up to 78% accuracy with a 2-second data analysis window. The length of the data analysis window did not significantly affect prediction accuracy. Furthermore, the earlier the behavior prediction was made, the lower the reliability of the prediction. These results demonstrate that eye gaze features and machine learning algorithms are useful in predicting human intention in visual search tasks.

The Gaze Dialogue Model: Nonverbal Communication in HHI and HRI.

When humans interact with each other, eye gaze movements have to support motor control as well as communication. On the one hand, we need to fixate the task goal to retrieve visual information required for safe and precise action-execution. On the other hand, gaze movements fulfil the purpose of communication, both for reading the intention of our interaction partners, as well as to signal our action intentions to others. We study this Gaze Dialogue between two participants working on a collaborative task involving two types of actions: 1) individual action and 2) action-in-interaction. We recorded the eye-gaze data of both participants during the interaction sessions in order to build a computational model, the Gaze Dialogue, encoding the interplay of the eye movements during the dyadic interaction. The model also captures the correlation between the different gaze fixation points and the nature of the action. This knowledge is used to infer the type of action performed by an individual. We validated the model against the recorded eye-gaze behavior of one subject, taking the eye-gaze behavior of the other subject as the input. Finally, we used the model to design a humanoid robot controller that provides interpersonal gaze coordination in human-robot interaction scenarios. During the interaction, the robot is able to: 1) adequately infer the human action from gaze cues; 2) adjust its gaze fixation according to the human eye-gaze behavior; and 3) signal nonverbal cues that correlate with the robot's own action intentions.

Effects of age on behavioural and eye gaze on Theory of Mind using movie for social cognition.

Evidence has shown that older adults have lower accuracy in Theory of Mind (ToM) tasks compared with young adults, but we are still unclear whether the difficulty in decoding mental states in older adults stems from not looking at the critical areas, and more so from the ageing Asian population. Most ToM studies use static images or short vignettes to measure ToM but these stimuli are dissimilar to everyday social interactions. We investigated this question using a dynamic task that measured both accuracy and error types, and examined the links between accuracy and error types to eye gaze fixation at critical areas (e.g., eyes, mouth, body). A total of 82 participants (38 older, 44 young adults) completed the Movie for the Assessment of Social Cognition (MASC) task on the eye tracker. Results showed that older adults had a lower overall accuracy with more errors in the ipo-ToM (under-mentalising) and no-ToM (lack of mentalisation) conditions compared with young adults. We analysed the eye gaze data using principal components analysis and found that increasing age and looking less at the face were related to lower MASC accuracy in our participants. Our findings suggest that ageing deficits in ToM are linked to a visual attention deficit specific to the perception of socially relevant nonverbal cues.

Improving cognitive-state analysis from eye gaze with synthetic eye-movement data

Eye movements can be used to analyze a viewer’s cognitive capacities or mental state. Neural networks that process the raw eye-tracking signal can outperform methods that operate on scan paths preprocessed into fixations and saccades. However, the scarcity of such data poses a major challenge. We therefore develop SP-EyeGAN, a neural network that generates synthetic raw eye-tracking data. SP-EyeGAN consists of Generative Adversarial Networks; it produces a sequence of gaze angles indistinguishable from human ocular micro- and macro-movements. We explore the use of these synthetic eye movements for pre-training neural networks using contrastive learning. We find that pre-training on synthetic data does not help for biometric identification, while results are inconclusive for the detection of ADHD and gender classification. However, for the eye movement-based assessment of higher-level cognitive skills such general reading comprehension, text comprehension, and the distinction of native from non-native readers, pre-training on synthetic eye-gaze data improves the models’ performance and even advances the state-of-the-art for reading comprehension. The SP-EyeGAN model, pre-trained on GazeBase, along with the code for developing your own raw eye-tracking machine learning model with contrastive learning, is available at https://github.com/aeye-lab/sp-eyegan.

Rectify ViT Shortcut Learning by Visual Saliency.

Shortcut learning in deep learning models occurs when unintended features are prioritized, resulting in degenerated feature representations and reduced generalizability and interpretability. However, shortcut learning in the widely used vision transformer (ViT) framework is largely unknown. Meanwhile, introducing domain-specific knowledge is a major approach to rectifying the shortcuts that are predominated by background-related factors. For example, eye-gaze data from radiologists are effective human visual prior knowledge that has the great potential to guide the deep learning models to focus on meaningful foreground regions. However, obtaining eye-gaze data can still sometimes be time-consuming, labor-intensive, and even impractical. In this work, we propose a novel and effective saliency-guided ViT (SGT) model to rectify shortcut learning in ViT with the absence of eye-gaze data. Specifically, a computational visual saliency model (either pretrained or fine-tuned) is adopted to predict saliency maps for input image samples. Then, the saliency maps are used to filter the most informative image patches. Considering that this filter operation may lead to global information loss, we further introduce a residual connection that calculates the self-attention across all the image patches. The experiment results on natural and medical image datasets show that our SGT framework can effectively learn and leverage human prior knowledge without eye-gaze data and achieves much better performance than baselines. Meanwhile, it successfully rectifies the harmful shortcut learning and significantly improves the interpretability of the ViT model, demonstrating the promise of transferring human prior knowledge derived visual saliency in rectifying shortcut learning.

Improving the Effectiveness of E-learning Videos by leveraging Eye-gaze Data

Recent advances in technology strengthen remote and lifelong learning by integrating e-videos into teaching-learning pedagogy. Therefore, educational content developers are tasked with creating engaging and qualitative e-content. The shift in paradigm from offline to online teaching brings forth several issues regarding the quality of online learning materials and the missing dynamic interaction between instructors and learners. Leveraging contemporary artificial intelligence techniques to provide insights into methods for developing quality e-content is the need of the hour. This study showed that the pattern and duration of the eye gaze of the learner on the text, image, or instructor in the video reveal valuable insights, not only regarding the comprehension of the learner but also giving suggestions to improve video lectures. The results show that learners perform better when they spend more time looking at the instructor compared to the image and text on a frame. Therefore, just like classroom teaching, the presence of the instructor in the video is vital, as looking directly at the instructor while they are delivering the lecture encourages comprehension. Furthermore, by applying classification techniques to learner eye gaze data, it was possible to predict with 97% confidence whether the learner would answer the post-quiz correctly or not.

Rethinking Pedagogical Use of Eye Trackers for Visual Problems with Eye Gaze Interpretation Tasks

Eye tracking technology enables the visualisation of a problem solver's eye movement while working on a problem. The eye movement of experts has been used to draw attention to expert problem solving processes in a bid to teach procedural skills to learners. Such affordances appear as eye movement modelling examples (EMME) in the literature. This work intends to further this line of work by suggesting how eye gaze data can not only guide attention but also scaffold learning through constructive engagement with the problem solving process of another human. Inferring the models’ problem solving process, be it that of an expert or novice, from their eye gaze display would require a learner to make interpretations that are rooted in the knowledge elements relevant to such problem solving. Such tasks, if designed properly, are expected to probe or foster a deeper understanding of a topic as their solutions would require not only following the expert gaze to learn a particular skill, but also interpreting the solution process as evident from the gaze pattern of an expert or even of a novice. This position paper presents a case for such tasks, which we call eye gaze interpretation (EGI) tasks. We start with the theoretical background of these tasks, followed by a conceptual example and representation to elucidate the concept of EGI tasks. Thereafter, we discuss design considerations and pedagogical affordances, using a domain-specific (chemistry) spectral graph problem. Finally, we explore the possibilities and constraints of EGI tasks in various fields that require visual representations for problem solving.

Head Mounted IMU-Based Driver’s Gaze Zone Estimation Using Machine Learning Algorithm

Predicting the driver’s gaze could be important information in preventing accidents while driving. In this study, machine learning models for estimating the driver’s gaze distraction through head movement data were created and their performance was compared and evaluated. Participants wore glasses-type eye trackers and performed the task of selecting the touch screen buttons while driving. The input variable used in the model was data obtained from a 3-axis accelerometer sensor and a 3-axis gyroscope sensor, and the target variable was eye-gaze data. As a result, it was confirmed that the gaze area could be estimated with a precision, sensitivity, specificity, and F1-score of 72.1%, 72.5%, 66.0%, and 69.3%, respectively, only with the head movement sensing data. The model trained using time-series datasets had higher performance than using non-time series datasets. This study presented one alternative that could be used to determine the driver’s status with an inexpensive sensor.

Surgical skill level classification model development using EEG and eye-gaze data and machine learning algorithms

The aim of this study was to develop machine learning classification models using electroencephalogram (EEG) and eye-gaze features to predict the level of surgical expertise in robot-assisted surgery (RAS). EEG and eye-gaze data were recorded from 11 participants who performed cystectomy, hysterectomy, and nephrectomy using the da Vinci robot. Skill level was evaluated by an expert RAS surgeon using the modified Global Evaluative Assessment of Robotic Skills (GEARS) tool, and data from three subtasks were extracted to classify skill levels using three classification models—multinomial logistic regression (MLR), random forest (RF), and gradient boosting (GB). The GB algorithm was used with a combination of EEG and eye-gaze data to classify skill levels, and differences between the models were tested using two-sample t tests. The GB model using EEG features showed the best performance for blunt dissection (83% accuracy), retraction (85% accuracy), and burn dissection (81% accuracy). The combination of EEG and eye-gaze features using the GB algorithm improved the accuracy of skill level classification to 88% for blunt dissection, 93% for retraction, and 86% for burn dissection. The implementation of objective skill classification models in clinical settings may enhance the RAS surgical training process by providing objective feedback about performance to surgeons and their teachers.

CASES

The process of reading has attracted decades of scientific research. Work in this field primarily focuses on using eye gaze patterns to reveal cognitive processes while reading. However, eye gaze patterns suffer from limited resolution, jitter noise, and cognitive biases, resulting in limited accuracy in tracking cognitive reading states. Moreover, using sequential eye gaze data alone neglects the linguistic structure of text, undermining attempts to provide semantic explanations for cognitive states during reading. Motivated by the impact of the semantic context of text on the human cognitive reading process, this work uses both the semantic context of text and visual attention during reading to more accurately predict the temporal sequence of cognitive states. To this end, we present a Cognition-Aware Smart Eyewear System (CASES), which fuses semantic context and visual attention patterns during reading. The two feature modalities are time-aligned and fed to a temporal convolutional network based multi-task classification deep model to automatically estimate and further semantically explain the reading state timeseries. CASES is implemented in eyewear and its use does not interrupt the reading process, thus reducing subjective bias. Furthermore, the real-time association between visual and semantic information enables the interactions between visual attention and semantic context to be better interpreted and explained. Ablation studies with 25 subjects demonstrate that CASES improves multi-label reading state estimation accuracy by 20.90% for sentence compared to eye tracking alone. Using CASES, we develop an interactive reading assistance system. Three and a half months of deployment with 13 in-field studies enables several observations relevant to the study of reading. In particular, observed how individual visual history interacts with the semantic context at different text granularities. Furthermore, CASES enables just-in-time intervention when readers encounter processing difficulties, thus promoting self-awareness of the cognitive process involved in reading and helping to develop more effective reading habits.

Performance and learning rate prediction models development in FLS and RAS surgical tasks using electroencephalogram and eye gaze data and machine learning

ObjectiveThis study explored the use of electroencephalogram (EEG) and eye gaze features, experience-related features, and machine learning to evaluate performance and learning rates in fundamentals of laparoscopic surgery (FLS) and robotic-assisted surgery (RAS).MethodsEEG and eye-tracking data were collected from 25 participants performing three FLS and 22 participants performing two RAS tasks. Generalized linear mixed models, using L1-penalized estimation, were developed to objectify performance evaluation using EEG and eye gaze features, and linear models were developed to objectify learning rate evaluation using these features and performance scores at the first attempt. Experience metrics were added to evaluate their role in learning robotic surgery. The differences in performance across experience levels were tested using analysis of variance.ResultsEEG and eye gaze features and experience-related features were important for evaluating performance in FLS and RAS tasks with reasonable results. Residents outperformed faculty in FLS peg transfer (p value = 0.04), while faculty and residents both excelled over pre-medical students in the FLS pattern cut (p value = 0.01 and p value < 0.001, respectively). Fellows outperformed pre-medical students in FLS suturing (p value = 0.01). In RAS tasks, both faculty and fellows surpassed pre-medical students (p values for the RAS pattern cut were 0.001 for faculty and 0.003 for fellows, while for RAS tissue dissection, the p value was less than 0.001 for both groups), with residents also showing superior skills in tissue dissection (p value = 0.03).ConclusionFindings could be used to develop training interventions for improving surgical skills and have implications for understanding motor learning and designing interventions to enhance learning outcomes.Graphical abstract

JS-11 Estimation of stress resilience from eye-gaze data collected by a tablet terminal when performing a dot- probe task

JS-11 Estimation of stress resilience from eye-gaze data collected by a tablet terminal when performing a dot- probe task

BIGaze: An eye-gaze action-guided Bayesian information gain framework for information exploration

Bayesian information gain (BIG) framework-based information exploration studies have generally focused on explicit interactions, such as user browsing via keyboard and mouse. However, if implicit information, such as eye-gaze data, is implemented in the BIG framework alongside explicit interactions, the system can accurately respond and proactively provide target information to the user. We, therefore, propose the BIGaze system, which uses real-time eye-gaze data to predict and recommend target information by analyzing the browsing and gazing action patterns corresponding to information searched by the user. To validate the performance of the proposed system within the information exploration process, we performed comparative user experiments with BIGaze, BIGbase (a BIG system without gaze data), and a non-BIG system. The experimental results reveal that BIGaze proactively captures user exploration targets, confidently generates user-intended recommendations, and assists a wide range of user exploration patterns.

The Effect of Time Pressure on Visual Search Eye Tracking Metrics in Virtual Reality

The study of visual search has focused on various guiding factors, but less attention has been given to how environmental factors affect visual search in virtual reality (VR). The visual search literature has primarily been based on 2D laboratory tasks, which lack the complexity of real-life search tasks. Thus, this study studies the effect of time pressure on visual search in a naturalistic environment. To do that, participants were immersed in a virtual living room using VR and tasked with finding objects under a time constraint. Eye gaze data was collected, and convex hull volumes and scanning rates were calculated and analyzed. The results show that time pressure reduced convex hull volume and increased scanning rate, indicating faster search speed and a gaze tunneling effect. Understanding how time pressure affects visual search can help improve training strategies and design better user interfaces for visual search critical domains.

The time course of moral decision making in bilinguals' native and foreign language

Previous work has shown that moral decision making depends on whether moral dilemmas are presented in the native or a foreign language (Foreign Language effect). In that work, the focus was on bilinguals' final decision, but the tasks used, failed to capture the processes involved over time. The aim of this study was to examine the time course of moral decision making in bilinguals' native and foreign language prior to and after their moral decision. In a visual-world eye-tracking experiment, 82 Dutch-English bilinguals listened to 20 moral dilemmas (e.g., would you kill one to save five?) in their native or foreign language, while looking at two pictures containing key people involved in the dilemmas. These pictures illustrated (1) the person/people that is/are sacrificed, and (2) the person/people that is/are not sacrificed, depending on the participants' decision which was measured with yes/no-questions. The Foreign Language effect was replicated for the decisions on the personal dilemmas. Importantly, the eye gaze data showed that in the native language, listeners looked at the person who they did not sacrifice, whereas in the foreign language, they looked at the people who they sacrificed. A speculative explanation is that bilinguals might have experienced guilt in the native language, and therefore focused attention on the person they did not sacrifice, while in the foreign language they might have experienced less guilt because they focused attention on the people they sacrificed. More research is needed to understand the influence of factors such as emotion reduction and cognitive load on moral decision making.