Gaze Data Research Articles

The impact of sound in people's behaviour in outdoor settings: A study using virtual reality and eye-tracking

This paper presents an analysis of space perception and how visual cues, such as landmarks and sound, are perceived and impact people's behaviour while exploring a given outdoor space. The primary goal of the research is to investigate how auditory sensations and visual stimuli influence people's behaviour in outdoor built environments. Our technique compares people's perception of the built environment in different conditions: the real world and a replicated virtual world. As a case study, a university campus was used, and four experimental conditions were designed. The study followed a between-subjects design, and the data collection included gaze data acquired from an eye-tracking device as well as self-reports. The study concludes that sound influences human behaviour in such settings. More specifically conclusions are that: i) human behaviour in virtual replications of the real space, including both visual and sound stimuli, is tendentially more similar to human behaviour in the real world than in simulations omitting sound; and ii) there is a difference in human behaviour when people explore the same virtually replicated outdoor space, by varying the presence of sound. This study is particularly useful for researchers working on the comparison between human behaviour in virtual and real environments, related to visual and sound stimuli.

What we see is what we do: a practical Peripheral Vision-Based HMM framework for gaze-enhanced recognition of actions in a medical procedural task

Deep learning models have shown remarkable performances in egocentric video-based action recognition (EAR), but rely heavily on a large quantity of training data. In specific applications with only limited data available, eye movement data may provide additional valuable sensory information to achieve accurate classification performances. However, little is known about the effectiveness of gaze data as a modality for egocentric action recognition. We, therefore, propose the new Peripheral Vision-Based HMM (PVHMM) classification framework, which utilizes context-rich and object-related gaze features for the detection of human action sequences. Gaze information is quantified using two features, the object-of-interest hit and the object–gaze distance, and human action recognition is achieved by employing a hidden Markov model. The classification performance of the framework is tested and validated on a safety-critical medical device handling task sequence involving seven distinct action classes, using 43 mobile eye tracking recordings. The robustness of the approach is evaluated using the addition of Gaussian noise. Finally, the results are then compared to the performance of a VGG-16 model. The gaze-enhanced PVHMM achieves high classification performances in the investigated medical procedure task, surpassing the purely image-based classification model. Consequently, this gaze-enhanced EAR approach shows the potential for the implementation in action sequence-dependent real-world applications, such as surgical training, performance assessment, or medical procedural tasks.

THREAT ANALYSIS AND SECURITY OF PERSONAL DATA USING USER DEVICES WITH EYE-TRACKERS

Eye-tracking technology has been extensively used in web development, marketing, and gaming mechanics because of the ability to obtain data regarding the trajectory and gaze fixation points. Eye movement information may be effectively used to study complex cognitive processes and human visual perception. However, a significant issue of leaking user privacy has arisen with the expansion of eye-tracking technology. Since the technology can obtain highly sensitive information regarding user behavior and preferences, measures for protecting personal data are required. At the current stage of the development of technology, research on threats and means to ensure the security of personal data has become a high priority in this field. Proper diligence in the development and use of eye-tracking devices is becoming an integral part of the process. Potential threats associated with collecting, storing, and trans mitting information about users' gaze should be considered. Various methods have been proposed to ensure the security of personal data through the use of eye-tracking devices. The anonymization of data, which is the removal or replacement of personal identifying elements, appears to be among these. Transparency and user agreement to the collection and use of gaze data has also been an important aspect. This can be accomplished by explicitly providing information about the purpose of the data collection and the ability to control privacy. Threats related to the use of eye-tracking technology were considered, providing methods of ensuring the security of personal data. These are important steps in the development of the field to protect user privacy while improving the safety and ethics of using eye-tracking devices.

IntelEye: An Intelligent Tool for the Detection of Stressful State based on Eye Gaze Data While Watching Video

Technology to monitor mental health is gaining popularity as it helps to improve the cognitive and behavioral performance of an individual. Considering the growing need to monitor mental health, there is subsequent research in continuous and real-time monitoring technologies that can increase the quality of life by reducing the cost of health care. Eye tracking technology has played a significant role in monitoring a person's mental health. An intelligent system can apply several computational procedures to extract meaningful information from the massive physiological data obtained from eye tracking. The proposed model IntelEye is a tool to detect the stressful states of an individual while watching calm and stressful videos. The eye gaze measures based on pupil diameter, fixation, and blink were used for detecting stressful conditions. The data was collected from hospital employees, and the K Nearest Neighbor algorithm could successfully recognize the stressful states and the corresponding gaze location during stressful situations. IntelEye is not only identifying the stressful states but also has the novelty of identifying the scene and gaze location, making them stressful while watching the video.

Gaze and Environmental Context-Guided Deep Neural Network and Sequential Decision Fusion for Grasp Intention Recognition.

Grasp intention recognition plays a crucial role in controlling assistive robots to aid older people and individuals with limited mobility in restoring arm and hand function. Among the various modalities used for intention recognition, the eye-gaze movement has emerged as a promising approach due to its simplicity, intuitiveness, and effectiveness. Existing gaze-based approaches insufficiently integrate gaze data with environmental context and underuse temporal information, leading to inadequate intention recognition performance. The objective of this study is to eliminate the proposed deficiency and establish a gaze-based framework for object detection and its associated intention recognition. A novel gaze-based grasp intention recognition and sequential decision fusion framework (GIRSDF) is proposed. The GIRSDF comprises three main components: gaze attention map generation, the Gaze-YOLO grasp intention recognition model, and sequential decision fusion models (HMM, LSTM, and GRU). To evaluate the performance of GIRSDF, a dataset named Invisible containing data from healthy individuals and hemiplegic patients is established. GIRSDF is validated by trial-based and subject-based experiments on Invisible and outperforms the previous gaze-based grasp intention recognition methods. In terms of running efficiency, the proposed framework can run at a frequency of about 22 Hz, which ensures real-time grasp intention recognition. This study is expected to inspire additional gaze-related grasp intention recognition works.

Distinguishing Orthodontic Experts From Laypersons Through Gaze Analysis

Visual inspection is an important process conducted as an initial diagnostic step in medical examinations. It is assumed that the gaze movements of an orthodontist (expert) differ from those of a layperson. In this study, to examine whether the degree of proficiency in conducting a visual examination can be estimated from gaze movement, we conducted a gaze measurement experiment in which facial images (frontal and lateral images of three patients) were viewed by ten experts and ten laypersons. The performance in discriminating whether a subject was an expert or layperson exhibited a certain improvement when applying an aggregation method for the gaze data, that is, the grid gaze frequency and AOI gaze frequency. We examined whether proficiency levels could be determined using machine learning techniques. The results demonstrated that our method distinguished experts and laypersons relatively effectively using gaze frequency based on the grid and area of interest set by an expert for each face part.

Theory-based approach for assessing cognitive load during time-critical resource-managing human–computer interactions: an eye-tracking study

Computerized systems are taking on increasingly complex tasks. Consequently, monitoring automated computerized systems is becoming increasingly demanding for human operators, which is particularly relevant in time-critical situations. A possible solution might be adapting human–computer interfaces (HCI) to the operators’ cognitive load. Here, we present a novel approach for theory-based measurement of cognitive load based on tracking eye movements of 42 participants while playing a serious game simulating time-critical situations that required resource management at different levels of difficulty. Gaze data was collected within narrow time periods, calculated based on log data interpreted in the light of the time-based resource-sharing model. Our results indicated that eye fixation frequency, saccadic rate, and pupil diameter significantly predicted task difficulty, while performance was best predicted by eye fixation frequency. Subjectively perceived cognitive load was significantly associated with the rate of microsaccades. Moreover our results indicated that more successful players tended to use breaks in gameplay to actively monitor the scene, while players who use these times to rest are more likely to fail the level. The presented approach seems promising for measuring cognitive load in realistic situations, considering adaptation of HCI.

Machine Learning Prediction of Locomotion Intention from Walking and Gaze Data

In many applications of human–computer interaction, a prediction of the human’s next intended action is highly valuable. To control direction and orientation of the body when walking towards a goal, a walking person relies on visual input obtained by eye and head movements. The analysis of these parameters might allow us to infer the intended goal of the walker. However, such a prediction of human locomotion intentions is a challenging task, since interactions between these parameters are nonlinear and highly dynamic. We employed machine learning models to investigate if walk and gaze data can be used for locomotor prediction. We collected training data for the models in a virtual reality experiment in which 18 participants walked freely through a virtual environment while performing various tasks (walking in a curve, avoiding obstacles and searching for a target). The recorded position, orientation- and eye-tracking data was used to train an LSTM model to predict the future position of the walker on two different time scales, short-term predictions of 50[Formula: see text]ms and long-term predictions of 2.5[Formula: see text]s. The trained LSTM model predicted free walking paths with a mean error of 5.14[Formula: see text]mm for the short-term prediction and 65.73[Formula: see text]cm for the long-term prediction. We then investigated how much the different features (direction and orientation of the head and body and direction of gaze) contributed to the prediction quality. For short-term predictions, position was the most important feature while orientation and gaze did not provide a substantial benefit. In long-term predictions, gaze and orientation of the head and body provided significant contributions. Gaze offered the greatest predictive utility in situations in which participants were walking short distances or in which participants changed their walking speed.

Assessment of Core Surgical Skills Using a Mixed Reality Headset - The MoTOR Study.

Surgical skill assessment utilises direct observation and feedback by an expert which is potentially subjective, therefore obtaining objective data for hand and eye tracking is essential. Our aim was to evaluate a wearable mixed reality (MR) headset in these domains. Participants with differing levels of surgical expertise [novice (N), intermediate (I) & expert (E)] performed 4 simulated surgical tasks; 2 general dexterity (tasks 1&2) and 2 surgical skills (tasks 3&4) wearing the MR headset capturing their hand and eye movements (median & range). Metrics included hand path length and the speed of each index or thumb tip. Gaze data were also captured. Participant demographics, prior expertise and current experience were captured with an electronic survey. Data were analysed with a Shapiro-Wilk test or ANOVA as appropriate. A p-value of < 0.05 was significant. Thirty-six participants were analysed (N = 18, I = 8, E = 8). Tasks 1&2 revealed 2 speed outcomes (left index and left-hand speed) which were significant. For tasks 3&4, various outcomes were significant: path length for left hand (N:45cm vs. I:31cm vs. E:27cm, p = 0.03) and right hand (N:48cm vs. I:29cm vs. E:28cm, p = 0.01) and total time (N:456s vs. I:292 vs. E: 245, p = 0.0002). With left-hand-tying, average path length (N:61cm vs. I:39 vs. E:36, p = 0.04), average speed (N:11cm/s vs. I:23 vs. E:24, p = 0.03), and total time (N:156s vs. I:43 vs. E:37, p = 0.003) were significant. The gaze-tracking was not statistically significant. The MR headset can be utilised as a valid tool for surgical performance assessment. Outcomes including path length and speed can be valuable metrics captured by the MR Headset during the task completion for detecting surgical proficiency.

Effects of virtual target size, position, and parallax on vergence-accommodation conflict as estimated by actual gaze

Due to the increased utilization of stereoscopic displays, the scope of the vergence–accommodation conflict has been studied extensively to reveal how the human visual system operates. The purpose of this work was to study the phenomenon of vergence–accommodation conflict by comparing the theoretical eye vergence angle (vergence response) and gaze-based eye vergence angle (vergence stimulus) based on eye tracker gaze data. The results indicated that the gaze-based eye vergence angle was largest at the greatest parallax. The result also revealed that the eye vergence angle accuracy was significantly highest at the nearest parallax. Generally, accuracy improves when virtual objects are put in the middle and close to participants' positions. Moreover, the signed error decreases significantly when the virtual object is in the middle. Based on the results of this study, we can gain a greater understanding of the vergence–accommodation conflict in the stereoscopic environment.

Detecting distracted students in educational VR environments using machine learning on eye gaze data

Virtual Reality (VR) has been found useful to improve engagement and retention level of students, for some topics, compared to traditional learning tools such as books, and videos. However, a student could still get distracted and disengaged due to a variety of factors including stress, mind-wandering, unwanted noise, and external alerts. Student eye gaze data could be useful for detecting these distracted students. Gaze data-based visualizations have been proposed in the past to help a teacher monitor distracted students. However, it is not practical for a teacher to monitor a large number of student indicators while teaching. To help filter students based on distraction level, we propose an automated system based on machine learning to classify students based on their distraction level. The key aspects are: (1) we created a labeled eye gaze dataset from an educational VR environment, (2) we propose an automatic system to gauge a student’s distraction level from gaze data, and (3) we apply and compare several classifiers for this purpose. Each classifier classifies distraction, per educational activity section, into one of three levels (low, mid or high). Our results show that Random Forest (RF) classifier had the best accuracy (98.88%) compared to the other models we tested. Additionally, a personalized machine learning model using either RF, kNN, or Extreme Gradient Boosting (XGBoost) model was found to improve the classification accuracy significantly.

Identifying Distractors for People with Computer Anxiety Based on Mouse Fixations

Abstract Computer anxiety (CA) can be defined as fear and worries that someone may feel when using computers. Thus, people with CA may face problems when using computers at home, at work or for study purposes, resulting in multiple forms of barriers even before the actual interaction with computers. In this context, the purpose of this research is to identify user interface elements impacting task performance (i.e. distractors) for people with CA, using mouse fixation analysis as a proxy for eye gaze data. The study explores the relationship of mouse and eye gaze data collected with the help of 39 older adults interacting with a website. Results show that it is possible to identify UI elements acting as distractors (e.g. carousel, top menu) as well as those with which people with CA faced problems (e.g. side menu, search box, map), based on mouse fixations. Moreover, statistical differences show that the number of mouse fixations in navigation, content and distractors is different for different levels of CA. Furthermore, differences were found between CA groups regarding mouse and eye fixations, indicating that participants with higher CA levels had difficulty differentiating which areas of interest they should interact with using mouse. From the results, one expects that personalized systems could use the proposed approach to identify UI elements acting as distractors using mouse data and then simplify UIs based on different levels of CA.

An eye-tracking investigation of visual search strategies and test performance of L1 and L2 listening test takers

Through the use of eye-tracking technology and a while-listening performance (WLP) test, this study examined the differences in gaze behaviors and measured listening performances on test items (across various stages of the test) and compared them between native English-speaking (E-L1) and non-native English-speaking (E-L2) candidates. One hundred students from a public university in Singapore participated in the study. A series of Mann-Whitney U tests indicated that E-L1 candidates outperformed E-L2 candidates in the test with higher test scores. Using stringent data processing cutoffs (presence≥80% gaze data) and a series of non-parametric multivariate analyses, the study further found that the dynamicity of gaze behaviors on the test items across various stages of the test was similar between E-L1 and E-L2 candidates. However, there were distinctive differences in gaze behaviors between the two groups. For E-L1 candidates, none of the gaze behaviors on the test items across the different stages of the test predicted their overall listening test scores. In contrast, the overall listening test scores for E-L2 candidates was predicted by the average proportion of time that they had dwelled on the test items while simultaneously answering them and listening to the auditory text. The study is the first to show that keyword matching on the test items during the while-listening stage significantly contributes to WLP test performance for E-L2 candidates. These results suggest that the focal construct of the listening test is confounded by group-specific reading behaviors on the test items. In line with previous research, the use of the WLP test format for assessing second language listening comprehension is not recommended.

Smooth-pursuit performance during Eye-typing from Memory indicates Mental Fatigue.

Mental fatigue is known to occur as a result of activities related to e.g. transportation, health-care, military operations and numerous other cognitive demanding tasks. Gaze tracking has wide-ranging applications, with the technology becoming more compact and processing power reducing. Though numerous techniques have been applied to measure mental fatigue using gaze tracking, smooth-pursuit movement, a natural eye movement generated when following a moving object with gaze, has not been explored with relation to mental fatigue. In this paper, we report the results from a smooth-pursuit movement based eye-typing experiment with varying task difficulty to generate cognitive load, performed in the morning and afternoon by 36 participants. We have investigated the effects of time-on-task and time of day on mental fatigue using self-reported questionnaires and smooth-pursuit performance, extracted from the gaze data. The self-reported mental fatigue increased due to time-on-task, but the time of day did not have an effect. The results illustrate that smooth-pursuit movement performance declined with time-on-task, with increased error in the gaze position and an inability to match the speed of the moving object. The findings exhibit the feasibility of mental fatigue detection using smoothpursuit movements during an eye-interactive task of eye-typing.

Skill-Based Differences in the Detection and Utilization of Opponent Action Preferences Following Increasing Exposure and Changes in Tendencies.

We examined skill-based differences in the detection and utilization of contextual information over a period of increasing exposure to an opponent's action preferences in soccer. Moreover, we investigated the ability of athletes to adapt to changes in these action preferences over time. In an initial detection phase, the attacking opponent demonstrated a proclivity to either pass or dribble, with these preferences being reversed in a subsequent adaptation phase of the same length. Skilled soccer players showed superior anticipation accuracy across both phases compared with less-skilled counterparts. The skilled participants significantly enhanced their performance over both phases, despite a significant drop in performance immediately following the change in opponent action preferences. In contrast, the less-skilled group only improved over the detection phase. Gaze data revealed that the skilled participants fixated more on kinematically relevant areas, compared with the less-skilled group, and increased the time spent fixating the player "off the ball" following greater volumes of exposure. Our novel findings elaborate on how skilled performers use both action preferences and motion information to anticipate an opponent's impending actions in sport.

Areas of Interest (AOI) on marketing mix elements of green and non-green products in customer decision making

Technological advancements in eye-tracking have enabled the development of interactive experimental setups for studying consumer behaviour. A common method for examining gaze data is Area of Interest (AOI). Therefore, this study fully utilised eye-tracking tools to measure participants' allocation of visual attention to the marketing mix elements in green and non-green products. This is because the product, price, place, and promotion are still the most crucial factors that customers consider when purchasing. The primary objective of this study is to discover and understand the primary marketing function that directly influences customer decision-making from a neuromarketing perspective. Their eye movements were simultaneously registered using SMI Eye Tracking Glasses 2 Wireless, and the gaze locations of participants were measured from AOI. The findings of this study have a significant impact on the importance of eye movements in decision-making, particularly when choosing important marketing elements before purchasing green and non-green products.

Virtual reality boxing: Gaze-contingent manipulation of stimulus properties using blur.

It has been reported that behavior of experts and novices in various sporting tasks is impervious to the introduction of blur. However, studies have used diverse methods of blurring the visual stimulus (i.e., dioptric blur and Gaussian blur), and tasks that did not always preserve the normal perception-action coupling. In the current study, we developed a novel experimental protocol to examine the effect of different levels of Gaussian blur on interception performance and eye gaze data using an immersive VR task. Importantly, this provided a realistic simulation of a real-world boxing scenario (e.g., the presence of a feint prior to the onset of different combinations of punches) in which expert combat athletes (n = 18) experienced a first-person, adaptive viewpoint of the visual environment, which could be blurred according to their gaze location (central blur, peripheral blur, no blur). We found that participants exhibited similar interception performance in the presence of central blur or peripheral blur compared to a control condition with no blur. However, interception performance was significantly better with a central blur compared to peripheral blur. Eye gaze data indicated that although participants fixated at similar areas of interest irrespective of the presence of blur, fixation duration was significantly longer with a strong level of blur in the peripheral viewing condition than all levels of central blur and the control condition. These findings can be explained by relocating attention to different areas of the environment, which thereby influenced the perception of salient information. Participants also performed better on the first punch of a sequence preceded by a foot feint compared to arm feint or no feint. Still, irrespective of feint type, performance was significantly better on the second and third punch compared to the first punch. These findings are consistent with participants using additional information from the opponent's body movements and situational probabilities to increase performance as the sequence of punches developed. Overall, these are the first evidence for the use of VR as a means to examine gaze-contingent manipulations of the environment, and hence highlight the potential for facilitating learning and transfer to a real sporting situations.

Effects of data preprocessing on detecting autism in adults using web-based eye-tracking data

ABSTRACT Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder, often associated with social and communication challenges and whose prevalence has increased significantly over the past two decades. The variety of different manifestations of ASD makes the condition difficult to diagnose, especially in the case of highly independent adults. A large body of work is dedicated to developing new and improved diagnostic techniques, emphasising approaches that rely on objective markers. One such paradigm is investigating eye-tracking data as a promising and objective method to capture attention-related differences between people with and without autism. This study builds upon prior work in this area that focussed on developing a machine-learning classifier trained on gaze data from web-related tasks to detect ASD in adults. Using the same data, we show that a new data pre-processing approach, combined with an exploration of the performance of different classification algorithms, leads to an increased classification accuracy compared to prior work. The proposed approach to data pre-processing is stimulus-independent, suggesting that the improvements in performance shown in these experiments can potentially generalise over other studies that use eye-tracking data for predictive purposes.

Evaluation of e-learners' concentration using recurrent neural networks.

Recently, interest in e-learning has increased rapidly owing to the lockdowns imposed by COVID-19. A major disadvantage of e-learning is the difficulty in maintaining concentration because of the limited interaction between teachers and students. The objective of this paper is to develop a methodology to predict e-learners’ concentration by applying recurrent neural network models to eye gaze and facial landmark data extracted from e-learners’ video data. One hundred eighty-four video data of ninety-two e-learners were obtained, and their frame data were extracted using the OpenFace 2.0 toolkit. Recurrent neural networks, long short-term memory, and gated recurrent units were utilized to predict the concentration of e-learners. A set of comparative experiments was conducted. As a result, gated recurrent units exhibited the best performance. The main contribution of this paper is to present a methodology to predict e-learners’ concentration in a natural e-learning environment.

User Identification Utilizing Minimal Eye-Gaze Features in Virtual Reality Applications

Emerging Virtual Reality (VR) displays with embedded eye trackers are currently becoming a commodity hardware (e.g., HTC Vive Pro Eye). Eye-tracking data can be utilized for several purposes, including gaze monitoring, privacy protection, and user authentication/identification. Identifying users is an integral part of many applications due to security and privacy concerns. In this paper, we explore methods and eye-tracking features that can be used to identify users. Prior VR researchers explored machine learning on motion-based data (such as body motion, head tracking, eye tracking, and hand tracking data) to identify users. Such systems usually require an explicit VR task and many features to train the machine learning model for user identification. We propose a system to identify users utilizing minimal eye-gaze-based features without designing any identification-specific tasks. We collected gaze data from an educational VR application and tested our system with two machine learning (ML) models, random forest (RF) and k-nearest-neighbors (kNN), and two deep learning (DL) models: convolutional neural networks (CNN) and long short-term memory (LSTM). Our results show that ML and DL models could identify users with over 98% accuracy with only six simple eye-gaze features. We discuss our results, their implications on security and privacy, and the limitations of our work.