Eye Trajectories Research Articles

Assessing the Link between Nurses' Proficiency and Situational Awareness in Neonatal Care Practice Using an Eye Tracker: An Observational Study Using a Simulator.

Nurses are expected to depend on a wide variety of visually available pieces of patient information to understand situations. Thus, we assumed a relationship between nurses' skills and their gaze trajectories. An observational study using a simulator was conducted to analyze gaze during neonatal care practice using eye tracking. We defined the face, thorax, and abdomen of the neonate, the timer, and the pulse oximeter as areas of interest (AOIs). We compared the eye trajectories for respiration and heart rate assessment between 7 experienced and 13 novice nurses. There were no statistically significant differences in the time spent on each AOI for breathing or heart rate confirmation. However, in novice nurses, we observed a significantly higher number of instances of gazing at the thorax and abdomen. The deviation in the number of instances of gazing at the face was also significantly higher among novice nurses. These results indicate that experienced and novice nurses differ in their gaze movements during situational awareness. These objective and quantitative differences in gaze trajectories may help to establish new educational tools for less experienced nurses.

Poster Session I: LissEYEjous Tracker - precise fundus tracking device based on ultrafast Lissajous scanning.

Retinal eye tracking has emerged as a promising alternative to conventional video-based trackers, offering direct access to retinal coordinates with refined spatial and temporal resolutions. These attributes make them attractive for applications ranging from image stabilization in advanced ophthalmic imaging to identifying biomarkers of neurological or ophthalmic disorders that affect eye motility. Existing retinal tracking method however face challenges related to reliance on reference frames and non-uniform sampling either in space or time. In this work we present a new approach for retinal tracking, which is based on imaging small retinal patches (~1.5-3°) using self-repeating Lissajous scanning patterns. Pattern repetition rates close to 4kHz are achieved with an optical design that employs two MEMS microscanners with closely matching resonant frequencies, working in mutually perpendicular dimensions. Several examples of fundus images acquired with different Lissajous patterns are presented. Based on this, eye trajectories may be extracted. Future works will further investigate tracking resolution and dependence on Lissajous pattern spatial density and repetition rate.

A Comprehensive Situation Awareness Measurement Method for Analyzing the Operators’ Situation Awareness of Multi-Module High Temperature Gas-Cooled Reactor Plants

For multi-modular nuclear power plants under the scheme of multiple nuclear steam supply system (NSSS) modules driving a single steam turbine, the NSSS modules are coupled tightly with each other by the common turbine, giving more complex normal operation with respect to the single-modular plants. To limit the operation cost of multi-modular plants, one operator is assigned to monitor and control two or more modules, whose feasibility should be verified. Combined with the characteristics of multi-module control rooms and multi-module running tasks, this paper designs a comprehensive situation awareness measurement method that combines SART, NASA-TLX (NASA Task Load Index), and eye movement tracking methods. The SART, NASA-TLX, and gaze entropy are adopted to measure the operators’ SA, and a series of accident handling experiments are performed on a full-scale simulator to gain enough data for analysis. The operators’ eye trajectories on the human–machine interface (HMI) during the experiments are all recorded for calculating the gaze entropy. Both the SART and NASA-TLX scales are filled by the operators after finishing the experiments. The experiment results show that the difference in operators’ workload and SA amongst all the experimental scenarios is limited, even between the toughest and tenderest scenarios, indicating the feasibility of one operator driving two NSSS modules simultaneously.

Machine learning algorithms on eye tracking trajectories to classify patients with spatial neglect

Background and Objective: Eye-movement trajectories are rich behavioral data, providing a window on how the brain processes information. We address the challenge of characterizing signs of visuo-spatial neglect from saccadic eye trajectories recorded in brain-damaged patients with spatial neglect as well as in healthy controls during a visual search task. Methods: We establish a standardized pre-processing pipeline adaptable to other task-based eye-tracker measurements. We use traditional machine learning algorithms together with deep convolutional networks (both 1D and 2D) to automatically analyze eye trajectories. Results: Our top-performing machine learning models classified neglect patients vs. healthy individuals with an Area Under the ROC curve (AUC) ranging from 0.83 to 0.86. Moreover, the 1D convolutional neural network scores correlated with the degree of severity of neglect behavior as estimated with standardized paper-and-pencil tests and with the integrity of white matter tracts measured from Diffusion Tensor Imaging (DTI). Interestingly, the latter showed a clear correlation with the third branch of the superior longitudinal fasciculus (SLF), especially damaged in neglect. Conclusions:The study introduces new methods for both the pre-processing and the classification of eye-movement trajectories in patients with neglect syndrome. The proposed methods can likely be applied to other types of neurological diseases opening the possibility of new computer-aided, precise, sensitive and non-invasive diagnostic tools.

Comparative Techniques Using Hierarchical Modelling and Machine Learning for Procedure Recognition in Smart Hospitals

6G is one of the key cornerstone elements of the futuristic smart system setup – the others being cloud computing, big data, wearable devices and Artificial Intelligence. Also, smart offices and homes have become even more popular than before, because of the advancement in computer vision and Machine Learning (ML) technologies. Recognition of human actions and situations are fundamental components of such systems, especially in complex environments like healthcare, for example at the dentist clinic, where we need cues such as eye movement to distinguish procedures being undertaken. In this work, we compare models based on hierarchical modelling and machine learning to identify the dental procedure. We used the objects seen while following the eye trajectories and focussed on elements including material used for treatment, equipment involved and the teeth conditions i.e. symptoms. Our experiments showed that using Artificial Neural Network (ANN) increased the accuracy of prediction compared to hierarchical modelling. Our experiments show an improvement in accuracy for each of the constituent parameters i.e., symptom (ANN: 95.58% vs. Hierarchical: 45.68%), material (ANN: 86.32% vs. Hierarchical: 45.18%) and equipment (ANN: 92.65% vs. Hierarchical: 59.39%).

Identifying physiological correlates of cybersickness using heartbeat-evoked potential analysis

Many studies have consistently proven that repeatedly watching virtual reality (VR) content can reduce cybersickness. Moreover, the discomfort level decreases when the VR content includes an unusual orientation, such as an inverted scene. However, few studies have investigated the physiological changes during these experiences. The present study aimed to identify psychophysiological correlates, especially the neural processing, of cybersickness. Twenty participants experienced two types of VR orientation (upright and inverted), which were repeated three times. During the experience, we recorded the participants’ subjective levels of discomfort, brain waves, cardiac signals, and eye trajectories. We performed a heartbeat-evoked potential (HEP) analysis to elucidate the cortical activity of heartbeats while experiencing cybersickness. The results showed that the severity of cybersickness decreased as the participants repeatedly watched the VR content. The participants also reported less nausea when watching the inverted orientation. We only found a significant suppression at the fronto-central HEP amplitudes in the upright orientation for the physiological changes. This study provides a comprehensive understanding of bodily responses to varying degrees of cybersickness. In addition, the HEP results suggest that this approach might reflect the neural correlates of transient changes in heartbeats caused by cybersickness.

Irrelevant Robot Signals in a Categorization Task Induce Cognitive Conflict in Performance, Eye Trajectories, the N2 Component of the EEG Signal, and Frontal Theta Oscillations.

Understanding others' nonverbal behavior is essential for social interaction, as it allows, among others, to infer mental states. Although gaze communication, a well-established nonverbal social behavior, has shown its importance in inferring others' mental states, not much is known about the effects of irrelevant gaze signals on cognitive conflict markers during collaborative settings. In the present study, participants completed a categorization task where they categorized objects based on their color while observing images of a robot. On each trial, participants observed the robot iCub grasping an object from a table and offering it to them to simulate a handover. Once the robot "moved" the object forward, participants were asked to categorize the object according to its color. Before participants were allowed to respond, the robot made a lateral head/gaze shift. The gaze shifts were either congruent or incongruent with the object's color. We expected that incongruent head cues would induce more errors (Study 1), would be associated with more curvature in eye-tracking trajectories (Study 2), and induce larger amplitude in electrophysiological markers of cognitive conflict (Study 3). Results of the three studies show more oculomotor interference as measured in error rates (Study 1), larger curvatures eye-tracking trajectories (Study 2), and higher amplitudes of the N2 ERP component of the EEG signals as well as higher event-related spectral perturbation amplitudes (Study 3) for incongruent trials compared with congruent trials. Our findings reveal that behavioral, ocular, and electrophysiological markers can index the influence of irrelevant signals during goal-oriented tasks.

Video-based eye tracking performance for computer-assisted diagnostic support of diabetic neuropathy

Diabetes is currently one of the major public health threats. The essential components for effective treatment of diabetes include early diagnosis and regular monitoring. However, health-care providers are often short of human resources to closely monitor populations at risk. In this work, a video-based eye-tracking method is proposed as a low-cost alternative for detection of diabetic neuropathy. The method is based on the tracking of the eye-trajectories recorded on videos while the subject follows a target on a screen, forcing saccadic movements. Upon extraction of the eye trajectories, representation of the obtained time-series is made with the help of heteroscedastic ARX (H-ARX) models, which capture the dynamics and latency on the subject's response, while features based on the H-ARX model's predictive ability are subsequently used for classification. The methodology is evaluated on a population constituted by 11 control and 20 insulin-treated diabetic individuals suffering from diverse diabetic complications including neuropathy and retinopathy. Results show significant differences on latency and eye movement precision between the populations of control subjects and diabetics, while simultaneously demonstrating that both groups can be classified with an accuracy of 95%. Although this study is limited by the small sample size, the results align with other findings in the literature and encourage further research.

Optimal time task in saccadic eye movement

This article is devoted to fast goal-directed human eye movements. Such movements are described in few ways. One of them is the time-optimal problem model. It this work, the saccade – fast single conformable ballistic eye movement, is under consideration. Saccadic eye trajectory is described by time-optimal task solution. Differential equations system are based on pendulum model. It defines control moment applied from extraocular muscles to eye globe and movement of it. Main feature of the system is all values in suggested system have physical meaning and were found in published experimental investigations. Consideration of the moment applied from extraocular muscles to eye glob as control is main difference of the system from other models for eye movement in submitted works. In this case, the open-loop system has two real roots and one zero. The particular case of the ratio of real roots of 1:3 is analyzed in the article. As a result of modelling, trajectory characteristics closed to the saccades observed in experimental studies is obtained. On the basis of the constructed model, the parameters of the synthesized saccades were compared with the parameters obtained on the experimental sample. The obtained models can be used both for problems of physiology and medicine, and for constructing virtual reality environments. For example, when developing 3D interfaces, the use of a mathematical model makes it possible to optimize the elements arrangement using game theory.

Motor representations underlie the reading of unfamiliar letter combinations

Silent reading is a cognitive operation that produces verbal content with no vocal output. One relevant question is the extent to which this verbal content is processed as overt speech in the brain. To address this, we acquired sound, eye trajectories and lips’ dynamics during the reading of consonant-consonant-vowel (CCV) combinations which are infrequent in the language. We found that the duration of the first fixations on the CCVs during silent reading correlate with the duration of the transitions between consonants when the CCVs are actually uttered. With the aid of an articulatory model of the vocal system, we show that transitions measure the articulatory effort required to produce the CCVs. This means that first fixations during silent reading are lengthened when the CCVs require a greater laryngeal and/or articulatory effort to be pronounced. Our results support that a speech motor code is used for the recognition of infrequent text strings during silent reading.

Spatiotemporal Coupling between Eye and Hand Trajectories during Curved Hand Movements

Eye and hand movements are often made in isolation but for reaching movements they are usually coupled. Despite this, evidence for spatial coupling between the eye and hand effector is mixed and have usually been restricted to straight-line movements, while real-world hand movements have complex trajectories. Here, using a novel obstacle avoidance task where an obstacle appeared in an infrequent number of trials, we establish a stronger link between the saccade and hand trajectory during more naturalistic curved hand trajectories. We illustrate that the hand trajectory was coupled to the end-point of the saccade which was executed just prior to the hand movement onset. Interestingly, while the saccade end-point was related to whether the hand trajectory followed a straight or a curved path, the y-component of saccade end-point was related to whether the hand took a path passing from over or below the obstacle. Further, we observed a relationship between saccade locations and hand sub-movements where the number and timing of saccades and number of hand velocity peaks were related. These results illustrate a robust spatiotemporal and kinematic coupling between saccades and complex hand movement trajectories suggesting a shared kinematic representation underlying eye-hand movements.

Transfer function of the rhesus macaque oculomotor system for small-amplitude slow motion trajectories.

Two main types of small eye movements occur during gaze fixation: microsaccades and slow ocular drifts. While microsaccade generation has been relatively well studied, ocular drift control mechanisms are unknown. Here we explored the degree to which monkey smooth eye movements, on the velocity scale of slow ocular drifts, can be generated systematically. Two male rhesus macaque monkeys tracked a spot moving sinusoidally, but slowly, along the horizontal or vertical direction. Maximum target displacement in the motion trajectory was 30 min arc (0.5°), and we varied the temporal frequency of target motion from 0.2 to 5 Hz. We obtained an oculomotor "transfer function" by measuring smooth eye velocity gain (relative to target velocity) as a function of frequency, similar to past work with large-amplitude pursuit. Monkey eye velocities as slow as those observed during slow ocular drifts were clearly target motion driven. Moreover, as with large-amplitude smooth pursuit, eye velocity gain varied with temporal frequency. However, unlike with large-amplitude pursuit, exhibiting low-pass behavior, small-amplitude motion tracking was band pass, with the best ocular movement gain occurring at ~0.8-1 Hz. When oblique directions were tested, we found that the horizontal component of pursuit gain was larger than the vertical component. Our results provide a catalog of the control abilities of the monkey oculomotor system for slow target motions, and they also support the notion that smooth fixational ocular drifts are controllable. This has implications for neural investigations of drift control and the image-motion consequences of drifts on visual coding in early visual areas. NEW & NOTEWORTHY We studied the efficacy of monkey smooth pursuit eye movements for very slow target velocities. Pursuit was impaired for sinusoidal motions of frequency less than ~0.8-1 Hz. Nonetheless, eye trajectory was still sinusoidally modulated, even at velocities lower than those observed during gaze fixation with slow ocular drifts. Our results characterize the slow control capabilities of the monkey oculomotor system and provide a basis for future understanding of the neural mechanisms for slow ocular drifts.

User-Centered Predictive Model for Improving Cultural Heritage Augmented Reality Applications: An HMM-Based Approach for Eye-Tracking Data

Today, museum visits are perceived as an opportunity for individuals to explore and make up their own minds. The increasing technical capabilities of Augmented Reality (AR) technology have raised audience expectations, advancing the use of mobile AR in cultural heritage (CH) settings. Hence, there is the need to define a criteria, based on users’ preference, able to drive developers and insiders toward a more conscious development of AR-based applications. Starting from previous research (performed to define a protocol for understanding the visual behaviour of subjects looking at paintings), this paper introduces a truly predictive model of the museum visitor’s visual behaviour, measured by an eye tracker. A Hidden Markov Model (HMM) approach is presented, able to predict users’ attention in front of a painting. Furthermore, this research compares users’ behaviour between adults and children, expanding the results to different kind of users, thus providing a reliable approach to eye trajectories. Tests have been conducted defining areas of interest (AOI) and observing the most visited ones, attempting the prediction of subsequent transitions between AOIs. The results demonstrate the effectiveness and suitability of our approach, with performance evaluation values that exceed 90%.

A Visual Attention Based Object Detection Model beyond Top-Down and Bottom-up Mechanism

Traditional saliency-based attention theory supposed that bottom-up and top-down factors combine to direct attentional behavior. This dichotomy fails to explain a growing number of cases in which neither bottom-up nor top-down can account for strong selection biases. Thus, the top-down versus bottom-up dichotomy is an inadequate taxonomy of attentional control. In our previous study, we presented a general computational framework for detecting task-oriented salient objects in images beyond top-down and bottom-up mechanism. It possesses three parts: selection history, current goal and physical salience. Selection history is integrated with current goal and physical salience to compose an integrative framework. In this extended version, our ameliorated model is applied to face and car detection and simulates task-dependent reasonable eye trajectories (visual scan paths). Experimental results demonstrate that selection history (reward) is shown to influence saccade trajectories. Our findings support the idea that attention and gaze can be directed voluntarily to regions of interest (by selection history and current goal) and can be captured by local features of an object that stand out from the background (by physical salience).

Eye tracking and early detection of confusion in digital learning environments: Proof of concept

Research on incidence of and changes in confusion during complex learning and problem-solving calls for advanced methods of confusion detection in digital learning environments (DLEs). In this study we attempt to address this issue by investigating the use of multiple measures, including psychophysiological indicators and self-ratings, to detect confusion in DLEs. Participants were subjected to two intrinsically confusing insight problems in the form of visual digital puzzles. They were asked to solve problems while their eye trajectories were recorded and these data were triangulated with self-ratings of confusion and cued retrospective verbal reports. All participants had a significant increase in fixations on relevant (i.e., related to the solution) and not-relevant areas at an early stage of the problem-solving process. However, only fixations on not-relevant areas were positively correlated with confusion ratings. Moreover, participants who significantly solved the problem differed in their fixations duration on relevant and not-relevant areas from non-solvers. The importance of early detection of confusion and the affordances of emerging technologies for this purpose are discussed.

Dynamic causal modelling of eye movements during pursuit: Confirming precision-encoding in V1 using MEG

This paper shows that it is possible to estimate the subjective precision (inverse variance) of Bayesian beliefs during oculomotor pursuit. Subjects viewed a sinusoidal target, with or without random fluctuations in its motion. Eye trajectories and magnetoencephalographic (MEG) data were recorded concurrently. The target was periodically occluded, such that its reappearance caused a visual evoked response field (ERF). Dynamic causal modelling (DCM) was used to fit models of eye trajectories and the ERFs. The DCM for pursuit was based on predictive coding and active inference, and predicts subjects' eye movements based on their (subjective) Bayesian beliefs about target (and eye) motion. The precisions of these hierarchical beliefs can be inferred from behavioural (pursuit) data. The DCM for MEG data used an established biophysical model of neuronal activity that includes parameters for the gain of superficial pyramidal cells, which is thought to encode precision at the neuronal level. Previous studies (using DCM of pursuit data) suggest that noisy target motion increases subjective precision at the sensory level: i.e., subjects attend more to the target's sensory attributes. We compared (noisy motion-induced) changes in the synaptic gain based on the modelling of MEG data to changes in subjective precision estimated using the pursuit data. We demonstrate that imprecise target motion increases the gain of superficial pyramidal cells in V1 (across subjects). Furthermore, increases in sensory precision – inferred by our behavioural DCM – correlate with the increase in gain in V1, across subjects. This is a step towards a fully integrated model of brain computations, cortical responses and behaviour that may provide a useful clinical tool in conditions like schizophrenia.

Persistence in eye movement during visual search.

As any cognitive task, visual search involves a number of underlying processes that cannot be directly observed and measured. In this way, the movement of the eyes certainly represents the most explicit and closest connection we can get to the inner mechanisms governing this cognitive activity. Here we show that the process of eye movement during visual search, consisting of sequences of fixations intercalated by saccades, exhibits distinctive persistent behaviors. Initially, by focusing on saccadic directions and intersaccadic angles, we disclose that the probability distributions of these measures show a clear preference of participants towards a reading-like mechanism (geometrical persistence), whose features and potential advantages for searching/foraging are discussed. We then perform a Multifractal Detrended Fluctuation Analysis (MF-DFA) over the time series of jump magnitudes in the eye trajectory and find that it exhibits a typical multifractal behavior arising from the sequential combination of saccades and fixations. By inspecting the time series composed of only fixational movements, our results reveal instead a monofractal behavior with a Hurst exponent , which indicates the presence of long-range power-law positive correlations (statistical persistence). We expect that our methodological approach can be adopted as a way to understand persistence and strategy-planning during visual search.

Refuting the hypothesis that a unilateral human parietal lesion abolishes saccade corollary discharge.

This paper questions the prominent role that the parietal lobe is thought to play in the processing of corollary discharges for saccadic eye movements. A corollary discharge copies the motor neurons' signal and sends it to brain areas involved in monitoring eye trajectories. The classic double-step saccade task has been used extensively to study these mechanisms: two targets (T1 and T2) are quickly (40-150 ms) flashed sequentially in the periphery. After the extinction of the fixation point, subjects are to make two saccades (S1 and S2), in the dark, to the remembered locations of the targets in the order they appeared. The success of S2 requires a corollary discharge encoding S1's vector. Patients with a parietal lobe lesion, particularly on the right, are impaired at generating an accurate S2 when S1 is directed contralesionally, but not ipsilesionally, thought due to an impaired contralesional corollary discharge. In contrast, we hypothesize that failure on the classic double-step task is due to visual processing and attentional deficits that commonly result from lesions of the parietal lobe and imperfect data analysis methods. Here, we studied parietal patients who fail in the classic double-step task when tested and data analysed according to previously published methods. We then tested our patients on two modified versions of the double-step task, designed to mitigate deficits other than corollary discharge that may have confounded previous investigations. In our 'exogenous' task, T2 was presented prior to T1 and for longer (T2: 800-1200 ms, T1: 350 ms) than in the classic task. S1 went to T1 and S2 to T2, all in the dark. All patients who completed sufficient trials had a corollary discharge for contralesional and ipsilesional S1s (5/5). In our 'endogenous' task, a single target was presented peripherally for 800-1200 ms. After extinction of target and fixation point, patients made first an 'endogenous' S1, of self-determined amplitude either to the left or right, before making S2 to the remembered location of the previously flashed target. To be successful, a corollary discharge of endogenous S1-generated in the dark-was required in the calculation of S2's motor vector. Every parietal patient showed evidence of using corollary discharges for endogenous S1s in the ipsilesional and contralesional directions (6/6). Our results support the hypothesis, based on our previous studies of corollary discharge mechanisms in hemidecorticate patients, and electrophysiological studies by others in monkey, that corollary discharges for left and right saccades are available to each cortical hemisphere.

Remember down, look down, read up: Does a word modulate eye trajectory away from remembered location?

Previous studies show that eye movement trajectory curves away from a remembered visual location if a saccade needs to be made in the same direction as the location. Data suggest that part of the process of maintaining the location in working memory is the mental simulation of that location, so that the oculomotor system treats the remembered location as a real one. Other research suggests that word meaning may also behave like a 'real object' in space. The current study aimed to combine the two streams of research examining the effect of word meaning on the memory of a dot location. The results of two experiments showed that word meaning for 'up' (but not 'down') modulated both eye movement trajectory and location recognition time. Thus, mental simulation of task-irrelevant space-related word meaning affected both earlier stages of memory processes (maintenance of the location in the working memory) and later ones (location recognition).

An inverse Yarbus process: predicting observers' task from eye movement patterns.

In this paper we develop a probabilistic method to infer the visual-task of a viewer given measured eye movement trajectories. This method is based on the theory of hidden Markov models (HMM) that employs a first order Markov process to predict the coordinates of fixations given the task. The prediction confidence level of each task-dependent model is used in a Bayesian inference formulation, whereby the task with the maximum a posteriori (MAP) probability is selected. We applied this technique to a challenging dataset consisting of eye movement trajectories obtained from subjects viewing monochrome images of real scenes tasked with answering questions regarding the scenes. The results show that the HMM approach, combined with a clustering technique, can be a reliable way to infer visual-task from eye movements data.