Human Eye Movements Research Articles

Intelligent Human Operator Mental Fatigue Assessment Method Based on Gaze Movement Monitoring.

Modern mental fatigue detection methods include many parameters for evaluation. For example, many researchers use human subjective evaluation or driving parameters to assess this human condition. Development of a method for detecting the functional state of mental fatigue is an extremely important task. Despite the fact that human operator support systems are becoming more and more widespread, at the moment there is no open-source solution that can monitor this human state based on eye movement monitoring in real time and with high accuracy. Such a method allows the prevention of a large number of potential hazardous situations and accidents in critical industries (nuclear stations, transport systems, and air traffic control). This paper describes the developed method for mental fatigue detection based on human eye movements. We based our research on a developed earlier dataset that included captured eye-tracking data of human operators that implemented different tasks during the day. In the scope of the developed method, we propose a technique for the determination of the most relevant gaze characteristics for mental fatigue state detection. The developed method includes the following machine learning techniques for human state classification: random forest, decision tree, and multilayered perceptron. The experimental results showed that the most relevant characteristics are as follows: average velocity within the fixation area; average curvature of the gaze trajectory; minimum curvature of the gaze trajectory; minimum saccade length; percentage of fixations shorter than 150 ms; and proportion of time spent in fixations shorter than 150 milliseconds. The processing of eye movement data using the proposed method is performed in real time, with the maximum accuracy (0.85) and F1-score (0.80) reached using the random forest method.

Language models outperform cloze predictability in a cognitive model of reading.

Although word predictability is commonly considered an important factor in reading, sophisticated accounts of predictability in theories of reading are lacking. Computational models of reading traditionally use cloze norming as a proxy of word predictability, but what cloze norms precisely capture remains unclear. This study investigates whether large language models (LLMs) can fill this gap. Contextual predictions are implemented via a novel parallel-graded mechanism, where all predicted words at a given position are pre-activated as a function of contextual certainty, which varies dynamically as text processing unfolds. Through reading simulations with OB1-reader, a cognitive model of word recognition and eye-movement control in reading, we compare the model's fit to eye-movement data when using predictability values derived from a cloze task against those derived from LLMs (GPT-2 and LLaMA). Root Mean Square Error between simulated and human eye movements indicates that LLM predictability provides a better fit than cloze. This is the first study to use LLMs to augment a cognitive model of reading with higher-order language processing while proposing a mechanism on the interplay between word predictability and eye movements.

Humanoid robot eyeballs driven by bubble artificial muscles

Currently, service robots play an increasingly significant role in human-computer interaction, and have their intelligent, humanized, emotional, and expressive capabilities have constantly improved. Eye movement is essential for humanoid service robot facial expression, which may significantly improve human-computer connection. This project aims to create bionic robot eyes powered by novel bubble artificial muscles (BAMs), resulting in low-cost and straightforward robot eyes with horizontal, vertical, and mixed ocular movement capabilities. By adjusting the air pressure input to the BAMs, an antagonistic mechanism composed of a BAM and a tension spring is meant to move an eyeball horizontally or vertically. The results of an experiment reveal that humanoid robot eyeballs can imitate human eye movement. This research will contribute to the development of a low-weight, low-cost humanoid service robot eyeball drive system that can realise the expression of robot facial expression by controlling the movement of robot eyeball, thereby enabling humanoid servers to effectively express their feelings and emotions in accordance with human needs and enhancing user interaction.

Perception-action Dissociations as a Window into Consciousness.

Understanding the neural correlates of unconscious perception stands as a primary goal of experimental research in cognitive psychology and neuroscience. In this Perspectives paper, we explain why experimental protocols probing qualitative dissociations between perception and action provide valuable insights into conscious and unconscious processing, along with their corresponding neural correlates. We present research that utilizes human eye movements as a sensitive indicator of unconscious visual processing. Given the increasing reliance on oculomotor and pupillary responses in consciousness research, these dissociations also provide a cautionary tale about inferring conscious perception solely based on no-report protocols.

Performance of antisaccades in patients with cerebral small vessel disease accompanied by white matter hyperintensities

ABSTRACT Objectives The antisaccades (AS) task is considered a reliable indicator of inhibitory control of eye movements in humans. Achieving good AS performance requires efficient cognitive processes that are sensitive to changes in brain structure. White matter hyperintensities (WMH) can cause subcortical-cortical dysconnectivity, affecting diverse cognitive domains. Thus, the AS task was investigated in patients with WMH in cerebral small vessel disease (CSVD). Methods In this retrospective study, 75 participants with WMH, determined by neuroimaging standards for CSVD research, were admitted to the Department of Neurology of Beijing Luhe Hospital, Capital Medical University from January 2021 to December 2022. All subjects underwent the AS task, Montreal Cognitive Assessment (MoCA), Mini-Mental State Examination (MMSE), and 3.0T brain MRI. Additionally, 61 healthy subjects were recruited to characterize WMH profiles. Results Compared to the control group, patients with WMH had a significantly increased AS error rate (49.81%, p = 0.001) and lower gain (76.00%, p = 0.042). The AS error rate was significantly higher in patients with WMH in the frontal lobe than in those without WMH (p = 0.004). After adjusting for confounders (age), a positive correlation was found between the AS error rate and MoCA scores for patients with WMH (coefficient = 0.262, p = 0.024). Conclusions Patients with WMH due to CSVD exhibited abnormal AS performances, particularly in the frontal lobe. The eye movement paradigms, the new diagnostic forms in neurology, can be utilized to investigate the distributed cortical and subcortical systems involved in cognitive control processes, offering simple, well-tolerated and highly sensitive advantages over traditional measures.

A review of machine learning in scanpath analysis for passive gaze-based interaction.

The scanpath is an important concept in eye tracking. It refers to a person's eye movements over a period of time, commonly represented as a series of alternating fixations and saccades. Machine learning has been increasingly used for the automatic interpretation of scanpaths over the past few years, particularly in research on passive gaze-based interaction, i.e., interfaces that implicitly observe and interpret human eye movements, with the goal of improving the interaction. This literature review investigates research on machine learning applications in scanpath analysis for passive gaze-based interaction between 2012 and 2022, starting from 2,425 publications and focussing on 77 publications. We provide insights on research domains and common learning tasks in passive gaze-based interaction and present common machine learning practices from data collection and preparation to model selection and evaluation. We discuss commonly followed practices and identify gaps and challenges, especially concerning emerging machine learning topics, to guide future research in the field.

Novel Electrophysiological Signatures of Learning and Forgetting in Human Rapid Eye Movement Sleep.

Despite the known behavioral benefits of rapid eye movement (REM) sleep, discrete neural oscillatory events in human scalp electroencephalography (EEG) linked with behavior have not been discovered. This knowledge gap hinders mechanistic understanding of the function of sleep, as well as the development of biophysical models and REM-based causal interventions. We designed a detection algorithm to identify bursts of activity in high-density, scalp EEG within theta (4-8 Hz) and alpha (8-13 Hz) bands during REM sleep. Across 38 nights of sleep, we characterized the burst events (i.e., count, duration, density, peak frequency, amplitude) in healthy, young male and female human participants (38; 21F) and investigated burst activity in relation to sleep-dependent memory tasks: hippocampal-dependent episodic verbal memory and nonhippocampal visual perceptual learning. We found greater burst count during the more REM-intensive second half of the night (p < 0.05), longer burst duration during the first half of the night (p < 0.05), but no differences across the night in density or power (p > 0.05). Moreover, increased alpha burst power was associated with increased overnight forgetting for episodic memory (p < 0.05). Furthermore, we show that increased REM theta burst activity in retinotopically specific regions was associated with better visual perceptual performance. Our work provides a critical bridge between discrete REM sleep events in human scalp EEG that support cognitive processes and the identification of similar activity patterns in animal models that allow for further mechanistic characterization.

Electrophysiological Mechanisms of Memory Consolidation in Human Non-rapid Eye Movement Sleep

Abstract Purpose of Review Pioneering work in rodents has shown that the reactivation of recently acquired memories during sleep is a key mechanism underlying the beneficial effect of sleep on memory consolidation. In this review, we consider recent evidence of memory reactivation processes in human sleep. Recent Findings The precise temporal coupling of sleep spindles to slow oscillations during non-rapid eye movement sleep plays a central role in sleep-associated memory consolidation. Both correlational studies and studies directly manipulating oscillatory activity in the sleeping brain have confirmed that spindles coupled to slow oscillations are better predictors of memory than uncoupled spindles and that the greatest memory benefit comes when spindles are tightly coupled to the up-state of the slow oscillation. Recent evidence suggests that memory content is reactivated during sleep, with a functional benefit for memory performance after sleep. Reactivation events are time-locked around slow oscillation-spindle coupling events, as well as sharp-wave ripples in hippocampus. Summary Memory reactivation, which is facilitated by slow oscillation-spindle coupling events, can be observed during human sleep and shows promise as a prime mechanism underlying sleep’s beneficial effects on memory.

Finite element model of ocular adduction with unconstrained globe translation

Details of the anatomy and behavior of the structures responsible for human eye movements have been extensively elaborated since the first modern biomechanical models were introduced. Based on these findings, a finite element model of human ocular adduction is developed based on connective anatomy and measured optic nerve (ON) properties, as well as active contractility of bilaminar extraocular muscles (EOMs), but incorporating the novel feature that globe translation is not otherwise constrained so that realistic kinematics can be simulated. Anatomy of the hemisymmetric model is defined by magnetic resonance imaging. The globe is modeled as suspended by anatomically realistic connective tissues, orbital fat, and contiguous ON. The model incorporates a material subroutine that implements active EOM contraction based on fiber twitch characteristics. Starting from the initial condition of 26° adduction, the medial rectus (MR) muscle was commanded to contract as the lateral rectus (LR) relaxed. We alternatively modeled absence or presence of orbital fat. During pursuit-like adduction from 26 to 32°, the globe translated 0.52 mm posteriorly and 0.1 mm medially with orbital fat present, but 1.2 mm posteriorly and 0.1 mm medially without fat. Maximum principal strains in the optic disk and peripapillary reached 0.05–0.06, and von-Mises stress 96 kPa. Tension in the MR orbital layer was ~ 24 g-force after 6° adduction, but only ~ 3 gm-f in the whole LR. This physiologically plausible simulation of EOM activation in an anatomically realistic globe suspensory system demonstrates that orbital connective tissues and fat are integral to the biomechanics of adduction, including loading by the ON.

Integrating visual language with architecture: Using VAS software to measure the coherence of pre-attentive vision

Visual language has evolved as a result of technological development over ages. The pre-attentive visual software (VAS) is a biometric technique that is used to simulate human eye movement, gaze sequence, and visual appeal at the first (3-5) seconds of responding. Hence, the pre-attentive response is the first objective phase that is shared by all. The second phase of human responses is related to the perception stage depending upon personal and subjective preferences. Post-perception represents the third phase of responding after evaluation and judgment. While the Gestalt theory focuses on the perception phase, we find that the (15) properties of wholeness for Christopher Alexander deal with the post-perception phase. It is claimed that pre-attentive vision, produced by (VAS), is the actual way to perceive the coherent whole. The experimental approach of testifying different, but unified and neutralized, images of residential facades by (VAS) shows that architectural elements and their visual relations act as a coherent whole that attract our attention. The research goals are: promote the idea of objective pre-attentive vision and its attractiveness. also It is a trail to predict the coherence of designs. Accordingly, it is suggested to use (VAS) as a sensitive, detective and reliable tool in architectural design, artificial intelligence, neurosciences technology, plastic surgeries… Further investigations in the field of architecture are recommended to develop a new language of visual structures.

Design, Implementation, and Observer-based Output Control of a Super-coiled Polymer-Driven Two Degree-of-Freedom Robotic Eye.

The prevalence of ineffective corrective surgeries for ocular motor disorders calls for a robotic eye platform in aiding ophthalmologists to better understand the biomechanisms of human eye movement. This letter presents the first hardware design and implementation of a 2-DOF robotic eye driven by super-coiled polymer (SCP) artificial muscles. While our previous work designed and simulated a deep deterministic policy gradient (DDPG) learning-based controller that requires full-state feedback of the SCP-driven robotic eye, measuring the temperature states of the slender SCPs is generally impractical for the ubiquitously aimed robot. To address this predicament, this letter proposes a reduced-order state observer to estimate the temperature of SCPs given the kinematic measurements. Combining the designed observer and the learning-based controller, the closed-loop output feedback control is implemented on the robotic eye prototype to examine its performance on three classical types of eye movements: visual fixation, saccadic pursuit, and smooth pursuit. The experimental results are presented which successfully validate the observer-based output control of the SCP-driven robotic eye.

A non-iterative foveated single-pixel imaging using fast transformation algorithm

Inspired by the human eye retina, foveated single-pixel imaging employs an iterative algorithm, which, unfortunately, poses a challenge for real-time imaging due to its time-consuming nature. To address this issue, we present a non-iterative foveated single-pixel imaging method based on a computationally fast algorithm, reducing time consumption by five orders of magnitude. The fast algorithmic transform facilitates the acquisition of foveated reconstructions. In addition to mimicking human saccadic eye movements, the foveated strategy adapts to motion tracking through the foveal gaze control method. During high-resolution imaging in the fovea area, the maximum sensing area expands 3.14 times, similar to human vision. The proposed retina-like single-pixel imaging has great potential in future applications with the advantages of avoiding complex optics design, low time consumption, and wider vision field.

The influence of stereopsis on visual saliency in a proto-object based model of selective attention

Some animals including humans use stereoscopic vision which reconstructs spatial information about the environment from the disparity between images captured by eyes in two separate adjacent locations. Like other sensory information, such stereoscopic information is expected to influence attentional selection. We develop a biologically plausible model of binocular vision to study its effect on bottom-up visual attention, i.e., visual saliency. In our model, the scene is organized in terms of proto-objects on which attention acts, rather than on unbound sets of elementary features. We show that taking into account the stereoscopic information improves the performance of the model in the prediction of human eye movements with statistically significant differences.

A 2D Synthesized Image Improves the 3D Search for Foveated Visual Systems.

Current medical imaging increasingly relies on 3D volumetric data making it difficult for radiologists to thoroughly search all regions of the volume. In some applications (e.g., Digital Breast Tomosynthesis), the volumetric data is typically paired with a synthesized 2D image (2D-S) generated from the corresponding 3D volume. We investigate how this image pairing affects the search for spatially large and small signals. Observers searched for these signals in 3D volumes, 2D-S images, and while viewing both. We hypothesize that lower spatial acuity in the observers' visual periphery hinders the search for the small signals in the 3D images. However, the inclusion of the 2D-S guides eye movements to suspicious locations, improving the observer's ability to find the signals in 3D. Behavioral results show that the 2D-S, used as an adjunct to the volumetric data, improves the localization and detection of the small (but not large) signal compared to 3D alone. There is a concomitant reduction in search errors as well. To understand this process at a computational level, we implement a Foveated Search Model (FSM) that executes human eye movements and then processes points in the image with varying spatial detail based on their eccentricity from fixations. The FSM predicts human performance for both signals and captures the reduction in search errors when the 2D-S supplements the 3D search. Our experimental and modeling results delineate the utility of 2D-S in 3D search-reduce the detrimental impact of low-resolution peripheral processing by guiding attention to regions of interest, effectively reducing errors.

Expertise increases planning depth in human gameplay

A hallmark of human intelligence is the ability to plan multiple steps into the future1,2. Despite decades of research3-5, it is still debated whether skilled decision-makers plan more steps ahead than novices6-8. Traditionally, the study of expertise in planning has used board games such as chess, but the complexity of these games poses a barrier to quantitative estimates of planning depth. Conversely, common planning tasks in cognitive science often have a lower complexity9,10 and impose a ceiling for the depth to which any player can plan. Here we investigate expertise in a complex board game that offers ample opportunity for skilled players to plan deeply. We use model fitting methods to show that human behaviour can be captured using a computational cognitive model based on heuristic search. To validate this model, we predict human choices, response times and eye movements. We also perform a Turing test and a reconstruction experiment. Using the model, we find robust evidence for increased planning depth with expertise in both laboratory and large-scale mobile data. Experts memorize and reconstruct board features more accurately. Using complex tasks combined with precise behavioural modelling might expand our understanding of human planning and help to bridge the gap with progress in artificial intelligence.

An active foveated gaze prediction algorithm based on a Bayesian ideal observer

Predicting human eye movements is a crucial task for understanding human behavior and has numerous applications in machine vision. Most current models for predicting eye movements are data-driven and require large datasets of recorded eye movements, which can be expensive and time-consuming to collect. In this paper, we present a novel theory-based model for predicting eye movements in a foveated visual system that maximizes information gain at each fixation. Our model uses a region-proposal network and eccentricity-based max pooling to account for the loss of detail in peripheral vision. We apply our model to predict human fixations in a visual search task for objects in real-world scenes. Unlike data-driven models, our model does not require training on large eye movement datasets and can generalize to any set of natural images and targets. We evaluate the generalization capability of our model by demonstrating its results on two publicly available visual search datasets, Ehinger and COCO-search18, without any further training on those datasets. Our model outperforms or performs comparably to data-driven models that are directly trained on human eye movement datasets.

Fast and Slow Rhythms of Naturalistic Reading Revealed by Combined Eye-Tracking and Electroencephalography.

Neural oscillations are thought to support speech and language processing. They may not only inherit acoustic rhythms, but might also impose endogenous rhythms onto processing. In support of this, we here report that human (both, male and female) eye movements during naturalistic reading exhibit rhythmic patterns that show frequency-selective coherence with the electroencephalogram (EEG)-in the absence of any stimulation rhythm. Periodicity was observed in two distinct frequency bands: First, word-locked saccades at 4-5 Hertz display coherence with whole-head theta-band activity. Second, fixation durations fluctuate rhythmically at ∼1 Hertz, in coherence with occipital delta-band activity. This latter effect was additionally phase-locked to sentence endings, suggesting a relationship with the formation of multi-word chunks. Taken together, eye movements during reading contain rhythmic patterns that occur in synchrony with oscillatory brain activity. This suggests that linguistic processing imposes preferred processing time scales onto reading, largely independent of actual physical rhythms in the stimulus.SIGNIFICANCE STATEMENT:The sampling, grouping, and transmission of information is supported by rhythmic brain activity, so-called neural oscillations. In addition to sampling external stimuli, such rhythms may also be endogenous, affecting processing from the inside out. In particular, endogenous rhythms may impose their pace onto language processing. Studying this is challenging because speech contains physical rhythms that mask endogenous activity. To overcome this challenge, we turned to naturalistic reading, where text does not require the reader to sample in a specific rhythm. We observed rhythmic patterns of eye movements that are synchronized to brain activity as recorded with EEG. This rhythmicity is not imposed by the external stimulus, which indicates that rhythmic brain activity may serve as a pacemaker for language processing.

A frontoparietal network for volitional control of gaze following.

Gaze following is a major element of non-verbal communication and important for successful social interactions. Human gaze following is a fast and almost reflex-like behavior, yet, it can be volitionally controlled and suppressed to some extent if inappropriate or unnecessary, given the social context. In order to identify the neural basis of the cognitive control of gaze following, we carried out an event-related fMRI experiment, in which human subjects' eye movements were tracked while they were exposed to gaze cues in two distinct contexts: a baseline gaze following condition in which subjects were instructed to use gaze cues to shift their attention to a gazed-at spatial target and a control condition in which the subjects were required to ignore the gaze cue and instead to shift their attention to a distinct spatial target to be selected based on a color mapping rule, requiring the suppression of gaze following. We could identify a suppression-related BOLD response in a frontoparietal network comprising dorsolateral prefrontal cortex (dlPFC), orbitofrontal cortex (OFC), the anterior insula, precuneus and posterior parietal cortex (PPC). These findings suggest that over excitation of frontoparietal circuits in turn suppressing the gaze following patch might be a potential cause of gaze following deficits in clinical populations.

EHTask: Recognizing User Tasks From Eye and Head Movements in Immersive Virtual Reality.

Understanding human visual attention in immersive virtual reality (VR) is crucial for many important applications, including gaze prediction, gaze guidance, and gaze-contingent rendering. However, previous works on visual attention analysis typically only explored one specific VR task and paid less attention to the differences between different tasks. Moreover, existing task recognition methods typically focused on 2D viewing conditions and only explored the effectiveness of human eye movements. We first collect eye and head movements of 30 participants performing four tasks, i.e., Free viewing, Visual search, Saliency, and Track, in 15 360-degree VR videos. Using this dataset, we analyze the patterns of human eye and head movements and reveal significant differences across different tasks in terms of fixation duration, saccade amplitude, head rotation velocity, and eye-head coordination. We then propose EHTask - a novel learning-based method that employs eye and head movements to recognize user tasks in VR. We show that our method significantly outperforms the state-of-the-art methods derived from 2D viewing conditions both on our dataset (accuracy of 84.4% versus 62.8%) and on a real-world dataset ( 61.9% versus 44.1%). As such, our work provides meaningful insights into human visual attention under different VR tasks and guides future work on recognizing user tasks in VR.

Artifact reduction in lenslet array near-eye displays

Lenslet array near-eye displays are a revolutionary technology that generates a virtual image in the field of view of the observer. Although this technology is advantageous in creating compact near-eye displays, undesirable artifacts occur when the user pupil moves outside of the pupil practical movable region (PPMR). Even with dynamic image updating based on eye-tracking techniques, artifacts can still be perceived when human eyes turn rapidly. To enlarge PPMR, we proposed a new rendering method in previous work. To improve the rendering speed in the eye tracking system, look-up tables are used. The disadvantage of the onboard system is the large memory consumption. In this study, we analyzed the system parameters of the incident pupil and pupil margin light columns, the feasibility of the optimized system, and evaluated the optimized system that can adapt to the maximum velocity of the saccadic pupil movement. We optimized the rendering method to reduce memory consumption in the process of generating microdisplay images. In addition, we provide GPU rendering method to improve system speed and reduce system latency to meet the maximum human eye rotation speed. We conducted user studies to evaluate the effect of the method using the optimized rendering method combined with eye tracking to reduce artifacts for fast eye rotation on different images and videos. Results showed that our method effectively reduced artifacts via the optimized rendering method with eye tracking, which adapted to faster human eye movements.