Gaze Control Model Research Articles

Drivers’ gaze patterns when resuming control with a head-up-display: Effects of automation level and time budget

The removal of drivers’ active engagement in driving tasks can lead to erratic gaze patterns in SAE Level 2 (L2) and Level 3 (L3) automation, which has been linked to their subsequential degraded take-over performance. To further address how changes in gaze patterns evolve during the take-over phase, and whether they are influenced by the take-over urgency and the location of the human-machine interface, this driving simulator study used a head-up display (HUD) to relay information about the automation status and conducted take-over driving experiments where the ego car was about to exit the highway with variations in the automation level (L2, L3) and time budget (2 s, 6 s). In L2 automation, drivers were required to monitor the environment, while in L3, they were engaged with a visual non-driving related task. Manual driving was also embodied in the experiments as the baseline. Results showed that, compared to manual driving, drivers in L2 automation focused more on the HUD and Far Road (roadway beyond 2 s time headway ahead), and less on the Near Road (roadway within 2 s time headway ahead); while in L3, drivers’ attention was predominantly allocated on the non-driving related task. After receiving take-over requests (TORs), there was a gradual diversion of attention from the Far Road to the Near Road in L2 take-overs. This trend changed nearly in proportion to the time within the time budget and it exaggerated given a shorter time budget of 2 s. While in L3, drivers’ gaze distribution was similar in the early stage of take-overs for both time budget conditions (2 s vs. 6 s), where they prioritized their early glances to Near Road with a gradual increase in attention towards Far Road. The HUD used in the present study showed the potential to maintain drivers’ attention around the road center during automation and to encourage drivers to glance the road earlier after TORs by reducing glances to the instrument cluster, which might be of significance to take-over safety. These findings were discussed based on an extended conceptual gaze control model, which advances our understanding of gaze patterns around control transitions and their underlying gaze control causations. Implications can be contributed to the design of autonomous vehicles to facilitate the transition of control by guiding drivers’ attention appropriately according to drivers’ attentional state and the take-over urgency.

Where the eyes wander: The relationship between mind wandering and fixation allocation to visually salient and semantically informative static scene content.

Vision is crucial for many everyday activities, but the mind is not always focused on what the eyes see. Mind wandering occurs frequently and is associated with attenuated visual and cognitive processing of external information. Corresponding changes in gaze behavior—namely, fewer, longer, and more dispersed fixations—suggest a shift in how the visual system samples external information. Using three computational models of visual salience and two innovative approaches for measuring semantic informativeness, the current work assessed whether these changes reflect how the visual system prioritizes visually salient and semantically informative scene content, two major determinants in most theoretical frameworks and computational models of gaze control. Findings showed that, in a static scene viewing task, fixations were allocated to scene content that was more visually salient 10 seconds prior to probe-caught, self-reported mind wandering compared to self-reported attentive viewing. The relationship between mind wandering and semantic content was more equivocal, with weaker evidence that fixations are more likely to fall on locally informative scene regions. This indicates that the visual system is still able to discriminate visually salient and semantically informative scene content during mind wandering and may fixate on such information more frequently than during attentive viewing. Theoretical implications are discussed in light of these findings.

Gaze-based signatures of mind wandering during real-world scene processing.

Physiological limitations on the visual system require gaze to move from location to location to extract the most relevant information within a scene. Therefore, gaze provides a real-time index of the information-processing priorities of the visual system. We investigated gaze allocation during mind wandering (MW), a state where cognitive priorities shift from processing task-relevant external stimuli (i.e., the visual world) to task-irrelevant internal thoughts. In both a main study and a replication, we recorded the eye movements of college-aged adults who studied images of urban scenes and responded to pseudorandom thought probes on whether they were mind wandering or attentively viewing at the time of the probe. Probe-caught MW was associated with fewer and longer fixations, greater fixation dispersion, and more frequent eyeblinks (only observed in the main study) relative to periods of attentive scene viewing. These findings demonstrate that gaze indices typically considered to represent greater engagement with scene processing (e.g., longer fixations) can also indicate MW. In this way, the current work exhibits a need for empirical investigations and computational models of gaze control to account for MW for a more accurate representation of the moment-to-moment information-processing priorities of the visual system. (PsycINFO Database Record

A predictive coding model of gaze shifts and the underlying neurophysiology

ABSTRACTA comprehensive model of gaze control must account for a number of empirical observations at both the behavioural and neurophysiological levels. The computational model presented in this article can simulate the coordinated movements of the eye, head, and body required to perform horizontal gaze shifts. In doing so it reproduces the predictable relationships between the movements performed by these different degrees of freedom (DOFs) in the primate. The model also accounts for the saccadic undershoot that accompanies large gaze shifts in the biological visual system. It can also account for our perception of a stable external world despite frequent gaze shifts and the ability to perform accurate memory-guided and double-step saccades. The proposed model also simulates peri-saccadic compression: the mis-localization of a briefly presented visual stimulus towards the location that is the target for a saccade. At the neurophysiological level, the proposed model is consistent with the existence of cortical neurons tuned to the retinal, head-centred, body-centred, and world-centred locations of visual stimuli and cortical neurons that have gain-modulated responses to visual stimuli. Finally, the model also successfully accounts for peri-saccadic receptive field (RF) remapping which results in reduced responses to stimuli in the current RF location and an increased sensitivity to stimuli appearing at the location that will be occupied by the RF after the saccade. The proposed model thus offers a unified explanation for this seemingly diverse range of phenomena. Furthermore, as the proposed model is an implementation of the predictive coding theory, it offers a single computational explanation for these phenomena and relates gaze shifts to a wider framework for understanding cortical function.

Scan patterns during real-world scene viewing predict individual differences in cognitive capacity.

From the earliest recordings of eye movements during active scene viewing to the present day, researchers have commonly reported individual differences in eye movement scan patterns under constant stimulus and task demands. These findings suggest viewer individual differences may be important for understanding gaze control during scene viewing. However, the relationship between scan patterns and viewer individual differences during scene viewing remains poorly understood because scan patterns are difficult to analyze. The present study uses a powerful technique called Successor Representation Scanpath Analysis (Hayes, Petrov, & Sederberg, 2011, 2015) to quantify the strength of the association between individual differences in scan patterns during real-world scene viewing and individual differences in viewer intelligence, working memory capacity, and speed of processing. The results of this analysis revealed individual differences in scan patterns that explained more than 40% of the variance in viewer intelligence and working memory capacity measures, and more than a third of the variance in speed of processing measures. The theoretical implications of our findings for models of gaze control and avenues for future individual differences research are discussed.

LATEST: A model of saccadic decisions in space and time.

Many of our actions require visual information, and for this it is important to direct the eyes to the right place at the right time. Two or three times every second, we must decide both when and where to direct our gaze. Understanding these decisions can reveal the moment-to-moment information priorities of the visual system and the strategies for information sampling employed by the brain to serve ongoing behavior. Most theoretical frameworks and models of gaze control assume that the spatial and temporal aspects of fixation point selection depend on different mechanisms. We present a single model that can simultaneously account for both when and where we look. Underpinning this model is the theoretical assertion that each decision to move the eyes is an evaluation of the relative benefit expected from moving the eyes to a new location compared with that expected by continuing to fixate the current target. The eyes move when the evidence that favors moving to a new location outweighs that favoring staying at the present location. Our model provides not only an account of when the eyes move, but also what will be fixated. That is, an analysis of saccade timing alone enables us to predict where people look in a scene. Indeed our model accounts for fixation selection as well as (and often better than) current computational models of fixation selection in scene viewing. (PsycINFO Database Record

Gaze-contingent manipulation of the FVF demonstrates the importance of fixation duration for explaining search behavior.

Hulleman & Olivers' (H&O's) model introduces variation of the functional visual field (FVF) for explaining visual search behavior. Our research shows how the FVF can be studied using gaze-contingent displays and how FVF variation can be implemented in models of gaze control. Contrary to H&O, we believe that fixation duration is an important factor when modeling visual search behavior.

Models of gaze control for manipulation tasks

Human studies have shown that gaze shifts are mostly driven by the current task demands. In manipulation tasks, gaze leads action to the next manipulation target. One explanation is that fixations gather information about task relevant properties, where task relevance is signalled by reward. This work presents new computational models of gaze shifting, where the agent imagines ahead in time the informational effects of possible gaze fixations. Building on our previous work, the contributions of this article are: (i) the presentation of two new gaze control models, (ii) comparison of their performance to our previous model, (iii) results showing the fit of all these models to previously published human data, and (iv) integration of a visual search process. The first new model selects the gaze that most reduces positional uncertainty of landmarks (Unc), and the second maximises expected rewards by reducing positional uncertainty (RU). Our previous approach maximises the expected gain in cumulative reward by reducing positional uncertainty (RUG). In experiment ii the models are tested on a simulated humanoid robot performing a manipulation task, and each model's performance is characterised by varying three environmental variables. This experiment provides evidence that the RUG model has the best overall performance. In experiment iii, we compare the hand-eye coordination timings of the models in a robot simulation to those obtained from human data. This provides evidence that only the models that incorporate both uncertainty and reward (RU and RUG) match human data.

Adaptive Gaze Control for Object Detection

We propose a novel gaze-control model for detecting objects in images. The model, named act-detect, uses the information from local image samples in order to shift its gaze towards object locations. The model constitutes two main contributions. The first contribution is that the model’s setup makes it computationally highly efficient in comparison with existing window-sliding methods for object detection, while retaining an acceptable detection performance. act-detect is evaluated on a face-detection task using a publicly available image set. In terms of detection performance, act-detect slightly outperforms the window-sliding methods that have been applied to the face-detection task. In terms of computational efficiency, act-detect clearly outperforms the window-sliding methods: it requires in the order of hundreds fewer samples for detection. The second contribution of the model lies in its more extensive use of local samples than previous models: instead of merely using them for verifying object presence at the gaze location, the model uses them to determine a direction and distance to the object of interest. The simultaneous adaptation of both the model’s visual features and its gaze-control strategy leads to the discovery of features and strategies for exploiting the local context of objects. For example, the model uses the spatial relations between the bodies of the persons in the images and their faces. The resulting gaze control is a temporal process, in which the object’s context is exploited at different scales and at different image locations relative to the object.

Revisión del modelo del control de la mirada vertical

The main structures involved in the control of vertical gaze, both saccades, smooth pursuit and oculovestibular reflexes, are the rostral interstitial nucleus of medial longitudinal fasciculus, posterior commissure, interstitial nucleus of Cajal, oculomotor complex and trochlear nerve nucleus. Despite knowing the functions of these nuclei, and their main interconnections, afferents and efferents, there is no definitive and contrasted model of vertical gaze control in humans.Through the description of three cases, and as described in scientific literature, our aim is to review the models described to date.The control of vertical saccades generates in the rostral interstitial nucleus of medial longitudinal fasciculus, projecting to the pertinent oculomotor nuclei ipsilaterally for the inferior gaze, and bilaterally for the superior gaze.The double cross-innervation of the nuclei responsible for superior gaze, implies that unilateral lesions predominantly affect the inferior gaze.

A head movement propensity model for animating gaze shifts and blinks of virtual characters

An automatic model is presented for animating gaze shifts of virtual characters towards target locations in a virtual environment. Two connected components are described: an eye–head controller and a blinking controller. The gaze control model is based on results from neuroscience, and dictates the contributions of the eyes and head to a gaze shift according to an individual's head movement propensity; that is, their tendency to recruit their head when making gaze motions under different conditions. The blink controller simulates gaze-evoked blinking, a specific category of behaviours that accompany gaze shifts. The probability of occurrence of such blinks, and their amplitude, is related to the gaze shift. The model forms the basis for a number of experiments investigating the impact of blinking, eye–head ratio and direction of head movements on user perception. In addition to other application domains, the findings are of significance to serious games environments, where the perceived quality of a character's gaze may affect engagement, immersion and learning outcomes.

Systems level model integration and embodiment: a case study with gaze control

Systems level model integration and embodiment: a case study with gaze control

How are eye fixation durations controlled during scene viewing? Further evidence from a scene onset delay paradigm

Recent research on eye movements during scene viewing has focused on where the eyes fixate. But eye fixations also differ in their durations. Here we investigated whether fixation durations in scene viewing are under the direct and immediate control of the current visual input. In two scene memorization and one visual search experiments, the scene was removed from view during critical fixations for a predetermined delay, and then restored following the delay. Experiment 1 compared filled (pattern mask) and unfilled (grey field) delays. Experiment 2 compared random to blocked delays. Experiment 3 extended the results to a visual search task. The results demonstrate that fixation durations in scene viewing comprise two fixation populations. One population remains relatively constant across delay, and the second population increases with scene onset delay. The results are consistent with a mixed eye movement control model that incorporates an autonomous control mechanism with process monitoring. The results suggest that a complete gaze control model will have to account for both fixation location and fixation duration.

Adaptive Gaze Control in Natural Environments

The sequential acquisition of visual information from scenes is a fundamental component of natural visually guided behavior. However, little is known about the control mechanisms responsible for the eye movement sequences that are executed in the service of such behavior. Theoretical attempts to explain gaze patterns have almost exclusively concerned two-dimensional displays that do not accurately reflect the demands of natural behavior in dynamic environments or the importance of the observer's behavioral goals. A difficult problem for all models of gaze control, intrinsic to selective perceptual systems, is how to detect important but unexpected stimuli without consuming excessive computational resources. We show, in a real walking environment, that human gaze patterns are remarkably sensitive to the probabilistic structure of the environment, suggesting that observers handle the uncertainty of the natural world by proactively allocating gaze on the basis of learned statistical structure. This is consistent with the role of reward in the oculomotor neural circuitry and supports a reinforcement learning approach to understanding gaze control in natural environments.

An Anthropomorphic Robotic Head for Investigating Gaze Control

This paper presents the biomechatronic design and development of an anthropomorphic robotic head able to perform human eye movements as a tool for experimental investigation in neuroscience. This robotic head has been designed upon specifications derived from models of the human head and of human neck and eye movements, in terms of total mass, geometry, number of degrees of freedom (d.o.f.), joint velocities and accelerations. The robotic head has 7 d.o.f.: 4 d.o.f. specifically allowing the neck movements and 3 d.o.f. dedicated to the faster eye movements; it has a total mass of approximately 5.6 kg, and it is capable of reaching a maximum eye velocity and acceleration of 1000°/s and 10 000°/s2, respectively, compatible with human data. The coordination of neck and eye movements allows smooth pursuit; and the independent eye yaw movements allow vergence. As an example, the use of this robotic head for validating a novel neurophysiological model of gaze control is presented. The model has been implemented on the robot, which can transform the retina images into their mapping onto the relevant brain area (superior colliculus), calculate the coordinates of a visual stimulus appearing in the periphery, generate the velocity profiles for the eye motors and execute the eye movements. This implementation shows that the robotic head can perform human-like saccades and allows experimental comparison with human data, so as to validate and revise the model itself.

Bioinspired velocity control of fast gaze shifts on a robotic anthropomorphic head

In this paper we address the problem of executing fast gaze shifts toward a visual target with a robotic platform. The robotic platform is an anthropomorphic head with seven degrees of freedom (DOFs) that was designed to mimic the physical dimensions (i.e. geometry and masses), the performances (i.e. angles and velocities) and the functional abilities (i.e. neck-movements and eyes vergence) of the human head. In our approach the "gold performance" of the robotic head is represented by the accurate eye-head coordination that is observed during head-free gaze saccades in humans. To this aim, we implemented and tested on the robotic head a well-characterized, biologically inspired model of gaze control and we investigate the effectiveness of the bioinspired paradigm to achieve an appropriate control of the multi-DOF robotic head. Moreover, in order to verify if the proposed model can reproduce the typical patterns of actual human movements, we performed a quantitative investigation of the relation between movement amplitude, duration and peak velocity. In the latter case, we compared the actual robot performances with existing data on human main sequence which is known to provide a general method for quantifying the dynamic of oculomotor control. The obtained results confirmed (1) the ability of the proposed bioinspired control to achieve and maintain and stable fixation of the target which was always well-positioned within the fovea and (2) the ability to reproduce the typical human main sequence diagrams which were never been successfully implemented on a fully anthropomorphic head. Even if fundamentally aimed at the experimental investigation of the underlying neurophysiologic models, the present study is also intended to provide some possible relevant solutions to the development of human-like eye movements in humanoid robots.

Visual-Vestibular Interaction Hypothesis for the Control of Orienting Gaze Shifts by Brain Stem Omnipause Neurons

Models of combined eye-head gaze shifts all aim to realistically simulate behaviorally observed movement dynamics. One of the most problematic features of such models is their inability to determine when a saccadic gaze shift should be initiated and when it should be ended. This is commonly referred to as the switching mechanism mediated by omni-directional pause neurons (OPNs) in the brain stem. Proposed switching strategies implemented in existing gaze control models all rely on a sensory error between instantaneous gaze position and the spatial target. Accordingly, gaze saccades are initiated after presentation of an eccentric visual target and subsequently terminated when an internal estimate of gaze position becomes nearly equal to that of the target. Based on behavioral observations, we demonstrate that such a switching mechanism is insufficient and is unable to explain certain types of movements. We propose an improved hypothesis for how the OPNs control gaze shifts based on a visual-vestibular interaction of signals known to be carried on anatomical projections to the OPN area. The approach is justified by the analysis of recorded gaze shifts interrupted by a head brake in animal subjects and is demonstrated by implementing the switching mechanism in an anatomically based gaze control model. Simulated performance reveals that a weighted sum of three signals: gaze motor error, head velocity, and eye velocity, hypothesized as inputs to OPNs, successfully reproduces diverse behaviorally observed eye-head movements that no other existing model can account for.

A situated model for sensory–motor coordination in gaze control

This paper shows that sensory–motor coordination contributes to the performance of situated models on the high-level task of artificial gaze control for gender recognition in static natural images. To investigate the advantage of sensory–motor coordination, we compare a non-situated model of gaze control with a situated model. The non-situated model is incapable of sensory–motor coordination. It shifts the gaze according to a fixed set of locations, optimised by an evolutionary algorithm. The situated model determines gaze shifts on the basis of local inputs in a visual scene. An evolutionary algorithm optimises the model’s gaze control policy. In the experiments performed, the situated model outperforms the non-situated model. By adopting a Bayesian framework, we show that the mechanism of sensory–motor coordination is the cause of this performance difference. The essence is that the mechanism maximises task-specific information in the observations over time, by establishing dependencies between multiple actions and observations.

Electrical stimulation of rhesus monkey nucleus reticularis gigantocellularis. II. Effects on metrics and kinematics of ongoing gaze shifts to visual targets.

Saccade kinematics are altered by ongoing head movements. The hypothesis that a head movement command signal, proportional to head velocity, transiently reduces the gain of the saccadic burst generator (Freedman 2001, Biol Cybern 84:453-462) can account for this observation. Using electrical stimulation of the rhesus monkey nucleus reticularis gigantocellularis (NRG) to alter the head contribution to ongoing gaze shifts, two critical predictions of this gaze control hypothesis were tested. First, this hypothesis predicts that activation of the head command pathway will cause a transient reduction in the gain of the saccadic burst generator. This should alter saccade kinematics by initially reducing velocity without altering saccade amplitude. Second, because this hypothesis does not assume that gaze amplitude is controlled via feedback, the added head contribution (produced by NRG stimulation on the side ipsilateral to the direction of an ongoing gaze shift) should lead to hypermetric gaze shifts. At every stimulation site tested, saccade kinematics were systematically altered in a way that was consistent with transient reduction of the gain of the saccadic burst generator. In addition, gaze shifts produced during NRG stimulation were hypermetric compared with control movements. For example, when targets were briefly flashed 30 degrees from an initial fixation location, gaze shifts during NRG stimulation were on average 140% larger than control movements. These data are consistent with the predictions of the tested hypothesis, and may be problematic for gaze control models that rely on feedback control of gaze amplitude, as well as for models that do not posit an interaction between head commands and the saccade burst generator.

Active vision system integrated with angular velocity sensors

This paper describes intelligent sensing and sensor fusion technology, which are studied through the design of a visual target tracking system. At first, a gaze control model integrating visual information is proposed. It is derived from the requirement that the target should be captured at the center of the camera's field of view at a certain time. It is interesting that motions similar to both human saccadic and smooth pursuit eye movement appear in the model. Then a method of integrating the gaze control system and angular velocity sensors is proposed. This method enables unexpectedly disturbed cameras to perform gaze control like the human vestibulo-ocular reflex (VOR). The essence of this method is, in short, the decomposition of a required task into subsystems and the reconstruction of a simple integrating system from them. We made a proto-type system and evaluated the gaze control model and the integration method experimentally.