Oculomotor Information Research Articles

Multimodal deep learning-based diagnostic model for BPPV

BackgroundBenign paroxysmal positional vertigo (BPPV) is a prevalent form of vertigo that necessitates a skilled physician to diagnose by observing the nystagmus and vertigo resulting from specific changes in the patient’s position. In this study, we aim to explore the integration of eye movement video and position information for BPPV diagnosis and apply artificial intelligence (AI) methods to improve the accuracy of BPPV diagnosis.MethodsWe collected eye movement video and diagnostic data from 518 patients with BPPV who visited the hospital for examination from January to March 2021 and developed a BPPV dataset. Based on the characteristics of the dataset, we propose a multimodal deep learning diagnostic model, which combines a video understanding model, self-encoder, and cross-attention mechanism structure.ResultOur validation test on the test set showed that the average accuracy of the model reached 81.7%, demonstrating the effectiveness of the proposed multimodal deep learning method for BPPV diagnosis. Furthermore, our study highlights the significance of combining head position information and eye movement information in BPPV diagnosis. We also found that postural and eye movement information plays a critical role in the diagnosis of BPPV, as demonstrated by exploring the necessity of postural information for the diagnostic model and the contribution of cross-attention mechanisms to the fusion of postural and oculomotor information. Our results underscore the potential of AI-based methods for improving the accuracy of BPPV diagnosis and the importance of considering both postural and oculomotor information in BPPV diagnosis.

Lexical Processing Strongly Affects Reading Times But Not Skipping During Natural Reading.

In a typical text, readers look much longer at some words than at others, even skipping many altogether. Historically, researchers explained this variation via low-level visual or oculomotor factors, but today it is primarily explained via factors determining a word's lexical processing ease, such as how well word identity can be predicted from context or discerned from parafoveal preview. While the existence of these effects is well established in controlled experiments, the relative importance of prediction, preview and low-level factors in natural reading remains unclear. Here, we address this question in three large naturalistic reading corpora (n = 104, 1.5 million words), using deep neural networks and Bayesian ideal observers to model linguistic prediction and parafoveal preview from moment to moment in natural reading. Strikingly, neither prediction nor preview was important for explaining word skipping-the vast majority of explained variation was explained by a simple oculomotor model, using just fixation position and word length. For reading times, by contrast, we found strong but independent contributions of prediction and preview, with effect sizes matching those from controlled experiments. Together, these results challenge dominant models of eye movements in reading, and instead support alternative models that describe skipping (but not reading times) as largely autonomous from word identification, and mostly determined by low-level oculomotor information.

Detection of Oculomotor Dysmetria From Mobile Phone Video of the Horizontal Saccades Task Using Signal Processing and Machine Learning Approaches.

Eye movement assessments have the potential to help in diagnosis and tracking of neurological disorders. Cerebellar ataxias cause profound and characteristic abnormalities in smooth pursuit, saccades, and fixation. Oculomotor dysmetria (i.e., hypermetric and hypometric saccades) is a common finding in individuals with cerebellar ataxia. In this study, we evaluated a scalable approach for detecting and quantifying oculomotor dysmetria. Eye movement data were extracted from iPhone video recordings of the horizontal saccade task (a standard clinical task in ataxia) and combined with signal processing and machine learning approaches to quantify saccade abnormalities. Entropy-based measures of eye movements during saccades were significantly different in 72 individuals with ataxia with dysmetria compared with 80 ataxia and Parkinson's participants without dysmetria. A template matching-based analysis demonstrated that saccadic eye movements in patients without dysmetria were more similar to the ideal template of saccades. A support vector machine was then used to train and test the ability of multiple signal processing features in combination to distinguish individuals with and without oculomotor dysmetria. The model achieved 78% accuracy (sensitivity= 80% and specificity= 76%). These results show that the combination of signal processing and machine learning approaches applied to iPhone video of saccades, allow for extraction of information pertaining to oculomotor dysmetria in ataxia. Overall, this inexpensive and scalable approach for capturing important oculomotor information may be a useful component of a screening tool for ataxia and could allow frequent at-home assessments of oculomotor function in natural history studies and clinical trials.

Recovering depth from stereo-input without using any oculomotor information: A computation model

Human beings can perceive depth by using binocular disparity, namely the difference between a stereo-pair of retinal images. It is commonly believed that the visual system requires oculomotor information about the relative orientation between the two eyes to perceive depth on the basis of binocular disparity. Such oculomotor information can be obtained from the efference copy of the oculomotor signal, or it can be derived from the vertical component of the binocular disparity (vertical disparity). Psychophysical studies have shown that stereo depth perception is affected by both the efference copy and by the vertical disparity. But note that any role for the efference signal for spatial perception is limited and the process using vertical disparity is slow. These effects are too restricted to explain depth perception under natural viewing conditions in which natural eye movements are made. In the present study, I describe a computational model that can recover depth from a stereo-pair of retinal images based only on the geometrical optics. This model treats depth recovery as a Direct problem rather than as an Inverse problem because it does not need to make use of any oculomotor information or of any a priori constraints. In this model, oculomotor information is not even implicitly recovered. Simply put, this paper shows that, at least from a computational perspective, oculomotor information and a priori constraints are not actually required for the stereo perception of depth. Having such a model allows us to discuss psychophysical and neurophysiological phenomena, as well as their mechanisms, without assuming that the human visual system needs oculomotor information to recover depth in a 3D scene from stereo retinal images.

Saccade dynamics during an online updating task change with healthy aging.

Goal-directed movements rely on the integration of both visual and motor information, especially during the online control of movement, to fluidly and flexibly control coordinated action. Eye–hand coordination typically plays an important role in goal-directed movements. As people age, various aspects of motor control and visual performance decline (Haegerstrom-Portnoy, Schneck, & Brabyn, 1999; Seidler et al., 2010), including an increase in saccade latencies (Munoz, Broughton, Goldring, & Armstrong, 1998). However, there is limited insight into how age-related changes in saccadic performance impact eye–hand coordination during online control. We investigated this question through the use of a target perturbation paradigm. Older and younger participants completed a perturbation task where target perturbations could occur either early (0 ms) or later (200 ms) after reach onset. We analyzed reach correction latencies and the frequency of the reach correction, coupled with analyses of saccades across all stages of movement. Older participants had slower correction latencies and initiated corrections less frequently compared to younger participants, with this trend being exacerbated in the later (200 ms) target perturbation condition. Older participants also produced slower saccade latencies toward both the initial target and the perturbed target. For trials in which a correction occurred to a late perturbation, touch responses were more accurate when there was more time between the saccade landing and the touch. Altogether, our results suggest that these age-related effects may be due to the delayed acquisition of visual and oculomotor information used to inform the reaching movement, stemming from the increase in saccade latencies before and after target perturbation.

A model that recovers depth from stereo without using any oculomotor information

A model that recovers depth from stereo without using any oculomotor information

A Computational Model that Recovers Depth from Stereo-Input without Using Any Oculomotor Information

It is commonly believed that the visual system requires oculomotor information to perceive depth from binocular disparity. However, any effect of the oculomotor information on depth perception is too restricted to explain depth perception under natural viewing conditions. In this study, I describe a computational model that can recover depth from a stereo-pair of retinal images without using any oculomotor information. The model shows that, at least from a computational perspective, any oculomotor information is not necessary for perceiving depth from the stereo retinal images.

(Disparity-Driven) Accommodation Response Contributes to Perceived Depth.

When looking at objects at various distances in the physical space, the accommodation and vergence systems adjust their parameters to provide a single and clear vision of the world. Subtended muscle activity provides oculomotor cues that can contribute to the perception of depth and distance. While several studies have outlined the role of vergence in distance perception, little is known about the contribution of its concommitant accommodation component. It is possible to unravel the role of each of these physiological systems by placing observers in a situation where there is a conflict between accommodation and vergence distances. We thus sought to determine the contribution of each response system to perceived depth by simultaneously measuring vergence and accommodation while participants judged the depth of 3D stimuli. The distance conflict decreased depth constancy for stimulus displayed with negative disparity steps (divergence). Although vergence was unaffected by the stimulus distance, accommodation responses were significantly reduced when the stimulus was displayed with negative disparities. Our results show that biases in perceived depth follow undershoots in the disparity-driven accommodation response. These findings suggest that accommodation responses (i.e., from oculomotor information) can contribute to perceived depth.

Change of information represented by thalamic mediodorsal neurons during the delay period.

The thalamic mediodorsal (MD) nucleus plays an important role in transforming visual information into motor information during spatial working-memory performances. To understand the neural mechanism of this transformation process, we examined whether or not the information represented in individual MD neuron's activity changes during a trial of the task. Two monkeys performed two oculomotor delayed-response tasks (an ordinary and a rotatory oculomotor delayed-response task). As MD neurons show directional delay-period activity, we compared the directional selectivity of the same MD neuron between these two tasks and determined whether the activity represented the cue direction or the saccade direction. Among the 26 MD neurons that showed directional delay-period activity, representing information of 27% of neurons gradually altered from the visual domain to the oculomotor domain, while the remaining neurons kept holding either the visual or the oculomotor information throughout the delay period. These results indicate that gradual alteration of information representing in individual MD neurons during the delay period plays an important role in spatial working-memory performance. This alteration could be performed by interactions among MD neurons representing different information during the delay period.

Spatial But Not Oculomotor Information Biases Perceptual Memory: Evidence From Face Perception and Cognitive Modeling

Recent research put forward the hypothesis that eye movements are integrated in memory representations and are reactivated when later recalled. However, "looking back to nothing" during recall might be a consequence of spatial memory retrieval. Here, we aimed at distinguishing between the effect of spatial and oculomotor information on perceptual memory. Participants' task was to judge whether a morph looked rather like the first or second previously presented face. Crucially, faces and morphs were presented in a way that the morph reactivated oculomotor and/or spatial information associated with one of the previously encoded faces. Perceptual face memory was largely influenced by these manipulations. We considered a simple computational model with an excellent match (4.3% error) that expresses these biases as a linear combination of recency, saccade, and location. Surprisingly, saccades did not play a role. The results suggest that spatial and temporal rather than oculomotor information biases perceptual face memory.

The role of eye movements in motor sequence learning

An experiment that utilized a 16-element movement sequence was designed to determine the impact of eye movements on sequence learning. The participants were randomly assigned to two experimental groups: a group that was permitted to use eye movements (FREE) and a second group (FIX) that was instructed to fixate on a marker during acquisition (ACQ). A retention test (RET) was designed to provide a measure of learning, and two transfer tests were designed to determine the extent to which eye movements influenced sequence learning. The results demonstrated that both groups decreased the response time to produce the sequence, but the participants in the FREE group performed the sequence more quickly than participants of the FIX group during the ACQ, RET and the two transfer tests. Furthermore, continuous visual control of response execution was reduced over the course of learning. The results of the transfer tests indicated that oculomotor information regarding the sequence can be stored in memory and enhances response production.

Directing Eye Gaze Enhances Auditory Spatial Cue Discrimination

The present study demonstrates, for the first time, a specific enhancement of auditory spatial cue discrimination due to eye gaze. Whereas the region of sharpest visual acuity, called the fovea, can be directed at will by moving one's eyes, auditory spatial information is derived primarily from head-related acoustic cues. Past auditory studies have found better discrimination in front of the head [1-3] but have not manipulated subjects' gaze, thus overlooking potential oculomotor influences. Electrophysiological studies have shown that the inferior colliculus, a critical auditory midbrain nucleus, shows visual and oculomotor responses [4-6] and modulations of auditory activity [7-9], and that auditory neurons in the superior colliculus show shifting receptive fields [10-13]. How the auditory system leverages this crossmodal information at the behavioral level remains unknown. Here we directed subjects' gaze (with an eccentric dot) or auditory attention (with lateralized noise) while they performed an auditory spatial cue discrimination task. We found that directing gaze toward a sound significantly enhances discrimination of both interaural level and time differences, whereas directing auditory spatial attention does not. These results show that oculomotor information variably enhances auditory spatial resolution even when the head remains stationary, revealing a distinct behavioral benefit possibly arising from auditory-oculomotor interactions at an earlier level of processing than previously demonstrated.

The Effect of Mass Representation and Oculomotor Overshoot on “Representational Momentum

Analogous to the physical momentum of objects in the real-world, visual memory for the final position of a moving target is usually displaced along its trajectory. Such displacement is referred to representational momentum. Several different approaches have been applied to interpret the representational momentum, which range from low-level perceptual processing to high-level cognitive mechanisms (Hubbard, 2010). An early approach tried to explain the displacement effect by cognitive factors dealing with principles of "internalized dynamics". In addition to this cognitive approach, "low-level" explanations of representational momentum also have been proposed, suggesting that sensory factors play a decisive role. In particular, one plausible interpretation attributes the displacement effect to oculomotor action. In addition, it is possible that high-level information regarding physical principles contribute to the displacement by modulating eye movements (Hubbard, 2006b). We hypothesize that if eye movements modulate or mediate the effects of such information on displacement, disrupting normal eye movement would interrupt the effective information. In the current study, three types of 3D balls (made of paper, wood or stone) rolled from left or right sides accompanied by sound corresponding to the ball material. In Experiment 1, subjects (22 college students) were instructed to indicate the stop position of the ball by pressing a button after tracking its movement for 1037ms and disappeared about 350ms. In Experiment 2, subjects (19 college students) were asked either to pursue the moving target until it disappeared or to track it till it stopped at the center of screen indicating by a "+". The other procedures are similar to the Experiment 1. The subjects’ eye positions in both experiments were recorded by SMI Hi-Speed eye tracking system with sampling rate of 350Hz. In Experiment 1, the forward shift in the direction of motion of the stone ball was larger than those of the wooden and paper balls’. There was a significant difference between the forward shift of the stone ball and wooden or paper ball. In addition, the oculomotor overshoot velocity of the stone ball was significantly faster than other two lighter balls’. In Experiment 2, when the oculomotor behavior was controlled, the forward shift in the trajectory and the oculomotor overshoot velocity were reduced compared to those in the eye tracking condition. The present findings revealed that high-level cognitive factors (mass representation) modulate both oculomotor behavior and representational momentum. Meanwhile, representational momentum is also influenced by oculomotor overshoot. Our results suggest that high-level cognitive factors (mass representation) influence representational momentum by oculomotor-related information. Yet, representational momentum is also affected by many other factors. Our study provides new evidence that perceptual factor (oculomotor information) is not the only way to produce representational momentum.

Localisation of Auditory Targets during Optokinetic Nystagmus

Previous studies have shown that the perceived location of visual stimuli briefly flashed during smooth pursuit, saccades, or optokinetic nystagmus (OKN) is not veridical. We investigated whether these mislocalisations can also be observed for brief auditory stimuli presented during OKN. Experiments were carried out in a lightproof sound-attenuated chamber. Participants performed eye movements elicited by visual stimuli. An auditory target (white noise) was presented for 5 ms. Our data clearly indicate that auditory targets are mislocalised during reflexive eye movements. OKN induces a shift of perceived location in the direction of the slow eye movement and is modulated in the temporal vicinity of the fast phase. The mislocalisation is stronger for look- as compared to stare-nystagmus. The size and temporal pattern of the observed mislocalisation are different from that found for visual targets. This suggests that different neural mechanisms are at play to integrate oculomotor signals and information on the spatial location of visual as well as auditory stimuli.

Saccade-Related Information in the Superior Temporal Motion Complex: Quantitative Functional Mapping in the Monkey

Although the role of the motion complex [cortical areas middle temporal (V5/MT), medial superior temporal (MST), and fundus of the superior temporal (FST)] in visual motion and smooth-pursuit eye movements is well understood, little is known about its involvement in rapid eye movements (saccades). To address this issue, we used the quantitative 14C-deoxyglucose method to obtain functional maps of the cerebral cortex lying in the superior temporal sulcus of rhesus monkeys executing saccades to visual targets and saccades to memorized targets in complete darkness. Fixational effects were observed in MT-foveal, FST, the anterior part of V4-transitional (V4t), and temporal-occipital areas. Saccades to memorized targets activated areas V5/MT, MST, and V4t, which were also activated for saccades to visual targets. Regions activated in the light and in the dark overlapped extensively. In addition, saccades to visual targets activated areas FST and the intermediate part of the polysensory temporal-parietal-occipital area. Cortical activity related to visually guided saccades could be explained, at least in part, by visual motion. Because only oculomotor signals can account for the equally robust activations induced by memory saccades in complete darkness, we suggest that areas V5/MT, MST, and V4t receive and/or process saccade-related oculomotor information.

Distances appear different on hills

When walking effort is increased due to manipulations such as wearing heavy backpacks, people perceive hills to be steeper and distances to be farther (Bhalla & Proffitt, 1999; Proffitt, Stefanucci, Banton, & Epstein, 2003). On the basis of these findings, we expected people to overestimate distances on steep hills relative to the same distances on flat ground, because of the increased effort required to ascend or descend them. This hypothesis is in contrast to the belief that distances are specified solely by optical and oculomotor information related to the geometry of the environment. To test the hypothesis, we investigated distance estimation on hills and flat terrains in natural and virtual environments. We found that participants judged steep uphill and downhill distances to be farther than the same distances on flat terrain. These results are inconsistent with the idea that spatial layout is perceived solely in terms of geometry, lending partial support to an effort hypothesis.

How does action resist visual illusion? Uncorrected oculomotor information does not account for accurate pointing in peripersonal space

Using spatially identical displays (variants of the Müller-Lyer illusion), we compared the accuracy of spatial verbal judgments with that of saccadic (eye) and pointing (hand) movements. Verbal judgments showed a clear effect of the illusion. The amplitude of the primary saccade from one endpoint of the pattern (at fixation) to the other also showed an effect of the illusion. Conversely, movement amplitudes when pointing from one endpoint (initial finger position) to the other were significantly more accurate than both saccades and verbal responses. In a control experiment in which the viewing conditions between the saccade and pointing experiments were equalized, saccade amplitude was again affected by the illusion. In several studies, systematic biases in conscious spatial judgments have been contrasted with accurate open-loop pointing in peripersonal space. It has been proposed that such seeming dissociations between vision-for-action and vision-for-consciousness might in fact be because of a simple oculomotor strategy: saccade to the target before it disappears, then use the efference copy of the (accurate) saccadic movement to drive pointing. The present data do not support the hypothesis in this simple form.

Mr. Chips 2002: new insights from an ideal-observer model of reading

The integration of visual, lexical, and oculomotor information is a critical part of reading. Mr. Chips is an ideal-observer model that combines these sources of information optimally to read simple texts in the minimum number of saccades. This model provides a computational framework for interpreting human reading saccades in both normal and low vision. The purpose of this paper is to report performance of the model for conditions emulating reading with normal vision––a visual span of nine characters, multiplicative saccade noise with a standard deviation of 30%, and texts based on three full-length children's books. Comparison of fixation locations by humans and Mr. Chips revealed: (1) that both exhibit very similar word-skipping behavior; (2) both show initial fixations near the center of words, but with a systematic difference suggestive of an asymmetry in the human visual span; and (3) differences in the pattern of refixations within words that may uncover non-optimal lexical inference by human readers. A human context effect––30% difference in mean saccade size between continuous text and random sequences of words––was very similar to the 25% effect for the model associated with a corresponding difference in the predictability of text words. Overall, our findings show that many of the complicated aspects of human reading saccades can be explained concisely by early information-processing constraints.

Combination of hand and gaze signals during reaching: activity in parietal area 7 m of the monkey.

The role of area 7 m has been studied by recording the activity of single neurons of monkeys trained to fixate and reach toward peripheral targets. The target was randomly selected from eight possible locations on a virtual circle, of radius 30 degrees visual angle from a central target. Three tasks were employed to dissociate hand- from eye-related contributions. In the first task, animals looked and reached to the peripheral target. In a second task, the animal reached to the peripheral target while maintaining fixation on the central target. In the third task, the monkey maintained fixation on peripheral targets that were spatially coincident with those of the reaching tasks. The results show that cell activity in area 7 m relates, for some cells to eye position, for others to hand position and movement, and for the majority of cells to a combination of visuomanual and oculomotor information. This area, therefore, seems to perform an early combination of information in the processing leading from target localization to movement generation.

Mr. Chips: an ideal-observer model of reading.

The integration of visual, lexical, and oculomotor information is a critical part of reading. Mr. Chips is an ideal-observer model that combines these sources of information optimally to read simple texts in the minimum number of saccades. In the model, the concept of the visual span (the number of letters that can be identified in a single fixation) plays a key, unifying role. The behavior of the model provides a computational framework for reexamining the literature on human reading saccades. Emergent properties of the model, such as regressive saccades and an optimal-viewing position, suggest new interpretations of human behavior. Because Mr. Chip's "retina" can have any (one-dimensional) arrangement of high-resolution regions and scotomas, the model can simulate common visual disorders. Surprising saccade strategies are linked to the pattern of scotomas. For example, Mr. Chips sometimes plans a saccade that places a decisive letter in a scotoma. This article provides the first quantitative model of the effects of scotomas on reading.