Fixation-related Potentials Research Articles

Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Using Fixation-Related Potentials and Eye Tracking Features

While Mixed Reality allows the seamless blending of digital content in users' surroundings, it is unclear if its fusion with physical information impacts users' perceptual and cognitive resources differently. While the fusion of digital and physical objects provides numerous opportunities to present additional information, it also introduces undesirable side effects, such as split attention and increased visual complexity. We conducted a visual search study in three manifestations of mixed reality (Augmented Reality, Augmented Virtuality, Virtual Reality) to understand the effects of the environment on visual search behavior. We conducted a multimodal evaluation measuring Fixation-Related Potentials (FRPs), alongside eye tracking to assess search efficiency, attention allocation, and behavioral measures. Our findings indicate distinct patterns in FRPs and eye-tracking data that reflect varying cognitive demands across environments. Specifically, AR environments were associated with increased workload, as indicated by decreased FRP - P3 amplitudes and more scattered eye movement patterns, impairing users' ability to identify target information efficiently. Participants reported AR as the most demanding and distracting environment. These insights inform design implications for MR adaptive systems, emphasizing the need for interfaces that dynamically respond to user cognitive load based on physiological inputs.

Fixation-related potentials during mobile map assisted navigation in the real world: The effect of landmark visualization style.

An often-proposed enhancement for mobile maps to aid assisted navigation is the presentation of landmark information, yet understanding of the manner in which they should be displayed is limited. In this study, we investigated whether the visualization of landmarks as 3D map symbols with either an abstract or realistic style influenced the subsequent processing of those landmarks during route navigation. We utilized a real-world mobile electroencephalography approach to this question by combining several tools developed to overcome the challenges typically encountered in real-world neuroscience research. We coregistered eye-movement and EEG recordings from 45 participants as they navigated through a real-world environment using a mobile map. Analyses of fixation event-related potentials revealed that the amplitude of the parietal P200 component was enhanced when participants fixated landmarks in the real world that were visualized on the mobile map in a realistic style, and that frontal P200 latencies were prolonged for landmarks depicted in either a realistic or abstract style compared with features of the environment that were not presented on the map, but only for the male participants. In contrast, we did not observe any significant effects of landmark visualization style on visual P1-N1 peaks or the parietal late positive component. Overall, the findings indicate that the cognitive matching process between landmarks seen in the environment and those previously seen on a map is facilitated by more realistic map display, while low-level perceptual processing of landmarks and recall of associated information are unaffected by map visualization style.

Parafoveal and foveal N400 effects in natural reading: A timeline of semantic processing from fixation-related potentials.

The depth at which parafoveal words are processed during reading is an ongoing topic of debate. Recent studies using RSVP-with-flanker paradigms have shown that implausible words within sentences elicit an N400 component while they are still in parafoveal vision, suggesting that the semantics of parafoveal words can be accessed to rapidly update the sentence representation. To study this effect in natural reading, we combined the coregistration of eye movements and EEG with the deconvolution modeling of fixation-related potentials (FRPs) to test whether semantic plausibility is processed parafoveally during Chinese sentence reading. For one target word per sentence, both its parafoveal and foveal plausibility were orthogonally manipulated using the boundary paradigm. Consistent with previous eye movement studies, we observed a delayed effect of parafoveal plausibility on fixation durations that only emerged on the foveal word. Crucially, in FRPs aligned to the pretarget fixation, a clear N400 effect emerged already based on parafoveal plausibility, with more negative voltages for implausible previews. Once participants fixated the target, we again observed an N400 effect of foveal plausibility. Interestingly, this foveal N400 was absent whenever the preview had been implausible, indicating that when a word's (im)plausibility is already processed in parafoveal vision, this information is not revised anymore upon direct fixation. Implausible words also elicited a late positive component (LPC), but exclusively when in foveal vision. Our results not only provide convergent neural and behavioral evidence for the parafoveal uptake of semantic information, but also indicate different contributions of parafoveal versus foveal information toward higher level sentence processing.

(Micro)saccade-related potentials during face recognition: A study combining EEG, eye-tracking, and deconvolution modeling.

Under natural viewing conditions, complex stimuli such as human faces are typically looked at several times in succession, implying that their recognition may unfold across multiple eye fixations. Although electrophysiological (EEG) experiments on face recognition typically prohibit eye movements, participants still execute frequent (micro)saccades on the face, each of which generates its own visuocortical response. This finding raises the question of whether the fixation-related potentials (FRPs) evoked by these tiny gaze shifts also contain psychologically valuable information about face processing. Here, we investigated this question by corecording EEG and eye movements in an experiment with emotional faces (happy, angry, neutral). Deconvolution modeling was used to separate the stimulus ERPs to face onset from the FRPs generated by subsequent microsaccades-induced refixations on the face. As expected, stimulus ERPs exhibited typical emotion effects, with a larger early posterior negativity (EPN) for happy/angry compared with neutral faces. Eye tracking confirmed that participants made small saccades in 98% of the trials, which were often aimed at the left eye of the stimulus face. However, while each saccade produced a strong response over visual areas, this response was unaffected by the face's emotional expression, both for the first and for subsequent (micro)saccades. This finding suggests that the face's affective content is rapidly evaluated after stimulus onset, leading to only a short-lived sensory enhancement by arousing stimuli that does not repeat itself during immediate refixations. Methodologically, our work demonstrates how eye tracking and deconvolution modeling can be used to extract several brain responses from each EEG trial, providing insights into neural processing at different latencies after stimulus onset.

Linguistic processes intervene much later than visuo-motor processes during an eye fixation: Evidence from Fixation-Related Potentials during reading.

Linguistic processes intervene much later than visuo-motor processes during an eye fixation: Evidence from Fixation-Related Potentials during reading.

Fixation-related electrical potentials during a free visual search task reveal the timing of visual awareness

It has been repeatedly claimed that emotional faces readily capture attention, and that they may be processed without awareness. Yet some observations cast doubt on these assertions. Part of the problem may lie in the experimental paradigms employed. Here, we used a free viewing visual search task during electroencephalographic recordings, where participants searched for either fearful or neutral facial expressions among distractor expressions. Fixation-related potentials were computed for fearful and neutral targets and the response compared for stimuli consciously reported or not. We showed that awareness was associated with an electrophysiological negativity starting at around 110ms, while emotional expressions were distinguished on the N170 and early posterior negativity only when stimuli were consciously reported. These results suggest that during unconstrained visual search, the earliest electrical correlate of awareness may emerge as early as 110ms, and fixating at an emotional face without reporting it may not produce any unconscious processing.

Fixation-related visual mismatch negativity.

Vision under natural conditions could be studied by combining electroencephalogram (EEG) and eye tracking as well as using saccades as triggers for the onset of the fixation-related potentials (FRPs) and for the oculomotor inhibition (OMI) that follows every saccade. The result of this analysis is thought to be equivalent to the event-related response following a peripheral preview. Previous studies that measured responses to visual deviants in a sequence of flashed stimuli found an increased negativity in the occipital N1 component (visual mismatch negativity [vMMN]), and prolonged saccadic inhibition for unexpected events. The aim of the current study was to develop an oddball paradigm in constrained natural-viewing and determine whether a similar mismatched FRP and prolonged OMI for deviance could be found. To this end, we developed a visual oddball paradigm on a static display to generate expectancy and surprise across successive saccades. Observers (n = 26) inspected, one after the other, seven small patterns of E and an inverted E arranged on the screen along a horizontal path, with one frequent (standard) and one rare (deviant), looking for a superimposed tiny dot target in each 5-second trial. Our results show a significantly larger FRP-N1 negativity for the deviant, compared with the standard and prolonged OMI of the following saccade, as previously found for transient oddballs. Our results show, for the first time, prolonged OMI and stronger fixation-related N1 to a task-irrelevant visual mismatch (vMMN) in natural, but task-guided viewing. These two signals combined could serve as markers of prediction error in free viewing.

Visual attention and cognitive process in construction hazard recognition: Study of fixation-related potential

Previous studies on hazard recognition focused on single-mode neuropsychological responses using eye tracking or neuroimaging techniques, reflecting a limited understanding of the interaction between ocular and brain activities. This study explores the construction workers' meta-cognitive processes underpinning first target fixations using co-registration of eye movement (EM) and electroencephalography (EEG). The authors conducted a hazard recognition experiment to collect EM and EEG signals under free viewing conditions and examined the relationship between visual attention and cognitive processing. The results show that the workers allocated cognitive resources first to processing targets' visual features and then to discriminating safe or hazardous characteristics. Visual perception and conscious discrimination within first fixations are related to the prior detection of surroundings and affect the subsequent refixations. This study promotes evaluating the hazard recognition process in view of synchronous visual search and brain activity and advances the workers' hazard search patterns.

Are eye movements and EEG on the same page?: A coregistration study on parafoveal preview and lexical frequency.

Readers extract visual and linguistic information not only from fixated words but also upcoming parafoveal words to introduce new input efficiently into the language processing pipeline. The lexical frequency of upcoming words and similarity with subsequent foveal information both influence the amount of time people spend once they fixate the word foveally. However, it is unclear from eye movements alone the extent to which parafoveal word processing, and the integration of that word with foveally obtained information, continues after saccade plans have been initiated. To investigate the underlying neural processes involved in word recognition after saccade planning, we coregistered electroencephalogram (EEG) and eye movements during a gaze-contingent display change paradigm. We orthogonally manipulated the frequency of the parafoveal and foveal words and measured fixation related potentials (FRPs) upon foveal fixation. Eye movements showed primarily an effect of preview frequency, suggesting that saccade planning is based on the familiarity of the parafoveal input. FRPs, on the other hand, demonstrated a disruption in downstream processing when parafoveal and foveal input differed, but only when the parafoveal word was high frequency. These findings demonstrate that lexical processing continues after the eyes have moved away from a word and that eye movements and FRPs provide distinct but complementary accounts about oculomotor behavior and neural processing that cannot be obtained from either method in isolation. Furthermore, these findings put constraints on models of reading by suggesting that lexical processes that occur before an eye movement program is initiated are qualitatively different from those that occur afterward. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Brain Network Reorganization During Visual Search Task Revealed by a Network Analysis of Fixation-Related Potential.

Visual search is ubiquitous in daily life and has attracted substantial research interest over the past decades. Although accumulating evidence has suggested complex neurocognitive processes underlying visual search, the neural communication across the brain regions remains poorly understood. The present work aimed to fill this gap by investigating functional networks of fixation-related potential (FRP) during the visual search task. Multi-frequency electroencephalogram (EEG) networks were constructed from 70 university students (male/female = 35/35) using FRPs time-locked to target and non-target fixation onsets, which were determined by concurrent eye-tracking data. Then graph theoretical analysis (GTA) and a data-driven classification framework were employed to quantitatively reveal the divergent reorganization between target and non-target FRPs. We found distinct network architectures between target and non-target mainly in the delta and theta bands. More importantly, we achieved a classification accuracy of 92.74% for target and non-target discrimination using both global and nodal network features. In line with the results of GTA, we found that the integration corresponding to target and non-target FRPs significantly differed, while the nodal features contributing most to classification performance primarily resided in the occipital and parietal-temporal areas. Interestingly, we revealed that females exhibited significantly higher local efficiency in delta band when focusing on the search task. In summary, these results provide some of the first quantitative insights into the underlying brain interaction patterns during the visual search process.

The influence of predictability, visual contrast, and preview validity on eye movements and N400 amplitude: co-registration evidence that the N400 reflects late processes

ABSTRACT Two eye movement/EEG co-registration experiments investigated effects of predictability, visual contrast, and parafoveal preview in normal reading. Replicating previous studies, in Experiment 1 contrast and predictability additively influenced fixation durations, and in Experiment 2 invalid preview eliminated the predictability effect on early eye movement measures. In both experiments, predictability influenced the amplitude of the N400 component of the fixation-related potential. In Experiment 1, visual contrast did not influence the N400, and in Experiment 2, the effect of predictability on the N400 was larger with invalid preview, in opposition to the eye movement pattern. The N400 may reflect a late process of accessing conceptual representations while the duration of the eyes’ fixation on a word is sensitive to the difficulty of perceptual encoding and early stages of word recognition. The effects of predictability on both fixation duration and the N400 suggest an influence of this variable at two distinct processing stages.

Parallel word reading revealed by fixation-related potentials

Parallel word reading revealed by fixation-related potentials

The influence of spatial frequency and luminance on early visual processing: A fixation-related potentials approach

The influence of spatial frequency and luminance on early visual processing: A fixation-related potentials approach

Face familiarity revealed by fixational eye movements and fixation-related potentials in free viewing

Event-related potentials (ERPs) and the oculomotor inhibition (OMI) in response to visual transients are known to be sensitive to stimulus properties, attention, and expectation. We have recently found that the OMI is also sensitive to face familiarity. In natural vision, stimulation of the visual cortex is generated primarily by saccades, and it has been recently suggested that fixation-related potentials (FRPs) share similar components with the ERPs. Here, we investigated whether FRPs and microsaccade inhibition (OMI) in free viewing are sensitive to face familiarity. Observers freely watched a slideshow of seven unfamiliar and one familiar facial images presented randomly for 4-s periods, with multiple images per identity. We measured the occipital fixation-related N1 relative to the P1 magnitude as well as the associated fixation-triggered OMI. We found that the average N1-P1 was significantly smaller and the OMI was shorter for the familiar face, compared with any of the seven unfamiliar faces. Moreover, the P1 was suppressed across saccades for the familiar but not for the unfamiliar faces. Our results highlight the sensitivity of the occipital FRPs to stimulus properties such as face familiarity and advance our understanding of the integration process across successive saccades in natural vision.

Brain Source Correlates of Speech Perception and Reading Processes in Children With and Without Reading Difficulties.

Neural correlates in reading and speech processing have been addressed extensively in the literature. While reading skills and speech perception have been shown to be associated with each other, their relationship remains debatable. In this study, we investigated reading skills, speech perception, reading, and their correlates with brain source activity in auditory and visual modalities. We used high-density event-related potentials (ERPs), fixation-related potentials (FRPs), and the source reconstruction method. The analysis was conducted on 12–13-year-old schoolchildren who had different reading levels. Brain ERP source indices were computed from frequently repeated Finnish speech stimuli presented in an auditory oddball paradigm. Brain FRP source indices were also computed for words within sentences presented in a reading task. The results showed significant correlations between speech ERP sources and reading scores at the P100 (P1) time range in the left hemisphere and the N250 time range in both hemispheres, and a weaker correlation for visual word processing N170 FRP source(s) in the posterior occipital areas, in the vicinity of the visual word form areas (VWFA). Furthermore, significant brain-to-brain correlations were found between the two modalities, where the speech brain sources of the P1 and N250 responses correlated with the reading N170 response. The results suggest that speech processes are linked to reading fluency and that brain activations to speech are linked to visual brain processes of reading. These results indicate that a relationship between language and reading systems is present even after several years of exposure to print.

Fixation-related saccadic inhibition in free viewing in response to stimulus saliency

Microsaccades that occur during fixation were studied extensively in response to transient stimuli, showing a typical inhibition (Oculomotor Inhibition, OMI), and a later release with a latency that depends on stimulus saliency, attention, and expectations. Here, we investigated the hypothesis that in free viewing every saccade provides a new transient stimulation that should result in a stimulus-dependent OMI like a flashed presentation during fixation. Participants (N = 16) freely inspected static displays of randomly oriented Gabor texture images, with varied contrast and spatial frequency (SF) for periods of 10 s each. Eye tracking recordings were divided into epochs triggered by saccade landing (> 1 dva), and microsaccade latency relative to fixation onset was computed (msRT). We found that the msRT in free viewing was shorter for more salient stimuli (higher contrast or lower SF), as previously found for flashed stimuli. It increased with saccade size and decreased across successive saccades, but only for higher contrast, suggesting contrast-dependent repetition enhancement in free viewing. Our results indicate that visual stimulus-dependent inhibition of microsaccades also applies to free viewing. These findings are in agreement with the similarity found between event-related and fixation-related potentials and open the way for studies combining both approaches to study natural vision.

Semantic parafoveal processing in natural reading: Insight from fixation-related potentials & eye movements.

Prior research suggests that we may access the meaning of parafoveal words during reading. We explored how semantic-plausibility parafoveal processing takes place in natural reading through the co-registration of eye movements (EM) and fixation-related potentials (FRPs), using the boundary paradigm. We replicated previous evidence of semantic parafoveal processing from highly controlled reading situations, extending their findings to more ecologically valid reading scenarios. Additionally, and exploring the time-course of plausibility preview effects, we found distinct but complementary evidence from EM and FRPs measures. FRPs measures, showing a different trend than EM evidence, revealed that plausibility preview effects may be long-lasting. We highlight the importance of a co-registration set-up in ecologically valid scenarios to disentangle the mechanisms related to semantic-plausibility parafoveal processing.

Fixation-related potentials in total darkness

Fixation-related potentials in total darkness

Fixation-related potentials in naming speed: A combined EEG and eye-tracking study on children with dyslexia

ObjectiveWe combined electroencephalography (EEG) and eye-tracking recordings to examine the underlying factors elicited during the serial Rapid-Automatized Naming (RAN) task that may differentiate between children with dyslexia (DYS) and chronological age controls (CAC). MethodsThirty children with DYS and 30 CAC (Mage = 9.79 years; age range 7.6 through 12.1 years) performed a set of serial RAN tasks. We extracted fixation-related potentials (FRPs) under phonologically similar (rime-confound) or visually similar (resembling lowercase letters) and dissimilar (non-confounding and discrete uppercase letters, respectively) control tasks. ResultsResults revealed significant differences in FRP amplitudes between DYS and CAC groups under the phonologically similar and phonologically non-confounding conditions. No differences were observed in the case of the visual conditions. Moreover, regression analysis showed that the average amplitude of the extracted components significantly predicted RAN performance. ConclusionFRPs capture neural components during the serial RAN task informative of differences between DYS and CAC and establish a relationship between neurocognitive processes during serial RAN and dyslexia. SignificanceWe suggest our approach as a methodological model for the concurrent analysis of neurophysiological and eye-gaze data to decipher the role of RAN in reading.

A visual and neural evaluation of the affective impression on humanoid robot appearances in free viewing

This study aims to characterize the fixation pattern and neutral dynamics underlying affective impressions on humanoid robot appearances, which have a moderate anthropomorphism, to provide an objective evaluation of the affective impressions. Thirty appearance pictures of existing humanoid robots leaving negative, neutral, or positive impressions were selected online. Users’ eye-tracking and electroencephalography signals for the appearances were co-registered in a free viewing paradigm, and eye-tracking metrics and eye fixation-related potentials were extracted. The results showed that the head attracted the most eye-tracking metrics, including glance count, fixation count, fixation time, and fixation duration. The torso attracted the second most, the legs and the hands attracted the third most, and the arms and feet drew the least. Humanoid robot appearances giving positive impressions caught less glance counts and shorter fixation duration than the appearances giving negative or neutral impressions on both partial and whole appearances. The appearances giving positive impressions evoked the smallest P1 amplitude, and the appearances giving neutral impressions evoked the largest P1 amplitude, compared with that giving negative impressions. The findings suggest that glance count and fixation duration could distinguish the positive impression on humanoid robot appearances, and the P1 amplitude could distinguish the three affective impressions. Relevance to industryThese findings could promote the understanding of fixation pattern on humanoid robot appearances, which could be referred to directly when designing appearances for humanoid robots. Moreover, this study could provide designers with a visual and neural method of evaluating affective impressions on humanoid robot appearances in free viewing.