Eye Fixation-related Potentials Research Articles

Assessment of driver’s attentional resource allocation to visual, cognitive, and action processing by brain and eye signals

Driving a vehicle is comprised of multiple tasks (e.g., monitoring the environment around the vehicle, planning the trajectory, and controlling the vehicle), and requires the allocation of capacity-limited attentional resources to visual, cognitive, and action processing; otherwise, the quality of task performance will deteriorate, increasing the risk of near-accidents or crashes. The present study proposes that variations in the total amount as well as the individual amounts of attentional resources allocated to visual, cognitive, and action processing depending on the driving situations could be objectively estimated by the combined use of three physiological measures: (1) the duration of eye blinks during driving, (2) the size of eye-fixation-related potentials (EFRPs), i.e., event-related potentials (ERPs) that are time-locked to the offset of saccadic eye-movements during driving, and (3) the size of auditory-evoked potentials (AEPs), i.e., ERPs time-locked to the onset of task-unrelated auditory stimuli discretely presented during driving. We implemented these measures when participants (N = 16) drove a vehicle on a slalom course under four driving conditions defined by a combination of two levels of speed requirement (fast and slow) and two levels of path width (narrow and wide). The findings suggested that, (1) when driving at fast compared to slow speeds, the total amount of resources allocated to overall processing increased, which consisted of an increase in the amount of resources allocated to cognitive (and possibly action) processing and a decrease in the amount of resources allocated to visual processing, and (2) when driving on narrow compared to wide paths, the total amount of resources allocated to overall processing remained almost the same (due to complementary speed reduction), which consisted of an increase in the amount of resources allocated to visual processing, a decrease in the amount of resources allocated to cognitive processing, and almost the same amount of resources allocated to action processing. The driver’s resource management strategies indicated by these results as well as the utility and limitations of the proposed method are discussed.

Eye-Fixation-Related Potentials (EFRPs) As a Predictor of Human Error Occurrences During a Visual Inspection Task.

Estimation of human attentional states using an electroencephalogram (EEG) has been demonstrated to help prevent human errors associated with the degradation. Since the use of the lambda response -one of eye-fixation-related potentials time-locked to the saccade offset- enables such estimation without external triggers, the measurements are compatible for an application in a real-world environment. With aiming to apply the lambda response as an index of human errors during the visual inspection, the current research elucidated whether the mean amplitude of the lambda response was a predictor of the number of inspection errors. EEGs were measured from 50 participants while inspecting the differences between two images of the circuit board. Twenty percent of the total number of image pairs included differences. The lambda response was obtained relative to a saccade offset starting a fixation of the inspection image. Participants conducted four sessions over two days (625 trials/ session, 2 sessions/ day). A Poisson regression of the number of inspection errors using a generalized linear mixed model showed that a coefficient of the mean amplitude of the lambda response was significant , suggesting that the response has a role in th$(\hat \beta = 0.24,p < 0.01)$e prediction of the number of human error occurrences in the visual inspection.

A visual and neural evaluation of the affective impression on humanoid robot appearances in free viewing

This study aims to characterize the fixation pattern and neutral dynamics underlying affective impressions on humanoid robot appearances, which have a moderate anthropomorphism, to provide an objective evaluation of the affective impressions. Thirty appearance pictures of existing humanoid robots leaving negative, neutral, or positive impressions were selected online. Users’ eye-tracking and electroencephalography signals for the appearances were co-registered in a free viewing paradigm, and eye-tracking metrics and eye fixation-related potentials were extracted. The results showed that the head attracted the most eye-tracking metrics, including glance count, fixation count, fixation time, and fixation duration. The torso attracted the second most, the legs and the hands attracted the third most, and the arms and feet drew the least. Humanoid robot appearances giving positive impressions caught less glance counts and shorter fixation duration than the appearances giving negative or neutral impressions on both partial and whole appearances. The appearances giving positive impressions evoked the smallest P1 amplitude, and the appearances giving neutral impressions evoked the largest P1 amplitude, compared with that giving negative impressions. The findings suggest that glance count and fixation duration could distinguish the positive impression on humanoid robot appearances, and the P1 amplitude could distinguish the three affective impressions. Relevance to industryThese findings could promote the understanding of fixation pattern on humanoid robot appearances, which could be referred to directly when designing appearances for humanoid robots. Moreover, this study could provide designers with a visual and neural method of evaluating affective impressions on humanoid robot appearances in free viewing.

Closed-loop EEG study on visual recognition during driving

Objective. In contrast to the classical visual brain–computer interface (BCI) paradigms, which adhere to a rigid trial structure and restricted user behavior, electroencephalogram (EEG)-based visual recognition decoding during our daily activities remains challenging. The objective of this study is to explore the feasibility of decoding the EEG signature of visual recognition in experimental conditions promoting our natural ocular behavior when interacting with our dynamic environment. Approach. In our experiment, subjects visually search for a target object among suddenly appearing objects in the environment while driving a car-simulator. Given that subjects exhibit an unconstrained overt visual behavior, we based our study on eye fixation-related potentials (EFRPs). We report on gaze behavior and single-trial EFRP decoding performance (fixations on visually similar target vs. non-target objects). In addition, we demonstrate the application of our approach in a closed-loop BCI setup. Main results. To identify the target out of four symbol types along a road segment, the BCI system integrated decoding probabilities of multiple EFRP and achieved the average online accuracy of 0.37 ± 0.06 (12 subjects), statistically significantly above the chance level. Using the acquired data, we performed a comparative study of classification algorithms (discriminating target vs. non-target) and feature spaces in a simulated online scenario. The EEG approaches yielded similar moderate performances of at most 0.6 AUC, yet statistically significantly above the chance level. In addition, the gaze duration (dwell time) appears to be an additional informative feature in this context. Significance. These results show that visual recognition of sudden events can be decoded during active driving. Therefore, this study lays a foundation for assistive and recommender systems based on the driver’s brain signals.

Eye Fixation-Related Potentials during Visual Search on Acquaintance and Newly-Learned Faces.

Searching familiar faces in the crowd may involve stimulus-driven attention by emotional significance, together with goal-directed attention due to task-relevant needs. The present study investigated the effect of familiarity on attentional processes by exploring eye fixation-related potentials (EFRPs) and eye gazes when humans searched for, among other distracting faces, either an acquaintance’s face or a newly-learned face. Task performance and gaze behavior were indistinguishable for identifying either faces. However, from the EFRP analysis, after a P300 component for successful search of target faces, we found greater deflections of right parietal late positive potentials in response to newly-learned faces than acquaintance’s faces, indicating more involvement of goal-directed attention in processing newly-learned faces. In addition, we found greater occipital negativity elicited by acquaintance’s faces, reflecting emotional responses to significant stimuli. These results may suggest that finding a familiar face in the crowd would involve lower goal-directed attention and elicit more emotional responses.

The effects of cognitive distraction on behavioural, oculomotor and electrophysiological metrics during a driving hazard perception task

Previous research has demonstrated that the distraction caused by holding a mobile telephone conversation is not limited to the period of the actual conversation (Haigney, 1995; Redelmeier & Tibshirani, 1997; Savage et al., 2013). In a prior study we identified potential eye movement and EEG markers of cognitive distraction during driving hazard perception. However the extent to which these markers are affected by the demands of the hazard perception task are unclear. Therefore in the current study we assessed the effects of secondary cognitive task demand on eye movement and EEG metrics separately for periods prior to, during and after the hazard was visible. We found that when no hazard was present (prior and post hazard windows), distraction resulted in changes to various elements of saccadic eye movements. However, when the target was present, distraction did not affect eye movements.We have previously found evidence that distraction resulted in an overall decrease in theta band output at occipital sites of the brain. This was interpreted as evidence that distraction results in a reduction in visual processing. The current study confirmed this by examining the effects of distraction on the lambda response component of subjects eye fixation related potentials (EFRPs). Furthermore, we demonstrated that although detections of hazards were not affected by distraction, both eye movement and EEG metrics prior to the onset of the hazard were sensitive to changes in cognitive workload. This suggests that changes to specific aspects of the saccadic eye movement system could act as unobtrusive markers of distraction even prior to a breakdown in driving performance.

Neural underpinnings of value-guided choice during auction tasks: An eye-fixation related potentials study

Values are attributed to goods during free viewing of objects which entails multi- and trans-saccadic cognitive processes. Using electroencephalographic eye-fixation related potentials, the present study investigated how neural signals related to value-guided choice evolved over time when viewing household and office products during an auction task. Participants completed a Becker-DeGroot-Marschak auction task whereby half of the stimuli were presented in either a free or forced bid protocol to obtain willingness-to-pay. Stimuli were assigned to three value categories of low, medium and high value based on subjective willingness-to-pay. Eye fixations were organised into five 800 ms time-bins spanning the objects total viewing time. Independent component analysis was applied to eye-fixation related potentials. One independent component (IC) was found to represent fixations for high value products with increased activation over the left parietal region of the scalp. An IC with a spatial maximum over a fronto-central region of the scalp coded the intermediate values. Finally, one IC displaying activity that extends over the right frontal scalp region responded to intermediate- and low-value items. Each of these components responded early on during viewing an object and remained active over the entire viewing period, both during free and forced bid trials. Results suggest that the subjective value of goods are encoded using sets of brain activation patterns which are tuned to respond uniquely to either low, medium, or high values. Data indicates that the right frontal region of the brain responds to low and the left frontal region to high values. Values of goods are determined at an early point in the decision making process and carried for the duration of the decision period via trans-saccadic processes.

The Effect of Art Expertise on Eye Fixation-Related Potentials During Aesthetic Judgment Task in Focal and Ambient Modes.

This study aimed to determine the effect of expertise on the eye fixation-related potentials (EFRPs) during the aesthetic evaluation of images, independently in focal and ambient modes of visual processing. Focal and ambient modes were identified by averaging EFRP waveforms about the beginning of long eye fixations followed by short saccades and short fixations followed by long saccades, respectively. Thirty experts with formal training in visual arts and thirty-two non-experts freely viewed 150 figurative paintings presented for 20 s, each. After viewing the painting, the participant answered the question: “Is this painting beautiful?” Differences were found between the group of experts and non-experts due to the amplitude of EFRPs but only in focal mode, which is related to top-down, focused attention on the objects. Long fixations of experts had a higher amplitude of the parietal P2 recorded from right site than non-experts. In the group of experts, the frontal P2 was higher for long fixations on not beautiful paintings in comparison to long fixation on beautiful paintings. Moreover, in focal mode, there were higher occipital lambda response and N1-P2 complex for not beautiful than beautiful paintings. These results are discussed in the light of the results of studies on the effect of visual art expertise on event-related potentials (ERPs), ERP studies during aesthetic judgment task, and the knowledge of different modes of visual processing and EFRPs.

Temporal Dynamics of Natural Static Emotional Facial Expressions Decoding: A Study Using Event- and Eye Fixation-Related Potentials.

This study aims at examining the precise temporal dynamics of the emotional facial decoding as it unfolds in the brain, according to the emotions displayed. To characterize this processing as it occurs in ecological settings, we focused on unconstrained visual explorations of natural emotional faces (i.e., free eye movements). The General Linear Model (GLM; Smith and Kutas, 2015a,b; Kristensen et al., 2017a) enables such a depiction. It allows deconvolving adjacent overlapping responses of the eye fixation-related potentials (EFRPs) elicited by the subsequent fixations and the event-related potentials (ERPs) elicited at the stimuli onset. Nineteen participants were displayed with spontaneous static facial expressions of emotions (Neutral, Disgust, Surprise, and Happiness) from the DynEmo database (Tcherkassof et al., 2013). Behavioral results on participants’ eye movements show that the usual diagnostic features in emotional decoding (eyes for negative facial displays and mouth for positive ones) are consistent with the literature. The impact of emotional category on both the ERPs and the EFRPs elicited by the free exploration of the emotional faces is observed upon the temporal dynamics of the emotional facial expression processing. Regarding the ERP at stimulus onset, there is a significant emotion-dependent modulation of the P2–P3 complex and LPP components’ amplitude at the left frontal site for the ERPs computed by averaging. Yet, the GLM reveals the impact of subsequent fixations on the ERPs time-locked on stimulus onset. Results are also in line with the valence hypothesis. The observed differences between the two estimation methods (Average vs. GLM) suggest the predominance of the right hemisphere at the stimulus onset and the implication of the left hemisphere in the processing of the information encoded by subsequent fixations. Concerning the first EFRP, the Lambda response and the P2 component are modulated by the emotion of surprise compared to the neutral emotion, suggesting an impact of high-level factors, in parieto-occipital sites. Moreover, no difference is observed on the second and subsequent EFRP. Taken together, the results stress the significant gain obtained in analyzing the EFRPs using the GLM method and pave the way toward efficient ecological emotional dynamic stimuli analyses.

An Eye Fixation-Related Potential Study in Two Reading Tasks: Reading to Memorize and Reading to Make a Decision.

We investigated how two different reading tasks, namely reading to memorize [Read & Memorize (RM)] and reading to decide whether a text was relevant to a given topic [Read & Decide (RD)], modulated both eye movements (EM) and brain activity. To this end, we set up an ecological paradigm using the eye fixation-related potentials (EFRP) technique, in which participants freely moved their eyes to process short paragraphs, while their electroencephalography (EEG) activity was recorded in synchronization with their EM. A general linear model was used to estimate at best EFRP, taking account of the overlap between adjacent potentials, and more precisely with the potential elicited at text onset, as well as saccadic potentials. Our results showed that EM patterns were top-down modulated by different task demands. More interestingly, in both tasks, we observed slow-wave potentials that gradually increased across the first eye fixations. These slow waves were larger in the RD task than in the RM task, specifically over the left hemisphere. These results suggest that the decision-making process during reading in the RD task engendered a greater memory load in working memory than that generated in a classic reading task. The significance of these findings is discussed in the light of recent theories and models of working memory processing.

Estimation of overlapped Eye Fixation Related Potentials: The General Linear Model, a more flexible framework than the ADJAR algorithm.

The Eye Fixation Related Potential (EFRP) estimation is the average of EEG signals across epochs at ocular fixation onset. Its main limitation is the overlapping issue. Inter Fixation Intervals (IFI) - typically around 300 ms in the case of unrestricted eye movement- depend on participants’ oculomotor patterns, and can be shorter than the latency of the components of the evoked potential. If the duration of an epoch is longer than the IFI value, more than one fixation can occur, and some overlapping between adjacent neural responses ensues. The classical average does not take into account either the presence of several fixations during an epoch or overlapping. The Adjacent Response algorithm (ADJAR), which is popular for event-related potential estimation, was compared to the General Linear Model (GLM) on a real dataset from a conjoint EEG and eye-tracking experiment to address the overlapping issue. The results showed that the ADJAR algorithm was based on assumptions that were too restrictive for EFRP estimation. The General Linear Model appeared to be more robust and efficient. Different configurations of this model were compared to estimate the potential elicited at image onset, as well as EFRP at the beginning of exploration. These configurations took into account the overlap between the event-related potential at stimulus presentation and the following EFRP, and the distinction between the potential elicited by the first fixation onset and subsequent ones. The choice of the General Linear Model configuration was a tradeoff between assumptions about expected behavior and the quality of the EFRP estimation: the number of different potentials estimated by a given model must be controlled to avoid erroneous estimations with large variances.

Individual differences in reading strategies: an Eye Fixation-Related Potentials study

Individual differences in reading strategies: an Eye Fixation-Related Potentials study

An eye fixation-related potentials analysis of the P300 potential for fixations onto a target object when exploring natural scenes.

The P300 event-related potential has been extensively studied in electroencephalography with classical paradigms that force observers to not move their eyes. This potential is classically used to infer whether a target or a task-relevant stimulus was presented. Few researches have studied this potential through more ecological paradigms where observers were able to move their eyes. In this study, we examined with an ecological paradigm and an adapted methodology the P300 potential using a visual search task that involves eye movements to actively explore natural scenes and during which eye movements and electroencephalographic activity were coregistered. Averaging the electroencephalography signal time-locked to fixation onsets, a P300 potential was observed for fixations onto the target object but not for other fixations recorded for the same visual search or for fixations recorded during the free viewing without any task. Our approach consists of using control experimental conditions with similar eye movements to ensure that the P300 potential was attributable to the fact that the observer gazed at the target rather than to other factors such as eye movement pattern (the size of the previous saccade) or the "overlap issue" between the potentials elicited by two successive fixations. We also proposed to model the time overlap issue of the potentials elicited by consecutive fixations with various durations. Our results show that the P300 potential can be studied in ecological situations without any constraint on the type of visual exploration, with some precautions in the interpretation of results due to the overlap issue.

Time course of the P300 Eye-Fixation Related Potential during the visual search for a target embedded in natural scenes

The Event Related Potential P300 has been intensively studied for over forty years (Polish, 2012). It is elicited at each updating of the stimulus representation, reflecting a cascade of cognitive processes engaging attentional and memory mechanisms. This potential is a positive component, appearing 300ms after stimulus onset, with maximal amplitude in centro-parietal regions. This study is mainly concerned by the analysis of the appearance and evolution of the P300 elicited by the onset of successive fixations recorded during a visual search using an ecological experimental paradigm. This goal is achieved through the co-registration of electroencephalographic (EEG) and eye-tracking (ET) signals. EEG and ET signals were recorded for thirty-four observers when exploring natural scenes during 5 sec to answer if a target object was or not present. To solve the main difficulty due to overlaps between the observed Eye Fixation Related Potentials (EFRP) elicited by consecutive fixations, we have used a new methodology based on the xDawn algorithm (Rivet, et al.; 2009). From each trial, a temporal window of 2s centered on the first fixation inside the Region of Interest (target). This duration was chosen to include five consecutive fixations (two before the first entrance, and two after). The fixations were tagged depending to their rank and position (inside/outside) the ROI. Neural activities were modelled as a cumulative response of specific latent responses elicited by each fixation with the corresponding onset. After xDawn estimation, the topographic maps have shown a potential identified as the P300 component elicited by the visual input through the consecutive fixations landed on the ROI. This result confirms previous studies but has been obtained for the first time on ecological paradigm thanks to the co-registration of EEG and ET signals and a new methodology to denoise potentials from overlaps of potentials elicited by consecutive fixations. Meeting abstract presented at VSS 2015

Physiological responses related to moderate mental load during car driving in field conditions

We measured physiological variables on nine car drivers to capture moderate magnitudes of mental load (ML) during driving in prolonged and repeated city and highway field conditions. Ecological validity was optimized by avoiding any artificial interference to manipulate drivers ML, drivers were alone in the car, they were free to choose their paths to the target, and the repeated drives familiarized drivers to the procedure. Our aim was to investigate if driver's physiological variables can be reliably measured and used as predictors of moderate individual levels of ML in naturally occurring unpredictably changing field conditions. Variables investigated were: heart-rate, skin conductance level, breath duration, blink frequency, blink duration, and eye fixation related potentials. After the drives, with support from video uptakes, a self-rating and a score made by external raters were used to distinguish moderately high and low ML segments. Variability was high but aggregated data could distinguish city from highway drives. Multivariate models could successfully classify high and low ML within highway and city drives using physiological variables as input. In summary, physiological variables have a potential to be used as indicators of moderate ML in unpredictably changing field conditions and to advance the evaluation and development of new active safety systems.

Affective processing in natural scene viewing: Valence and arousal interactions in eye-fixation-related potentials

Attention is drawn to emotionally salient stimuli. The present study investigates processing of emotionally salient regions during free viewing of emotional scenes that were categorized according to the two-dimensional model comprising of valence (unpleasant, pleasant) and arousal (high, low). Recent studies have reported interactions between these dimensions, indicative of stimulus-evoked approach or withdrawal tendencies. We addressed the valence and arousal effects when emotional items were embedded in complex real-world scenes by analyzing both eye movement behavior and eye-fixation-related potentials (EFRPs) time-locked to the critical event of fixating the emotionally salient items for the first time. Both data sets showed an interaction between the valence and arousal dimensions. First, the fixation rates and gaze durations on emotionally salient regions were enhanced for unpleasant versus pleasant images in the high arousal condition. In the low arousal condition, both measures were enhanced for pleasant versus unpleasant images. Second, the EFRP results at 140–170ms [P2] over the central site showed stronger responses for high versus low arousing images in the unpleasant condition. In addition, the parietal LPP responses at 400–500ms post-fixation were enhanced for stimuli reflecting congruent stimulus dimensions, that is, stronger responses for high versus low arousing images in the unpleasant condition and stronger responses for low versus high arousing images in the pleasant condition. The present findings support the interactive two-dimensional approach, according to which the integration of valence and arousal recruits brain regions associated with action tendencies of approach or withdrawal.

Precision is in the Eye of the Beholder: Application of Eye Fixation-Related Potentials to Information Systems Research

Precision is in the Eye of the Beholder: Application of Eye Fixation-Related Potentials to Information Systems Research

Determinants of attentive blank stares. An EFRP study.

Attentive blank stares mean a failure to notice changes in a visual scene, despite looking at the area of change (Caplovitz, Fendrich, & Hughes, 2008). In this research project we have shown that people differ in terms of attentive blank stare incidences. Novices tend to fail to notice changes in the target area more often than experts. This effect is greater in persons with low visual working memory capacity (VWMC) than with high VWMC. In addition, in a group of novices with low VWMC, attentive blank stares are more frequent compared to a group with high VWMC. Attentive blank stares did not disappear even after the high VWMC group were given expertise training. With the method of eye-fixation-related potentials (EFRP) we analyzed the amplitude of lambda response, which may reflect the state of the attentional system, during encoding information about a change, prior to a decision whether a change has occurred or not. We demonstrate that the cases of attentive blank stares are accompanied by significantly lower amplitude of the lambda response compared with cases involving change detection. In addition, we discovered greater lambda responses in a group with expertise who noticed the change than in novices. The EFRP record coming from occipital electrodes in the 80-180ms window function was marked by left-sided asymmetry in the cases of change detection and by right-sided asymmetry in the cases of attentive blank stares.

Eye movement-related brain activity during perceptual and cognitive processing.

Eye movement-related brain activity during perceptual and cognitive processing.

Eye-fixation related potentials on regions of interest when viewing natural scenes

EFRPs (Eye-Fixation Related Potentials): Segmentation of EEG data into epochs from 500 ms before the scene onset to 4000 ms after Rejection of segments contaminated with blinks, muscular activity and/or nonphysiological artifacts Correction of artifacts relating to ocular movements (IVA) [Phlypo, 2013] EFRPs: [-200 ms;700 ms] according to fixation onset Baseline : [-200 ms;-100 ms] Recordings Eye movements: SR Research Eyelink 1000 EEG: 32 active electrodes g.GAMMAsys gtec system