Eye Fixations Research Articles

Gaze Mapping for Immersive Virtual Environments Based on Image Retrieval

In this paper, we introduce a novel gaze mapping approach for free viewing conditions in dynamic immersive virtual environments (VEs), which projects recorded eye fixation data of users, who viewed the VE from different perspectives, to the current view. This generates eye fixation maps, which can serve as ground truth for training machine learning (ML) models to predict saliency and the user’s gaze in immersive virtual reality (VR) environments. We use a flexible image retrieval approach based on SIFT features, which can also map the gaze under strong viewpoint changes and dynamic changes. A vocabulary tree enables to scale to the large amounts of data with typically several hundred thousand frames and a homography transform re-projects the fixations to the current view. To evaluate our approach, we measure the predictive quality of our eye fixation maps to model the gaze of the current user and compare our maps to computer-generated saliency maps on the DGaze and the Saliency in VR datasets. The results show that our method often outperform these saliency predictors. However, in contrast to these methods, our approach collects real fixations from human observers, and can thus serve to estimate ground truth fixation maps in dynamic VR environments, which can be used to train and evaluate gaze predictors.

Damage to the human dentate gyrus impairs the perceptual discrimination of complex, novel objects

The hippocampus (HPC), and the dentate gyrus (DG) subregion in particular, is purported to be a pattern separator, orthogonally representing similar information so that distinct memories may be formed. The HPC may also be involved in complex perceptual discrimination. It is unclear if this role is limited to spatial/scene stimuli or extends to the discrimination of objects. Also unclear is whether the DG itself contributes to pattern separation beyond memory. BL, an individual with bilateral DG lesions, was previously shown to have poor discrimination of similar, everyday objects in memory. Here, we demonstrate that BL's deficit extends to complex perceptual discrimination of novel objects. Specifically, BL was presented with closely matched possible and impossible objects, which give rise to fundamentally different 3D perceptual representations despite being visually similar. BL performed significantly worse than controls when asked to select an odd object (e.g., impossible) amongst three identical counterpart objects (e.g., possible) presented at different rotations. His deficit was also evident in an atypical eye fixation pattern during this task. In contrast, BL's performance was indistinguishable from that of controls on other tasks involving the same objects, indicating that he could visually differentiate the object pairs, that he perceived the objects holistically in 3D, and that he has only a mild weakness in categorizing object possibility. Furthermore, his performance on standardized neuropsychological measures indicated intact mental rotation, visual-spatial attention, and working memory (visual and auditory). Collectively, these results provide evidence that the DG is necessary for complex perceptual discrimination of novel objects, indicating that the DG might function as a generic pattern separator of a wide range of stimuli within high-level perception, and that its role is not limited to memory.

The role of eye tracker in teaching video-assisted thoracoscopic surgery: the differences in visual strategies between novice and expert surgeons in thoracoscopic surgery.

BackgroundIn light of continued advances in surgical techniques, it is essential to improve the efficacy of training young thoracic surgeons. Increasingly, the feedback of visual information is believed to be a crucial teaching tool. When surgeons are looking at surgical teaching videos, eye gaze technology can be used to record the gaze movement to examine the stages of information access.MethodsA total of 8 novice surgeons and 8 expert surgeons were recruited to watch an instructional video of surgery while Tobii Pro Fusion was used to recorded their movement trajectory. Another 36 novice surgeons were recruited and divided into two groups. One group watched the video following the trajectory of the senior thoracic surgeons’ eye movements and the other group watched the raw video. All 36 novice surgeons subsequently underwent a post-test which was conducted via a questionnaire.ResultsThe differences in eye fixation patterns and search strategies between novices and experts were assessed using 4 indexes, including total duration of fixation, time to entry saccade, number of saccades, and peak velocity of saccade. The results were compared using independent-sample t-tests and repeated-measures analysis. When experts watched the videos, there was a 67.8% overlap of eye gaze compared to a 51.4% overlap in novices (P≤0.001). The post-test showed that novice surgeons who watched the video marked with the senior thoracic surgeon’s eye movements had a better understanding of the key points of the operation.ConclusionsNovice and expert surgeons have different eye fixation patterns and search strategies when watching videos of thoracoscopic surgery. Watching surgical videos marked with expert eye movement trajectories is conducive to novice surgeons to master operative skills.

Uncovering the course of analogical mapping using eye tracking

Analogical mapping – the core component of analogical reasoning – consists of establishing the relational structure shared by two analogous situations and inferring the missing elements in a less familiar situation from a more familiar one. Several existing models of analogy predicted that the complete relational structure can be considered in parallel. Other models postulated that mapping can be less or more incremental – it can access only a relatively small part of the structure, and needs to move to its other parts in steps in order to construct the final relational correspondence. However, the precise time course of analogical mapping, especially in sufficiently complex analogies, to date was rarely studied empirically. In two studies, eye tracking was used to assess in a rigorous way the extent to which mapping can be incremental. In a newly designed geometric A:B::C:D task, pattern D was generated from C according to the same shape transformations that generated pattern B from A. The six possible response options differed systematically in the number of correct transformations, from no transformation matching, via partial relational match, up to the full match. In Study 1, the relational match of options fixated on by participants was initially low but increased monotonically over the course of analogy. The number of corresponding eye fixations predicted 68% variance in relational match of the final response. The correct option was chosen only if fixated on for a sufficiently long time. Study 2 replicated the findings using a more ecologically valid and less demanding task variant that required to map the changes in people's appearance. The results support these theoretical models of analogy which postulate strictly incremental mapping.

Prediction of Human-Computer Interaction Intention Based on Eye Movement and Electroencephalograph Characteristics.

In order to solve the problem of unsmooth and inefficient human-computer interaction process in the information age, a method for human-computer interaction intention prediction based on electroencephalograph (EEG) signals and eye movement signals is proposed. This approach is different from previous methods where researchers predict using data from human-computer interaction and a single physiological signal. This method uses the eye movements and EEG signals that clearly characterized the interaction intention as the prediction basis. In addition, this approach is not only tested with multiple human-computer interaction intentions, but also takes into account the operator in different cognitive states. The experimental results show that this method has some advantages over the methods proposed by other researchers. In Experiment 1, using the eye movement signal fixation point abscissa Position X (PX), fixation point ordinate Position Y (PY), and saccade amplitude (SA) to judge the interaction intention, the accuracy reached 92%, In experiment 2, only relying on the pupil diameter, pupil size (PS) and fixed time, fixed time (FD) of eye movement signals can not achieve higher accuracy of the operator’s cognitive state, so EEG signals are added. The cognitive state was identified separately by combining the screened EEG parameters Rα/β with the eye movement signal pupil diameter and fixation time, with an accuracy of 91.67%. The experimental combination of eye movement and EEG signal features can be used to predict the operator’s interaction intention and cognitive state.

The Presence of a Visual Dividing Line Increases Consumer Memory Through Attention Grabbing.

Marketers often use a visual line to divide the product information on an advertisement into left-right (or top-bottom) segments for aesthetic or categorization purposes. The present research examined the effect of the dividing line on the consumer memory. Across three studies (including an eye-tracking study and a field one), we showed that the presence of a dividing line enhances consumers’ memory about the products displayed on the left/top of an advertisement. This effect occurs because the dividing line orients participants’ first eye fixation to the left/top area of the advertisement, such that their visual attention is largely restricted to that area and they could better remember the contents displayed on that area. The theoretical contributions and implications for marketers and consumers are discussed.

CFN: A coarse‐to‐fine network for eye fixation prediction

Many image-to-image computer vision approaches have made great progress by an end-to-end framework with the encoder–decoder architecture. However, the same image-to-image eye fixation prediction task is not the same as those computer vision tasks in that it focuses more on salient regions rather than precise predictions for every pixel. Thus, it is not appropriate to directly apply the end-to-end encoder–decoder to the eye fixation prediction task. In addition, although high-level feature is important, the contribution of low-level feature should also be kept and balanced in computational model. Nevertheless, some low-level features that attract attention are easily neglected while transiting through the deep network. Therefore, the effective way to integrate low-level and high-level features for improving eye fixation prediction performance is still a challenging task. In this paper, a coarse-to-fine network (CFN) that encompasses two pathways with different training strategies are proposed: coarse perceiving network (CFN-Coarse) can be a simple encoder network or any of the existing pretrained network to capture the distribution of salient regions and generate high-quality feature maps; fine integrating network (CFN-Fine) uses fixed parameters from the CFN-Coarse and combines features from deep to shallow in the deconvolution process by adding skip connections between down-sampling and up-sampling paths to efficiently integrate deep and shallow features. The saliency map obtained by the method is evaluated over 6 standard benchmark datasets, namely SALICON, MIT1003, MIT300, Toronto, OSIE, and SUN500. The results demonstrate that the method can surpass the state-of-the-art accuracy of eye fixation prediction and achieves the competitive performance to date under most evaluation metrics on SALICON Saliency Prediction Challenge (LSUN2017).

Object Categorization Capability of Psychological Potential Field in Perceptual Assessment Using Line-Drawing Images.

Affective/cognitive engineering investigations typically require the quantitative assessment of object perception. Recent research has suggested that certain perceptions of object categorization can be derived from human eye fixation and that color images and line drawings induce similar neural activities. Line drawings contain less information than color images; therefore, line drawings are expected to simplify the investigations of object perception. The psychological potential field (PPF), which is a psychological feature, is an image feature of line drawings. On the basis of the PPF, the possibility that the general human perception of object categorization can be assessed from the similarity to fixation maps (FMs) generated from human eye fixations has been reported. However, this may be due to chance because image features other than the PPF have not been compared with FMs. This study examines the potential and effectiveness of the PPF by comparing its performance with that of other image features in terms of the similarity to FMs. The results show that the PPF shows the ideal performance for assessing the perception of object categorization. In particular, the PPF effectively distinguishes between animal and nonanimal targets; however, real-time assessment is difficult.

Application of Machine Learning Techniques to Detect the Children with Autism Spectrum Disorder.

Early detection of autism spectrum disorder (ASD) is highly beneficial to the health sustainability of children. Existing detection methods depend on the assessment of experts, which are subjective and costly. In this study, we proposed a machine learning approach that fuses physiological data (electroencephalography, EEG) and behavioral data (eye fixation and facial expression) to detect children with ASD. Its implementation can improve detection efficiency and reduce costs. First, we used an innovative approach to extract features of eye fixation, facial expression, and EEG data. Then, a hybrid fusion approach based on a weighted naive Bayes algorithm was presented for multimodal data fusion with a classification accuracy of 87.50%. Results suggest that the machine learning classification approach in this study is effective for the early detection of ASD. Confusion matrices and graphs demonstrate that eye fixation, facial expression, and EEG have different discriminative powers for the detection of ASD and typically developing children, and EEG may be the most discriminative information. The physiological and behavioral data have important complementary characteristics. Thus, the machine learning approach proposed in this study, which combines the complementary information, can significantly improve classification accuracy.

Individual differences in the allocation of visual attention during navigation.

Individual differences exist in the ability to create an accurate mental survey representation (i.e., a cognitive map) of a novel environment, yet the mechanisms underlying differences in cognitive map accuracy are still under investigation. To determine whether differences in overt attention allocation contribute to these individual differences, the current study examined whether looking times to landmarks and other objects while navigating in a dynamic virtual environment were related to cognitive map accuracy. Participants completed a battery of spatial tests; some tests assessed spatial skills prior to the navigation task (the Santa Barbara Sense of Direction Scale and the Spatial Orientation Test), and others tested memory of the virtual environment Silcton after an exploration period (a landmark recognition task, a direction estimation task, a map-building task, and a route construction task). Individuals with inaccurate cognitive maps of Silcton, as measured by the direction estimation and map-building tasks, showed equivalent eye fixations to buildings and objects when exploring Silcton as those with accurate maps. Despite similar looking times, the inaccurate mappers were significantly worse at judgments of relative direction between landmarks in Silcton and showed poorer memory for landmarks in Silcton than accurate mappers. These findings suggest that cognitive mechanisms, such as mental perspective-taking, occurring after attention allocation underlie differences in cognitive map accuracy. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Children’s Ability to Read from Computers and Smartphones

Children now use digital devices for learning and leisure activities. The study assessed children's ability to read from computers, smartphones and printed sheets. We provided 60 children aged seven to ten years with reading material and recorded the time taken by them to complete reading and their navigation pattern and eye movements using software developed by us. There was no significant difference in the time taken to complete reading and the ability to recall the content when the children read from the three mediums ( P > .05). The average eye fixation duration was however significantly higher when the children read from a smartphone ( P < .05). Children aged nine and ten years used hyperlinks more often while reading from a smartphone than a computer ( P < .05). We concluded that children can read fluently from all the three mediums by the age of seven years, and navigate hypertext effectively on smartphones by nine years of age.

Unsupervised Text Segmentation Predicts Eye Fixations During Reading.

Words typically form the basis of psycholinguistic and computational linguistic studies about sentence processing. However, recent evidence shows the basic units during reading, i.e., the items in the mental lexicon, are not always words, but could also be sub-word and supra-word units. To recognize these units, human readers require a cognitive mechanism to learn and detect them. In this paper, we assume eye fixations during reading reveal the locations of the cognitive units, and that the cognitive units are analogous with the text units discovered by unsupervised segmentation models. We predict eye fixations by model-segmented units on both English and Dutch text. The results show the model-segmented units predict eye fixations better than word units. This finding suggests that the predictive performance of model-segmented units indicates their plausibility as cognitive units. The Less-is-Better (LiB) model, which finds the units that minimize both long-term and working memory load, offers advantages both in terms of prediction score and efficiency among alternative models. Our results also suggest that modeling the least-effort principle for the management of long-term and working memory can lead to inferring cognitive units. Overall, the study supports the theory that the mental lexicon stores not only words but also smaller and larger units, suggests that fixation locations during reading depend on these units, and shows that unsupervised segmentation models can discover these units.

Toward Explainable Artificial Intelligence for Early Anticipation of Traffic Accidents

Traffic accident anticipation is a vital function of Automated Driving Systems (ADS) in providing a safety-guaranteed driving experience. An accident anticipation model aims to predict accidents promptly and accurately before they occur. Existing Artificial Intelligence (AI) models of accident anticipation lack a human-interpretable explanation of their decision making. Although these models perform well, they remain a black-box to the ADS users who find it to difficult to trust them. To this end, this paper presents a gated recurrent unit (GRU) network that learns spatio-temporal relational features for the early anticipation of traffic accidents from dashcam video data. A post-hoc attention mechanism named Grad-CAM (Gradient-weighted Class Activation Map) is integrated into the network to generate saliency maps as the visual explanation of the accident anticipation decision. An eye tracker captures human eye fixation points for generating human attention maps. The explainability of network-generated saliency maps is evaluated in comparison to human attention maps. Qualitative and quantitative results on a public crash data set confirm that the proposed explainable network can anticipate an accident on average 4.57 s before it occurs, with 94.02% average precision. Various post-hoc attention-based XAI methods are then evaluated and compared. This confirms that the Grad-CAM chosen by this study can generate high-quality, human-interpretable saliency maps (with 1.23 Normalized Scanpath Saliency) for explaining the crash anticipation decision. Importantly, results confirm that the proposed AI model, with a human-inspired design, can outperform humans in accident anticipation.

Understanding Cardiology Practitioners' Interpretations of Electrocardiograms: An Eye-Tracking Study.

BackgroundVisual expertise refers to advanced visual skills demonstrated when performing domain-specific visual tasks. Prior research has emphasized the fact that medical experts rely on such perceptual pattern-recognition skills when interpreting medical images, particularly in the field of electrocardiogram (ECG) interpretation. Analyzing and modeling cardiology practitioners’ visual behavior across different levels of expertise in the health care sector is crucial. Namely, understanding such acquirable visual skills may help train less experienced clinicians to interpret ECGs accurately.ObjectiveThis study aims to quantify and analyze through the use of eye-tracking technology differences in the visual behavior and methodological practices for different expertise levels of cardiology practitioners such as medical students, cardiology nurses, technicians, fellows, and consultants when interpreting several types of ECGs.MethodsA total of 63 participants with different levels of clinical expertise took part in an eye-tracking study that consisted of interpreting 10 ECGs with different cardiac abnormalities. A counterbalanced within-subjects design was used with one independent variable consisting of the expertise level of the cardiology practitioners and two dependent variables of eye-tracking metrics (fixations count and fixation revisitations). The eye movements data revealed by specific visual behaviors were analyzed according to the accuracy of interpretation and the frequency with which interpreters visited different parts/leads on a standard 12-lead ECG. In addition, the median and SD in the IQR for the fixations count and the mean and SD for the ECG lead revisitations were calculated.ResultsAccuracy of interpretation ranged between 98% among consultants, 87% among fellows, 70% among technicians, 63% among nurses, and finally 52% among medical students. The results of the eye fixations count, and eye fixation revisitations indicate that the less experienced cardiology practitioners need to interpret several ECG leads more carefully before making any decision. However, more experienced cardiology practitioners rely on their skills to recognize the visual signal patterns of different cardiac abnormalities, providing an accurate ECG interpretation.ConclusionsThe results show that visual expertise for ECG interpretation is linked to the practitioner’s role within the health care system and the number of years of practical experience interpreting ECGs. Cardiology practitioners focus on different ECG leads and different waveform abnormalities according to their role in the health care sector and their expertise levels.

Controlling an effector with eye movements: The effect of entangled sensory and motor responsibilities.

Restoring arm and hand function has been indicated by individuals with tetraplegia as one of the most important factors for regaining independence. The overall goal of our research is to develop assistive technologies that allow individuals with tetraplegia to control functional reaching movements. This study served as an initial step toward our overall goal by assessing the feasibility of using eye movements to control the motion of an effector in an experimental environment. We aimed to understand how additional motor requirements placed on the eyes affected eye-hand coordination during functional reaching. We were particularly interested in how eye fixation error was affected when the sensory and motor functions of the eyes were entangled due to the additional motor responsibility. We recorded participants' eye and hand movements while they reached for targets on a monitor. We presented a cursor at the participant's point of gaze position which can be thought of as being similar to the control of an assistive robot arm. To measure eye fixation error, we used an offline filter to extract eye fixations from the raw eye movement data. We compared the fixations to the locations of the targets presented on the monitor. The results show that not only are humans able to use eye movements to direct the cursor to a desired location (1.04 ± 0.15 cm), but they can do so with error similar to that of the hand (0.84 ± 0.05 cm). In other words, despite the additional motor responsibility placed on the eyes during direct eye-movement control of an effector, the ability to coordinate functional reaching movements was unaffected. The outcomes of this study support the efficacy of using the eyes as a direct command input for controlling movement.

Visual Attention Pattern of Middle School Students During Problem‐Solving in Physics

ABSTRACTThis study investigated how visual attention differed between students who hold scientific knowledge and those who have misconceptions. We analyzed visual fixations of 98 middle students on problems containing information relevant to scientific knowledge and common misconceptions. It showed that students who acquired scientific concepts paid more attention to appropriate information but took less consideration to irrelevant information. On the contrary, students who held misconceptions fixated more on misleading information but failed to select appropriate information. In addition, the senior students demonstrated more eye fixations on relevant information than their younger peers. These findings reflect that students' conceptual understanding affects the way they evaluate the problem information. Our results also suggest that formal science education improves students' ability to select relevant information and efficiently solve a problem. Studying the visual attention pattern during problem‐solving provides implications for the development of effective instructions to promote their ability to solve scientific problems.

Presaccadic attention enhances contrast sensitivity, but not at the upper vertical meridian

SummaryVisual performance has striking polar performance asymmetries: At a fixed eccentricity, it is better along the horizontal than vertical meridian and the lower than upper vertical meridian. These asymmetries are not alleviated by covert exogenous or endogenous attention, but have been studied exclusively during eye fixation. However, a major driver of everyday attentional orienting is saccade preparation, during which attention automatically shifts to the future eye fixation. This presaccadic attention shift is considered strong and compulsory, and relies on different neural computations and substrates than covert attention. Thus, we asked: Can presaccadic attention compensate for the ubiquitous performance asymmetries observed during eye fixation? Our data replicate polar performance asymmetries during fixation and document the same asymmetries during saccade preparation. Crucially, however, presaccadic attention enhanced contrast sensitivity at the horizontal and lower vertical meridian, but not at the upper vertical meridian. Thus, instead of attenuating performance asymmetries, presaccadic attention exacerbates them.

Elements of exogenous attentional cueing preserved during optokinetic motion of the visual scene.

Navigating through our environment raises challenges for perception by generating salient background visual motion and eliciting prominent eye movements to stabilise the retinal image. It remains unclear if exogenous spatial attentional orienting is possible during background motion and the eye movements it causes and whether this compromises the underlying neural processing. To test this, we combined exogenous orienting, visual scene motion, and electroencephalography (EEG). A total of 26 participants viewed a background of moving black and grey bars (optokinetic stimulation). We tested for effects of non-spatially predictive peripheral cueing on visual motion discrimination of a target dot, presented either at the same (valid) or opposite (invalid) location as the preceding cue. Valid cueing decreased reaction times not only when participants kept their gaze fixed on a central point (fixation blocks) but also even when there was no fixation point, so that participants performed intensive, repetitive tracking eye movements (eye movement blocks). Overall, manual response reaction times were slower during eye movements. Cueing also produced reliable effects on neural activity on either block, including within the first 120 ms of neural processing of the target. The key pattern with larger event-related potential (ERP) amplitudes on invalid versus valid trials showed that the neural substrate of exogenous cueing was highly similar during eye movements or fixation. Exogenous peripheral cueing and its neural correlates are robust against distraction from the moving visual scene, important for perceptual cognition during navigation.

The effect of binocular summation on visual acuity and reading performance with single vision and multifocal contact lenses: an eye-fixation based analysis

Purpose: Simultaneous-image correction with aspheric multifocal contact lenses (CLs) usually requires perceptual processes, such as binocular summation, to help the interpretation of “in-focus”, while suppressing out-of-focus images. Here, the effect of binocular summation on reading performance is evaluated using a new method, based on eye fixation analysis.

Visual Saliency via Multiscale Analysis in Frequency Domain and Its Applications to Ship Detection in Optical Satellite Images.

This article proposes a bottom-up visual saliency model that uses the wavelet transform to conduct multiscale analysis and computation in the frequency domain. First, we compute the multiscale magnitude spectra by performing a wavelet transform to decompose the magnitude spectrum of the discrete cosine coefficients of an input image. Next, we obtain multiple saliency maps of different spatial scales through an inverse transformation from the frequency domain to the spatial domain, which utilizes the discrete cosine magnitude spectra after multiscale wavelet decomposition. Then, we employ an evaluation function to automatically select the two best multiscale saliency maps. A final saliency map is generated via an adaptive integration of the two selected multiscale saliency maps. The proposed model is fast, efficient, and can simultaneously detect salient regions or objects of different sizes. It outperforms state-of-the-art bottom-up saliency approaches in the experiments of psychophysical consistency, eye fixation prediction, and saliency detection for natural images. In addition, the proposed model is applied to automatic ship detection in optical satellite images. Ship detection tests on satellite data of visual optical spectrum not only demonstrate our saliency model's effectiveness in detecting small and large salient targets but also verify its robustness against various sea background disturbances.