Eye Fixation Data Research Articles

Predicting First-Language and Second-Language Proficiency Using Eye Fixation Data and Demographic Information: Assumptions, Data Representations and Methods

Predicting First-Language and Second-Language Proficiency Using Eye Fixation Data and Demographic Information: Assumptions, Data Representations and Methods

The quest for simplicity in human learning: Identifying the constraints on attention

For better or worse, humans live a resource-constrained existence; only a fraction of physical sensations ever reach conscious awareness, and we store a shockingly small subset of these experiences in memory for later use. Here, we examined the effects of attention constraints on learning. Among models that frame selective attention as an optimization problem, attention orients toward information that will reduce errors. Using this framing as a basis, we developed a suite of models with a range of constraints on the attention available during each learning event. We fit these models to both choice and eye-fixation data from four benchmark category-learning data sets, and choice data from another dynamic categorization data set. We found consistent evidence for computations we refer to as “simplicity”, where attention is deployed to as few dimensions of information as possible during learning, and “competition”, where dimensions compete for selective attention via lateral inhibition.

The development of an eye movement-based deep learning system for laparoscopic surgical skills assessment

The development of valid, reliable, and objective methods of skills assessment is central to modern surgical training. Numerous rating scales have been developed and validated for quantifying surgical performance. However, many of these scoring systems are potentially flawed in their design in terms of reliability. Eye-tracking techniques, which provide a more objective investigation of the visual-cognitive aspects of the decision-making process, recently have been utilized in surgery domains for skill assessment and training, and their use has been focused on investigating differences between expert and novice surgeons to understand task performance, identify experienced surgeons, and establish training approaches. Ten graduate students at the National Taiwan University of Science and Technology with no prior laparoscopic surgical skills were recruited to perform the FLS peg transfer task. Then k-means clustering algorithm was used to split 500 trials into three dissimilar clusters, grouped as novice, intermediate, and expert levels, by an objective performance assessment parameter incorporating task duration with error score. Two types of data sets, namely, time series data extracted from coordinates of eye fixation and image data from videos, were used to implement and test our proposed skill level detection system with ensemble learning and a CNN algorithm. Results indicated that ensemble learning and the CNN were able to correctly classify skill levels with accuracies of 76.0% and 81.2%, respectively. Furthermore, the incorporation of coordinates of eye fixation and image data allowed the discrimination of skill levels with a classification accuracy of 82.5%. We examined more levels of training experience and further integrated an eye tracking technique and deep learning algorithms to develop a tool for objective assessment of laparoscopic surgical skill. With a relatively unbalanced sample, our results have demonstrated that the approach combining the features of visual fixation coordinates and images achieved a very promising level of performance for classifying skill levels of trainees.

Gaze Mapping for Immersive Virtual Environments Based on Image Retrieval

In this paper, we introduce a novel gaze mapping approach for free viewing conditions in dynamic immersive virtual environments (VEs), which projects recorded eye fixation data of users, who viewed the VE from different perspectives, to the current view. This generates eye fixation maps, which can serve as ground truth for training machine learning (ML) models to predict saliency and the user’s gaze in immersive virtual reality (VR) environments. We use a flexible image retrieval approach based on SIFT features, which can also map the gaze under strong viewpoint changes and dynamic changes. A vocabulary tree enables to scale to the large amounts of data with typically several hundred thousand frames and a homography transform re-projects the fixations to the current view. To evaluate our approach, we measure the predictive quality of our eye fixation maps to model the gaze of the current user and compare our maps to computer-generated saliency maps on the DGaze and the Saliency in VR datasets. The results show that our method often outperform these saliency predictors. However, in contrast to these methods, our approach collects real fixations from human observers, and can thus serve to estimate ground truth fixation maps in dynamic VR environments, which can be used to train and evaluate gaze predictors.

Application of Machine Learning Techniques to Detect the Children with Autism Spectrum Disorder.

Early detection of autism spectrum disorder (ASD) is highly beneficial to the health sustainability of children. Existing detection methods depend on the assessment of experts, which are subjective and costly. In this study, we proposed a machine learning approach that fuses physiological data (electroencephalography, EEG) and behavioral data (eye fixation and facial expression) to detect children with ASD. Its implementation can improve detection efficiency and reduce costs. First, we used an innovative approach to extract features of eye fixation, facial expression, and EEG data. Then, a hybrid fusion approach based on a weighted naive Bayes algorithm was presented for multimodal data fusion with a classification accuracy of 87.50%. Results suggest that the machine learning classification approach in this study is effective for the early detection of ASD. Confusion matrices and graphs demonstrate that eye fixation, facial expression, and EEG have different discriminative powers for the detection of ASD and typically developing children, and EEG may be the most discriminative information. The physiological and behavioral data have important complementary characteristics. Thus, the machine learning approach proposed in this study, which combines the complementary information, can significantly improve classification accuracy.

Investigating the Use of Eye Fixation Data for Emotion Classification in VR

Eye-tracking technology has become popular recently and widely used in research on emotion recognition since its usability. In this paper, we presented a preliminary investigation on a novelty approach for detecting emotions using eye-tracking data in virtual reality (VR) to classify 4-quadrant of emotions according to russell’scircumplex model of affects. A presentation of 3600 videos is used as the experiment stimuli to evoke the emotions of the user in VR. An add-on eye-tracker within the VR headset is used for the recording and collecting device of eye-tracking data. Fixation data is extracted and chosen as the eye feature used in this investigation. The machine learning classifier is support vector machine (SVM) with radial basis function (RBF) kernel. The best classification accuracy achieved is 69.23%. The findings showed that emotion classification using fixation data has promising results in the prediction accuracy from a four-class random classification.

Pupil diameter as a biomarker of effort in goal-directed gait.

Subjects' eye movement behavior related to cognitive effort during gait was measured as subjects walked to perform low and high cognitive load tasks. We found that all pupil diameter measures, fixation durations, and the proportion of blink duration changed significantly during gait as a function of task load. In contrast, the number of fixations, saccade durations and travel time did not change significantly as a function of task load. Findings showed that pupil diameter was the best predictor of task load during one's gait preceding the performance of the task. While other studies have demonstrated the importance of eye fixation characteristics during gait, our findings showed that eye measures related to pupil diameter were better at detecting cognitive load while walking to perform a task compared to eye fixation data. We also found that cognitive effort was not limited to just the performance of the task, but that it was also exerted during one's gait preceding the performance of the task. Therefore, the additional attention demand caused by an increase in task complexity may result in less attentional resources being available to adequately handle distractions (such as obstacle avoidance) while walking to perform the task. Consequently, this may increase the likelihood of falls in those individuals with lower attentional capacity.

Effect of lexical accessibility on syntactic production in aphasia: an eyetracking study

ABSTRACTBackground & Aims: Healthy speakers use both word-level and structure-level information to ease sentence production processes. Structural priming facilitates message-structure mapping in aphasia. However, it remains unclear if and how word-level information affects off-line and on-line sentence production in persons with aphasia (PWA). This eyetracking-while-speaking study examined the effect of lexical priming on production of syntactic (active/passive) structures in PWA.Methods: Eleven PWA and twenty healthy older adults (HOA) described transitive actions (woman pulling horse) following lexical priming, wherein the relative ease of lexical retrieval for the Agent or Theme was manipulated via an auditory probe (what is happening with the woman/horse?). It was examined whether or not PWA produce the sentence structure that allows earlier production of the primed word (e.g., passives when Theme was primed). Participants’ eye fixation times to each character (Agent, Theme) were also monitored to examine if PWA show priming-induced preferential looks to one character from the earliest stage of production, consistent with word-driven planning.Results: HOA showed increased production of passives over actives in the Theme vs. Agent prime condition. In eye fixation data, HOA showed priming-induced Theme advantage from the earliest time window (picture onset-400 milliseconds). PWA also showed a significant priming effect in off-line sentence production, with this priming effect being greater for the individuals whose syntactic processing is better preserved. In eye fixation data, however, PWA showed preferential fixations to the primed character at a later stage of sentence planning (400–800 milliseconds), following equal fixation time to Agent and Theme during the earliest time window.Conclusion: HOA showed word-driven production in both off-line and real-time (eye fixations) production. Lexical accessibility effectively drove off-line syntactic production in PWA, especially for those whose syntactic capacity remains relatively preserved. However, PWA showed advanced processing of both characters in earliest eye fixation data, suggesting that successful word-driven off-line syntactic production was associated with atypical real-time sentence planning in aphasia.

Validity of an eyetracking method for capturing auditory–visual cross-format semantic priming

ABSTRACTIntroduction: Semantic priming paradigms are important for understanding lexical–semantic processing and the nature of linguistic deficits accompanying language performance in neurologically impaired individuals such as people with aphasia. Reaction-time-(RT)-based traditional semantic priming tasks entail potential confounds, especially problematic when applied to people with aphasia, who may have concomitant neurocognitive challenges that limit task performance. Some of these confounds include requirements of following complex instructions, making metalinguistic judgments, and using speech or limb-based motor actions to indicate overt responses. Eyetracking methods have great potential for avoiding some of these confounds. We tested the validity of an eyetracking method in capturing semantic priming in an auditory–visual cross-format priming paradigm (auditory word prime–visual image target).Method: A total of 72 neurologically unimpaired adults participated in two phases: a stimulus development phase using traditional priming (n = 32) and an experimental eyetracking phase (n = 40). Each phase included two conditions, representing distinct levels of prime–target semantic relatedness: unrelated and related. Mean RT data from the traditional priming (stimuli development) phase guided image selection for the eyetracking experiment. Eyetracking indices of fixation duration and latency of fixation were recorded to capture semantic priming in the eyetracking experiment.Results: Eye fixation data indicated that images related to auditory primes were attended to earlier and attracted significantly greater visual attention than unrelated images. These results mirrored RT data from the traditional priming method, which showed faster RT latencies and more accurate naming performance for related images than for unrelated images.Conclusions: Results support the validity of eyetracking indices of semantic priming and offer a robust testing protocol for future studies in this line of research. Current clinical relevance for people with aphasia is highlighted. Further empirical testing of the psychometric properties of the eyetracking measures in various semantic priming contexts is recommended.

Unsupervised Uncertainty Estimation Using Spatiotemporal Cues in Video Saliency Detection.

In this paper, we address the problem of quantifying the reliability of computational saliency for videos, which can be used to improve saliency-based video processing algorithms and enable more reliable performance and objective risk assessment of saliency-based video processing applications. Our approach to quantify such reliability is twofold. First, we explore spatial correlations in both the saliency map and the eye-fixation map. Then, we learn the spatiotemporal correlations that define a reliable saliency map. We first study spatiotemporal eye-fixation data from the public CRCNS data set and investigate a common feature in human visual attention, which dictates a correlation in saliency between a pixel and its direct neighbors. Based on the study, we then develop an algorithm that estimates a pixel-wise uncertainty map that reflects our supposed confidence in the associated computational saliency map by relating a pixel's saliency to the saliency of its direct neighbors. To estimate such uncertainties, we measure the divergence of a pixel, in a saliency map, from its local neighborhood. In addition, we propose a systematic procedure to evaluate uncertainty estimation performance by explicitly computing uncertainty ground truth as a function of a given saliency map and eye fixations of human subjects. In our experiments, we explore multiple definitions of locality and neighborhoods in spatiotemporal video signals. In addition, we examine the relationship between the parameters of our proposed algorithm and the content of the videos. The proposed algorithm is unsupervised, making it more suitable for generalization to most natural videos. Also, it is computationally efficient and flexible for customization to specific video content. Experiments using three publicly available video data sets show that the proposed algorithm outperforms state-of-the-art uncertainty estimation methods with improvement in accuracy up to 63% and offers efficiency and flexibility that make it more useful in practical situations.

Initial Accuracy of Word-Referent Mappings Affects Word Learning


 It was previously assumed that a correct, or accurate, initial association between a word and its referent allows optimal language learning, since fewer cognitive resources are required. However, some studies have found that initially incorrect, or inaccurate, associations can cause adults to learn word-referent mappings significantly better, compared to initially accurate ones. The opposite effect was found in children, who typically learn better when a word-referent association is consistently accurate.Our research project further explores these findings. The study involves a cross-situational word-learning paradigm examining whether the correction of inaccurate initial word-referent associations benefits word learning. Eye fixation and pupil dilation data are used to determine initial mappings and cognitive load, respectively. In the familiarisation phase of the study, participants are presented with associations between made-up words and the images to which the words refer; initial accuracy is manipulated such that only half of the presented associations are correct. Then, in the learning phase, each word is presented with its correct referent. Finally, the accuracy of the learned word-referent mappings is tested.Our initial findings confirm that word learning benefits when adults are presented with initially inaccurate associations and subsequent corrections. We will discuss the theories behind these findings, as well as the implications for language learning. In the most recent phase of the project, we are determining whether pupil dilation is a good measure for increased cognitive load associated with word-referent inaccuracy.

Indirect request processing, sentence types and illocutionary forces

According to the literalist view of speech acts, morpho-syntactic sentence types are associated directly at the semantic level with an illocutionary force. By contrast, according to contextualist theories illocutionary force emerges from contexts of use. To date, however, there is little experimental evidence relevant to this debate. We propose two experimental, eye-tracking studies to test two predictions of the literalist view: First, unlike for the highly conventionalised Can you? forms, whenever a non-conventionalised construction such as Is it possible to? is interpreted as a request, its question interpretation should also be activated. Second, the directive interpretation of modal You must declaratives should activate the statement interpretation and, therefore, be costlier than that of imperatives. In Study 1, we show, first, that, in contexts where both the non-directive and directive interpretation of indirect requests are available, the latter are processed as fast as that of the corresponding imperatives, independently of the conventionalisation degree of the indirect request at hand. Second, eye fixation data show that the comprehension of indirect requests does not activate their direct meaning. Study 2 shows that modal You must declaratives are understood as imperatives and do not activate a statement interpretation; this supports the view that obligation modal requests are as direct as imperative requests.

Visual Attention Modeling for Stereoscopic Video: A Benchmark and Computational Model

In this paper, we investigate the visual attention modeling for stereoscopic video from the following two aspects. First, we build one large-scale eye tracking database as the benchmark of visual attention modeling for stereoscopic video. The database includes 47 video sequences and their corresponding eye fixation data. Second, we propose a novel computational model of visual attention for stereoscopic video based on Gestalt theory. In the proposed model, we extract the low-level features, including luminance, color, texture, and depth, from discrete cosine transform coefficients, which are used to calculate feature contrast for the spatial saliency computation. The temporal saliency is calculated by the motion contrast from the planar and depth motion features in the stereoscopic video sequences. The final saliency is estimated by fusing the spatial and temporal saliency with uncertainty weighting, which is estimated by the laws of proximity, continuity, and common fate in Gestalt theory. Experimental results show that the proposed method outperforms the state-of-the-art stereoscopic video saliency detection models on our built large-scale eye tracking database and one other database (DML-ITRACK-3D).

Effects of attentional and lexical priming on syntactic production in aphasia: eye-tracking while speaking

Effects of attentional and lexical priming on syntactic production in aphasia: eye-tracking while speaking

A learning-based visual saliency prediction model for stereoscopic 3D video (LBVS-3D)

Saliency prediction models provide a probabilistic map of relative likelihood of an image or video region to attract the attention of the human visual system. Over the past decade, many computational saliency prediction models have been proposed for 2D images and videos. Considering that the human visual system has evolved in a natural 3D environment, it is only natural to want to design visual attention models for 3D content. Existing monocular saliency models are not able to accurately predict the attentive regions when applied to 3D image/video content, as they do not incorporate depth information. This paper explores stereoscopic video saliency prediction by exploiting both low-level attributes such as brightness, color, texture, orientation, motion, and depth, as well as high-level cues such as face, person, vehicle, animal, text, and horizon. Our model starts with a rough segmentation and quantifies several intuitive observations such as the effects of visual discomfort level, depth abruptness, motion acceleration, elements of surprise, size and compactness of the salient regions, and emphasizing only a few salient objects in a scene. A new fovea-based model of spatial distance between the image regions is adopted for considering local and global feature calculations. To efficiently fuse the conspicuity maps generated by our method to one single saliency map that is highly correlated with the eye-fixation data, a random forest based algorithm is utilized. The performance of the proposed saliency model is evaluated against the results of an eye-tracking experiment, which involved 24 subjects and an in-house database of 61 captured stereoscopic videos. Our stereo video database as well as the eye-tracking data are publicly available along with this paper. Experiment results show that the proposed saliency prediction method achieves competitive performance compared to the state-of-the-art approaches.

Perception of place of articulation of assimilated nasal and oral stops: What do response times and eye fixations tell us?

Two forced-choice identification and two visual world eyetracking experiments examined perception of the place of articulation (PoA) of word-final nasal and oral stops either in canonical (coronal) or assimilated form (coronal-to-labial place assimilation, e.g., ‘phonem /batp button’). Listeners’ response times (RT) and eye fixations were measured as they heard and, using a screen-based paradigm, identified assimilated or unassimilated words presented auditorily either in isolation (excised from recorded sentences) or with the assimilation-triggering context present (next word began with a labial consonant). Listeners were slower to identify isolated words ending in nasals, especially when words were assimilated. When the triggering context was present, RTs were overall faster and no longer different between nasal and oral stops. The eye fixation data further showed an early sensitivity to PoA cues carried by vowel transitions for assimilated oral stops. For words ending in assimilated nasal stops, howeve...

RETRACTED ARTICLE: Tabu search-based classification for eye-movement behavioral decisions

Adaptive human–computer interfaces (HCIs) are fundamental to designing adaptive websites and adaptive decision support systems. Integrating these intelligent systems with modern eye trackers provides more effective ways to exploit eye fixation data and offers improved services to users. We develop an exemplar-based classifier using the tabu search algorithm to predict which decision strategy may underlie an empirical search behavior. Our algorithm reduces the size of decision concept representations to find the best exemplars for each concept. Experimental results show that our classifier is highly accurate in classifying the sequence of empirical eye fixations, demonstrating the promise of integrating adaptive HCIs with modern eye trackers.

Principled animation design improves comprehension of complex dynamics

Learners can have difficulty in decomposing conventionally designed animations to obtain raw material suitable for building high quality mental models. A composition approach to designing animations based on the Animation Processing Model was developed as a principled alternative to prevailing approaches. Outcomes from studying novel and conventional animation designs (independent variable) were compared with respect to mental model quality, knowledge of local kinematics, and capacity to transfer (dependent variables). Study of a compositional animation that presented material in a contiguous fashion resulted in higher quality mental models of a piano mechanism than non-contiguous or control (conventional) versions but no significant differences regarding local kinematics or transfer. Eye fixation data indicated that the compositional animation led to superior mental models because it particularly fostered relational processing. Implications for future research and the design of educational animations are discussed.

Processing and learning of enhanced English collocations: An eye movement study

Research to date suggests that textual enhancement may positively affect the learning of multiword combinations known as collocations, but may impair recall of unenhanced text. However, the attentional mechanisms underlying such effects remain unclear. In this study, 38 undergraduate students were divided into two groups: one read a text containing typographically enhanced collocations (ET group) and the other read the same text with unenhanced collocations (the baseline text, or BT group). While reading, participants’ eye movements were recorded with an eye-tracker. Results showed that the ET group spent significantly longer time processing target collocations, and performed better than the BT group in a post-reading collocation test. However, apart from the enhanced collocations, the ET group recalled significantly less unenhanced text than the BT group. Further investigation of eye fixation data showed that the ET group spent substantially longer time processing collocations which, according to a pretest, they were not familiar with than did the BT group, whereas the two groups did not differ significantly in their processing of familiar collocations. Collectively, the results suggest that the trade-off between collocation learning and recall of unenhanced text is due to additional cognitive resources being allocated to enhanced collocations that are new to the reader.

Assessing drivers’ visual-motor coordination using eye tracking, GNSS and GIS: a spatial turn in driving psychology

Vehicle-driving in real traffic can be considered as a human-machine system involving not only the attribute of the vehicle movement but also the human visual perception, cognition and motion of the driver. The study of driving behaviours, therefore, would integrate information related to driver psychology, vehicle dynamics and road information in order to tackle research questions concerning driving safety. This paper describes a conceptual framework and an integrated GIS data model of a visual-motor coordination model (VMCM) to investigate drivers’ driving behaviour via the combination of vision tracking and vehicle positioning. The eye tracker recorded eye fixations and duration on video images to exhibit the driver’s visual search pattern and the traffic scenes. Real-time kinematic (RTK) post-processing of multi-GNSS (global navigation satellite system) tracking generated the vehicle movement trajectory at centimeter-level accuracy, which encompasses precise lateral positioning and speed control parameters of driving behaviours. The eye fixation data were then geocoded and linked to the vehicle movement trajectory to represent the VMCM on the GIS platform. An implementation prototype of the framework and the VMCM for a study of older drivers is presented in this paper. The spatial-temporal visualisation and statistical analysis based on the VMCM data-set allow for a greater insight into the inherent variability of older drivers’ visual search and motor behaviours. The research framework has demonstrated a discriminant and ecologically valid approach in driving behaviour assessment, which can also be used in studies for other cohort populations with modified driving scenarios or experiment designs.