Eye Fixation Locations Research Articles

An egocentric video and eye-tracking dataset for visual search in convenience stores

We introduce an egocentric video and eye-tracking dataset, comprised of 108 first-person videos of 36 shoppers searching for three different products (orange juice, KitKat chocolate bars, and canned tuna) in a convenience store, along with the frame-centered eye fixation locations for each video frame. The dataset also includes demographic information about each participant in the form of an 11-question survey. The paper describes two applications using the dataset — an analysis of eye fixations during search in the store, and a training of a clustered saliency model for predicting saliency of viewers engaged in product search in the store. The fixation analysis shows that fixation duration statistics are very similar to those found in image and video viewing, suggesting that similar visual processing is employed during search in 3D environments and during viewing of imagery on computer screens. A clustering technique was applied to the questionnaire data, which resulted in two clusters being detected. Based on these clusters, personalized saliency prediction models were trained on the store fixation data, which provided improved performance in prediction saliency on the store video data compared to state-of-the art universal saliency prediction methods.

The Alleviation of Perceptual Blindness During Driving in Urban Areas Guided by Saccades Recommendation

In advanced industrial applications, computational visual attention models (CVAMs) could predict visual attention very similarly to actual human attention allocation. This has been used as a very important component of technology in advanced driver assistance systems (ADAS). Given that the biological inspiration of the driving-related CVAMs could be obtained from skilled drivers in complex driving conditions, in which the driver’s attention is constantly directed at various salient and informative visual stimuli by alternating the eye fixations via saccades to drive safely, this paper proposes a saccade recommendation strategy to enhance the driving safety under urban road environment, particularly when the driver’s vision is often impaired by the visual crowding. The altered and directed saccades are collected and optimized by extracting four innate features from human dynamic vision. A neural network isdesigned to classify preferable saccades to reduce perceptual blindness due to visual crowding under urban scenes. A state-of-the-art CVAM is firstly adopted to localize the predicted eye fixation locations (EFLs) in driving video clips. Besides, human subjects’ gaze at the recommended EFLs is measured via an eye-tracker. The time delays between the predicted EFLs and drivers’ EFLs are analyzed under different driving conditions, followed by the time delays between the predicted EFLs and the driver’s hand control. The visually safe margin is then measured by mediating the driving speed and the total delay. Experimental results demonstrate that the recommended saccades can effectively reduce the amount of perceptual blindness, which is known to be of help to further improve road driving safety.

Form Follows Brain Function: A Computational Mapping Approach

The strong association between computation and the built environment gives birth to new research approaches to urban design. Although these approaches enable the implementation of urban systems based on various collections of datasets, they usually neglect human experience. To address this problem, our research identifies and quantifies spatial information and patterns of urban organization based on experience, by gathering two parallel types of user data: sensory and recollection. The presented experiment, executed in Copenhagen, Denmark with 23 participants, involves a walking task, and a recollection task. The walking task is 1 km walk from the Copenhagen Court House building to the Royal Danish Theatre through Strøget Street, while wearing eye-tracking spectacles. The recollection task includes the underlining – on a 3D photorealistic digital model – of the elements that the participants can recall from their walk. We monitor the eye movement of the participants to locally characterize the urban path by computing four spatial attributes: a) The fixations on elements (counted through iMotions Software), b) The distance from the viewer (taken by GPS points through FUMapp software), c) The position of the stimuli related to the eye-level (calculated in FUMapp), and d) The degree of spatial transformations on street intersections (through spatial variables in FUMapp). Then we compare the eye-tracking and the recollection data and we evaluate their heat maps. The originality of this research is twofold: First, for the first time a substantial amount of quantifiable optical data related to urban walking is recorded; Second, for the first time this experiment is performed on the public street. The proposed method could offer a quantifiable basis for predicting eye fixation locations, determining human centric guidelines for urban design. Additional modes of body sensors could yield even more all-encompassing body-metrics related to the urban walking experience.

Eye Fixation Location Recommendation in Advanced Driver Assistance System

Recent research progress on the approach of visual attention modeling for mediated perception to advanced driver assistance system (ADAS) has drawn the attention of computer and human vision researchers. However, it is still debatable whether the actual driver’s eye fixation locations (EFLs) or the predicted EFLs which are calculated by computational visual attention models (CVAMs) are more reliable for safe driving under real-life driving conditions. We analyzed the suitability of the following two EFLs using ten typical categories of natural driving video clips: the EFLs of human drivers and the EFLs predicted by CVAMs. In the suitability analysis, we used the EFLs confirmed by two experienced drivers as the reference EFLs. We found that both approaches alone are not suitable for safe driving and EFL suitable for safe driving depends on the driving conditions. Based on this finding, we propose a novel strategy for recommending one of the EFLs to the driver for ADAS under predefined 10 real-life driving conditions. We propose to recommend one of the following 3 EFL modes for different driving conditions: driver’s EFL only, CVAM’s EFL only, and interchangeable EFL. In interchangeable EFL mode, driver’s EFL and CVAM’s EFL are interchangeable. The selection of two EFLs is a typical binary classification problem, so we apply support vector machines (SVMs) to solve this problem. We also provide a quantitative evaluation of the classifiers. The performance evaluation of the proposed recommendation method indicates that it is potentially useful to ADAS for future safe driving.

Visual Saliency Detection via Sparse Residual and Outlier Detection

This letter proposes a bottom-up saliency model to predict eye fixation locations. Unlike traditional models that measure saliency by computing local or global distinctness, the proposed model considers saliency as the prediction error, because we believe that image patches or pixels with higher prediction error are more salient than others. The prediction error consists of both mispredicted error and unpredicted error. We propose a new algorithm called sparse residual to compute the mispredicted error. We then adopt outlier detection to compute the unpredicted error. Finally, we obtain the saliency map from merging the two results together via a guided filter. Extensive experiments on three benchmark databases show that our model is superior to 12 state-of-the-art models.

Early biological vision inspired system for salience computation in images

A bio-inspired salience computation system comprising of novel signal processing modules is proposed in this paper that predicts human eye fixation locations in images. The modules are designed based on evidence from early biological vision. Analysis is carried out to intuitively, theoretically and graphically justify various considerations in the proposed modules. The proposed system concurs with existing feature integration theory and guiding representation based visual processing model. The effectiveness of the proposed salience computation system in generating salience maps closer to human eye fixation density maps is demonstrated by comparing it both qualitatively and quantitatively to other existing state-of-the-art salience computation approaches. Several synthetic images and psychological patterns, and real-life images from four well-known datasets are considered for this purpose.

Predicting the eye fixation locations in the gray scale images in the visual scenes with different semantic contents

In recent years, there has been considerable interest in visual attention models (saliency map of visual attention). These models can be used to predict eye fixation locations, and thus will have many applications in various fields which leads to obtain better performance in machine vision systems. Most of these models need to be improved because they are based on bottom-up computation that does not consider top-down image semantic contents and often does not match actual eye fixation locations. In this study, we recorded the eye movements (i.e., fixations) of fourteen individuals who viewed images which consist natural (e.g., landscape, animal) and man-made (e.g., building, vehicles) scenes. We extracted the fixation locations of eye movements in two image categories. After extraction of the fixation areas (a patch around each fixation location), characteristics of these areas were evaluated as compared to non-fixation areas. The extracted features in each patch included the orientation and spatial frequency. After feature extraction phase, different statistical classifiers were trained for prediction of eye fixation locations by these features. This study connects eye-tracking results to automatic prediction of saliency regions of the images. The results showed that it is possible to predict the eye fixation locations by using of the image patches around subjects' fixation points.

Differences of eye movement pattern in natural and man-made scenes and image categorization with the help of these patterns.

In this paper, we investigated the parameters related to eye movement patterns of individuals while viewing images that consist of natural and man-made scenes. These parameters are as follows: number of fixations and saccades, fixation duration, saccade amplitude and distribution of fixation locations. We explored the way in which individuals look at images of different semantic categories, and used this information for automatic image classification. We showed that the eye movements and the contents of eye fixation locations of observers differ for images of different semantic categories. These differences were used effectively in automatic image categorization. Another goal of this study was to find the answer of this question that "whether the image patches of fixation points have sufficient information for image categorization?" To achieve this goal, a number of patches with different sizes from two different image categories was extracted. These patches, which were selected at the location of eye fixation points, were used to form a feature vector based on K-means clustering algorithm. Then, different statistical classifiers were trained for categorization purpose. The results showed that it is possible to predict the image category by using the feature vectors derived from the image patches. We found significant differences in parameters of eye movement pattern between the two image categories (average across subjects). We could categorize images by using these parameters as features. The results also showed that it is possible to predict the image category by using image patches around the subjects' fixation points.

A bio-inspired center-surround model for salience computation in images

A center-surround model inspired by photoreceptor interactions and visual receptive field organization is presented in this paper for salience computation that predicts human eye fixation locations in images. The essence of photoreceptor interactions is implemented considering different nonlinear combinations of responses to stimuli given by values at nearby image pixels. These combinations are then fed to difference of Gaussian filtered outputs operation and Gabor filter based processes simulating visual receptive field organization. The proposed center-surround model is used in Itti et al.’s bio-inspired framework to perform salience computation. Analysis is carried out to present the information-theoretic aspect of the nonlinear combinations. Significance of the proposed center-surround model is shown both qualitatively and quantitatively by comparing its use in salience computation with the use of existing models considering different psychological patterns, and synthetic and real-life images. Quantitative and qualitative performance of salience computation using the novel center-surround model for three well-known datasets of images are also compared to that of relevant existing salience computation approaches to demonstrate the effectiveness of the proposed approach in generating salience maps closer to human eye fixation density maps.

Salience computation in images based on perceptual distinctness

A salience computation approach based on perceptual distinctness is proposed in this paper that predicts human eye fixation locations in images. The approach uses a novel representation of pattern in a patch around a pixel such that discrimination between patterns is in compliance with distinctness between them in-terms of a few well-known perceptual cues. Image patches of different sizes (resolutions) are considered. The representations of the value at and pattern around a pixel in different features such as lightness, color and orientation are discriminated from all those in the rest of the image to compute salience at that pixel. The effectiveness of the proposed salience computation approach in generating salience maps closer to human eye fixation density maps is demonstrated by comparing it both qualitatively and quantitatively to other existing state-of-the-art salience computation approaches. Several synthetic images and psychological patterns, and real-life images from five well-known datasets are considered for this purpose.

Lexical activation during sentence comprehension in adolescents with history of Specific Language Impairment

One remarkable characteristic of speech comprehension in typically developing (TD) children and adults is the speed with which the listener can integrate information across multiple lexical items to anticipate upcoming referents. Although children with Specific Language Impairment (SLI) show lexical deficits (Sheng & McGregor, 2010) and slower speed of processing (Leonard et al., 2007), relatively little is known about how these deficits manifest in real-time sentence comprehension. In this study, we examine lexical activation in the comprehension of simple transitive sentences in adolescents with a history of SLI and age-matched, TD peers. Participants listened to sentences that consisted of the form, Article-Agent-Action-Article-Theme, (e.g., The pirate chases the ship) while viewing pictures of four objects that varied in their relationship to the Agent and Action of the sentence (e.g., Target, Agent-Related, Action-Related, and Unrelated). Adolescents with SLI were as fast as their TD peers to fixate on the sentence's final item (the Target) but differed in their post-action onset visual fixations to the Action-Related item. Additional exploratory analyses of the spatial distribution of their visual fixations revealed that the SLI group had a qualitatively different pattern of fixations to object images than did the control group. The findings indicate that adolescents with SLI integrate lexical information across words to anticipate likely or expected meanings with the same relative fluency and speed as do their TD peers. However, the failure of the SLI group to show increased fixations to Action-Related items after the onset of the action suggests lexical integration deficits that result in failure to consider alternate sentence interpretations.Learning outcomes: As a result of this paper, the reader will be able to describe several benefits of using eye-tracking methods to study populations with language disorders. They should also recognize several potential explanations for lexical deficits in SLI, including possible reduced speed of processing, and degraded lexical representations. Finally, they should recall the main outcomes of this study, including that adolescents with SLI show different timing and location of eye-fixations while interpreting sentences than their age-matched peers.

Software for the automatic correction of recorded eye fixation locations in reading experiments

Because the recorded location of an eyetracking fixation is not a perfect measure of the actual fixated location, the recorded fixation locations must be adjusted before analysis. Fixations are typically corrected manually. Making such changes, however, is time-consuming and necessarily involves a subjective component. The goal of this article is to introduce software to automate parts of the correction process. The initial focus is on the correction of vertical locations and the removal of outliers and ambiguous fixations in reading experiments. The basic idea behind the algorithm is to use linear regression to assign each fixation to a text line and to identify outliers. The freely available software is implemented as a function, , written in R.

Limits of spatial attention in three-dimensional space and dual-task driving performance

The present study examined the limits of spatial attention while performing two driving relevant tasks that varied in depth. The first task was to maintain a fixed headway distance behind a lead vehicle that varied speed. The second task was to detect a light-change target in an array of lights located above the roadway. In Experiment 1 the light detection task required drivers to encode color and location. The results indicated that reaction time to detect a light-change target increased and accuracy decreased as a function of the horizontal location of the light-change target and as a function of the distance from the driver. In a second experiment the light change task was changed to a singleton search (detect the onset of a yellow light) and the workload of the car following task was systematically varied. The results of Experiment 2 indicated that RT increased as a function of task workload, the 2D position of the light-change target and the distance of the light-change target. A multiple regression analysis indicated that the effect of distance on light detection performance was not due to changes in the projected size of the light target. In Experiment 3 we found that the distance effect in detecting a light change could not be explained by the location of eye fixations. The results demonstrate that when drivers attend to a roadway scene attention is limited in three-dimensional space. These results have important implications for developing tests for assessing crash risk among drivers as well as the design of in vehicle technologies such as head-up displays.

How Does the Purpose of Inspection Influence the Potency of Visual Salience in Scene Perception?

Salience-map models have been taken to suggest that the locations of eye fixations are determined by the extent of the low-level discontinuities in an image. While such models have found some support, an increasing emphasis on the task viewers are performing implies that these models must combine with cognitive demands to describe how the eyes are guided efficiently. An experiment is reported in which eye movements to objects in photographs were examined while viewers performed a memory-encoding task or one of two search tasks. The objects depicted in the scenes had known salience ranks according to a popular model. Participants fixated higher-salience objects sooner and more often than lower-salience objects, but only when memorising scenes. This difference shows that salience-map models provide useful predictions even in complex scenes and late in viewing. However, salience had no effects when searching for a target defined by category or exemplar. The results suggest that salience maps are not used to guide the eyes in these tasks, that cognitive override by task demands can be total, and that modelling top-down search is important but may not be easily accomplished within a salience-map framework.

Word identification and eye fixation locations in visual and visual-plus-auditory presentations of spoken sentences.

In this study, we investigated where people look on talkers' faces as they try to understand what is being said. Sixteen young adults with normal hearing and demonstrated average speechreading proficiency were evaluated under two modality presentation conditions: vision only versus vision plus low-intensity sound. They were scored for the number of words correctly identified from 80 unconnected sentences spoken by two talkers. The results showed two competing tendencies: an eye primacy effect that draws the gaze to the talkers eyes during silence and an information source attraction effect that draws the gaze to the talker's mouth during speech periods. Dynamic shifts occur between eyes and mouth prior to speech onset and following the offset of speech, and saccades tend to be suppressed during speech periods. The degree to which the gaze is drawn to the mouth during speech and the degree to which saccadic activity is suppressed depend on the difficulty of the speech identification task. Under the most difficult modality presentation condition, vison only, accuracy was related to average sentence difficulty and individual proficiency in visual speech perception, but not to the proportion of gaze time directed toward the talkers mouth or toward other parts of the talker's face.

Detection of Unexpected Motion While Driving:From Psychophysics to Real World via Virtual Environments

This paper describes a set of experiments investigating the interaction between the location of eye fixations and the detection of unexpected motion while driving. Both psychophysical and real-world observations indicate that there are differences between the upper and lower visual fields with respect to driving. We began with psychophysical experiments to test whether the detection of unexpected motion Is inherently different in the upper and lower visual fields. No difference was found. However, when texture was added to the driving surface, a large difference was found, possibly due to optokinetic nystagmus stimulated by the texture. These results were confirmed in a driving simulator, and their implications for head-up displays (HUDs) explored. We found that the same upper/lower field asymmetry could be found with digital HUDs but not with analog HUDs. These experiments illustrate how virtual environment technology can connect knowledge from psychophysical experimentation to more realistic situations.