Human Gaze Behavior Research Articles

Effects of individual’s emotions on saliency and visual search

Psychological studies have already proven a connection between emotional stimuli and visual attention. However, there is a lack of evidence of how much an individual’s mood influences visual information processing of emotionally neutral stimuli. In contrast to prior studies, we explored the effect of positive mood on bottom-up low-level visual saliency. In our extensive experimental studies, we induced positive or neutral emotions in 106 subjects using a psychological method of autobiographical memories and listening to positive music and then performed an eye-tracking study with three task types—free viewing, memorizing the image content and visual search tasks. We explored differences in the human gaze behaviour between both emotions and relate them to the bottom-up attention correlation between emotionally aroused and non-aroused subjects. We observed that positive emotions induce a strong broadening and engagement effect on the bottom-up attention while performing all task types. On the other hand, we found out that certain tasks are solved less efficiently when experiencing a positive mood. Therefore, we claim that positive mood may also act as a distractor while performing certain types of tasks.

Predicting Goal-directed Human Attention Using Inverse Reinforcement Learning.

Human gaze behavior prediction is important for behavioral vision and for computer vision applications. Most models mainly focus on predicting free-viewing behavior using saliency maps, but do not generalize to goal-directed behavior, such as when a person searches for a visual target object. We propose the first inverse reinforcement learning (IRL) model to learn the internal reward function and policy used by humans during visual search. We modeled the viewer's internal belief states as dynamic contextual belief maps of object locations. These maps were learned and then used to predict behavioral scanpaths for multiple target categories. To train and evaluate our IRL model we created COCO-Search18, which is now the largest dataset of high-quality search fixations in existence. COCO-Search18 has 10 participants searching for each of 18 target-object categories in 6202 images, making about 300,000 goal-directed fixations. When trained and evaluated on COCO-Search18, the IRL model outperformed baseline models in predicting search fixation scanpaths, both in terms of similarity to human search behavior and search efficiency. Finally, reward maps recovered by the IRL model reveal distinctive target-dependent patterns of object prioritization, which we interpret as a learned object context.

Deep gaze pooling: Inferring and visually decoding search intents from human gaze fixations

Predicting the target of visual search from human eye fixations (gaze) is a difficult problem with many applications, e.g. in human-computer interaction. While previous work has focused on predicting specific search target instances, we propose the first approach to predict categories and attributes of search intents from gaze data and to visually reconstruct plausible targets. However, state-of-the-art models for categorical recognition, in general, require large amounts of training data, which is prohibitive for gaze data. To address this challenge, we further propose a novel Gaze Pooling Layer that combines gaze information with visual representations from Deep Learning approaches. Our scheme incorporates both spatial and temporal aspects of human gaze behavior as well as the appearance of the fixated locations. We propose an experimental setup and novel dataset and demonstrate the effectiveness of our method for gaze-based search target prediction and reconstruction. We highlight several practical advantages of our approach, such as compatibility with existing architectures, no need for gaze training data, and robustness to noise from common gaze sources.

ISeeColor: Method for Advanced Visual Analytics of Eye Tracking Data

Recent advances in head-mounted eye-tracking technology have allowed researchers to monitor eye movements during locomotion in real-world environments, increasing the ecological validity of research on human gaze behavior. While collecting eye-tracking data is becoming more accessible, visual analytics of eye-tracking data remains difficult and time-consuming. As such, there is a significant need for developing efficient visualization and analysis tools for large-scale eye-tracking data. This work develops a first-of-its-kind eye-tracking data visualization and analysis system that allows for automatic recognition of independent objects within field-of-vision, using deep-learning-based semantic segmentation. This system recolors the fixated objects-of-interest by integrating gaze fixation information with semantic maps. The system effectively allows researchers to automatically infer what objects users view and for how long in dynamic contexts. The contributions are 1) a data visualization and analysis system that uses deep-learning technology along with eye-tracking data to automatically recognize objects-of-interest from head-mounted eye-tracking video recordings, and 2) a graphical user interface that presents objects-of-interest annotation along with eye-tracking data information. The architecture is tested with an outdoor case study of users walking around the Tufts University campus as part of a navigation study, which was administered by a team of research psychologists.

Toward Prediction of Driver Awareness of Automotive Hazards: Driving-Video-Based Simulation Approach

Automated vehicles need to monitor not only the dynamic driving environment but also the human driver’s behavior for appropriate assistance or intervention. One such challenge in observing the driver is the ability for vehicles to recognize whether the driver is aware of specific road hazards or not. As a first step toward data-driven predictive models of driver awareness, we propose a driving-video-based simulation method for empirical data collection in an ecologically valid and safe environment. A human-subject experiment conducted in a driving simulator demonstrated the potential of the proposed method. Furthermore, this work investigated the relationship between driver awareness and eye movement. Our preliminary results support that human gaze behavior may be a promising but insufficient indicator of situation awareness, and, in particular, human perception of road hazards. The proposed approach can be further used for larger-scale data collections to inform machine learning algorithms for prediction of driver awareness based on observable driver behavior and the characteristics of road hazards.

Scene Categorization by Deeply Learning Gaze Behavior in a Semisupervised Context.

Accurately recognizing different categories of sceneries with sophisticated spatial configurations is a useful technique in computer vision and intelligent systems, e.g., scene understanding and autonomous driving. Competitive accuracies have been observed by the deep recognition models recently. Nevertheless, these deep architectures cannot explicitly characterize human visual perception, that is, the sequence of gaze allocation and the subsequent cognitive processes when viewing each scenery. In this paper, a novel spatially aware aggregation network is proposed for scene categorization, where the human gaze behavior is discovered in a semisupervised setting. In particular, as semantically labeling a large quantity of scene images is labor-intensive, a semisupervised and structure-preserved non-negative matrix factorization (NMF) is proposed to detect a set of visually/semantically salient regions from each scenery. Afterward, the gaze shifting path (GSP) is engineered to characterize the process of humans perceiving each scene picture. To deeply describe each GSP, a novel spatially aware CNN termed SA-Net is developed. It accepts input regions with various shapes and statistically aggregates all the salient regions along each GSP. Finally, the learned deep GSP features from the entire scene images are fused into an image kernel, which is subsequently integrated into a kernel SVM to categorize different sceneries. Comparative experiments on six scene image sets have shown the advantage of our method.

How does navigation system behavior influence human behavior?

Navigation systems are ubiquitous tools to assist wayfinders of the mobile information society with various navigational tasks. Whenever such systems assist with self-localization and path planning, they reduce human effort for navigating. Automated navigation assistance benefits navigation performance, but research seems to show that it negatively affects attention to environment properties, spatial knowledge acquisition, and retention of spatial information. Very little is known about how to design navigation systems for pedestrian navigation that increase both navigation performance and spatial knowledge acquisition. To this end, we empirically tested participants (N = 64) using four different navigation system behaviors (between-subject design). Two cognitive processes with varying levels of automation, self-localization and allocation of attention, define navigation system behaviors: either the system automatically executes one of the processes (high level of automation), or the system leaves the decision of when and where to execute the process to the navigator (low level of automation). In two experimental phases, we applied a novel empirical framework for evaluating spatial knowledge acquisition in a real-world outdoor urban environment. First, participants followed a route assisted by a navigation system and, simultaneously, incidentally acquired spatial knowledge. Second, participants reversed the route using the spatial knowledge acquired during the assisted phase, this time without the aid of the navigation system. Results of the route-following phase did not reveal differences in navigation performance across groups using different navigation system behaviors. However, participants using systems with higher levels of automation seemed not to acquire enough spatial knowledge to reverse the route without navigation errors. Furthermore, employing novel methods to analyze mobile eye tracking data revealed distinct patterns of human gaze behavior over time and space. We thus can demonstrate how to increase spatial knowledge acquisition without harming navigation performance when using navigation systems, and how to influence human navigation behavior with varying navigation system behavior. Thus, we provide key findings for the design of intelligent automated navigation systems in real-world scenarios.

Aesthetics-Guided Graph Clustering With Absent Modalities Imputation.

Accurately clustering Internet-scale Internet users into multiple communities according to their aesthetic styles is a useful technique in image modeling and data mining. In this paper, we present a novel partially supervised model which seeks a sparse representation to capture photo aesthetics. It optimally fuzes multi-channel features, i.e., human gaze behavior, quality scores, and semantic tags, each of which could be absent. Afterward, by leveraging the KL-divergence to distinguish the aesthetic distributions between photo sets, a large-scale graph is constructed to describe the aesthetic correlations between users. Finally, a dense subgraph mining algorithm which intrinsically supports outliers (i.e., unique users not belong to any community) is adopted to detect aesthetic communities. The comprehensive experimental results on a million-scale image set grabbed from Flickr have demonstrated the superiority of our method. As a byproduct, the discovered aesthetic communities can enhance photo retargeting and video summarization substantially.

Perceptually learning multi-view sparse representation for scene categorization

Utilizing multi-channel visual features to characterize scenery images is standard for state-of-the-art scene recognition systems. However, how to encode human visual perception for scenery image modeling and how to optimally combine visual features from multiple views remains a tough challenge. In this paper, we propose a perceptual multi-view sparse learning (PMSL) framework to distinguish sceneries from different categories. Specifically, we first project regions from each scenery into the so-called perceptual space, which is established by combining human gaze behavior, color and texture. Afterward, a novel PMSL is developed which fuzes the above three visual cues into a sparse representation. PMSL can support absent channel visual features, which is frequently occurred in practical circumstances. Finally, the sparse representation from each scenery image is incorporated into an image kernel, which is further fed into a kernel SVM for scene categorization. Comprehensive experimental results on popular data sets have demonstrated the superiority of our method over well-known shallow/deep recognition models.

Multi-view non-negative matrix factorization for scene recognition

Accurately discriminating complicated sceneries from different categories is a useful technique in multimedia and computer vision. In this work, we propose a novel multi-view non-negative matrix factorization to detect human gaze behavior, which is subsequently integrated into an image kernel machine for scene categorization. More specifically, we first project regions from each scenery into the so-called perceptual space, which is established by combining color, texture, and semantic features. Then, a novel non-negative matrix factorization (NMF) algorithm is developed which decomposes the regions’ feature matrix into the product of the basis matrix and the sparse codes. The sparse codes indicate the saliency level of different regions which is used to constructed gaze shifting path. Thereby, the path from each scenery is derived and further incorporated into an image kernel for scene categorization. Comprehensive experiments on six scenery data sets have demonstrated the superiority of our method over a series of recognition models.

Nature Catches the Eye – Human Gaze Behaviour as a Detector of Spontaneous Visual Attention

Nature Catches the Eye – Human Gaze Behaviour as a Detector of Spontaneous Visual Attention

Perceptually Aware Image Retargeting for Mobile Devices.

Retargeting aims at adapting an original high-resolution photograph/video to a low-resolution screen with an arbitrary aspect ratio. Conventional approaches are generally based on desktop PCs, since the computation might be intolerable for mobile platforms (especially when retargeting videos). Typically, only low-level visual features are exploited, and human visual perception is not well encoded. In this paper, we propose a novel retargeting framework that rapidly shrinks a photograph/video by leveraging human gaze behavior. Specifically, we first derive a geometry-preserving graph ranking algorithm, which efficiently selects a few salient object patches to mimic the human gaze shifting path (GSP) when viewing a scene. Afterward, an aggregation-based CNN is developed to hierarchically learn the deep representation for each GSP. Based on this, a probabilistic model is developed to learn the priors of the training photographs that are marked as aesthetically pleasing by professional photographers. We utilize the learned priors to efficiently shrink the corresponding GSP of a retargeted photograph/video to maximize its similarity to those from the training photographs. Extensive experiments have demonstrated that: 1) our method requires less than 35 ms to retarget a photograph (or a video frame) on popular iOS/Android devices, which is orders of magnitude faster than the conventional retargeting algorithms; 2) the retargeted photographs/videos produced by our method significantly outperform those of its competitors based on a paired-comparison-based user study; and 3) the learned GSPs are highly indicative of human visual attention according to the human eye tracking experiments.

Cognition-Cognizant Sentiment Analysis With Multitask Subjectivity Summarization Based on Annotators' Gaze Behavior

For document level sentiment analysis (SA), Subjectivity Extraction, ie., extracting the relevant subjective portions of the text that cover the overall sentiment expressed in the document, is an important step. Subjectivity Extraction, however, is a hard problem for systems, as it demands a great deal of world knowledge and reasoning. Humans, on the other hand, are good at extracting relevant subjective summaries from an opinionated document (say, a movie review), while inferring the sentiment expressed in it. This capability is manifested in their eye-movement behavior while reading: words pertaining to the subjective summary of the text attract a lot more attention in the form of gaze-fixations and/or saccadic patterns. We propose a multi-task deep neural framework for document level sentiment analysis that learns to predict the overall sentiment expressed in the given input document, by simultaneously learning to predict human gaze behavior and auxiliary linguistic tasks like part-of-speech and syntactic properties of words in the document. For this, a multi-task learning algorithm based on multi-layer shared LSTM augmented with task specific classifiers is proposed. With this composite multi-task network, we obtain performance competitive with or better than state-of-the-art approaches in SA. Moreover, the availability of gaze predictions as an auxiliary output helps interpret the system better; for instance, gaze predictions reveal that the system indeed performs subjectivity extraction better, which accounts for improvement in document level sentiment analysis performance.

InvisibleEye

Analysis of everyday human gaze behaviour has significant potential for ubiquitous computing, as evidenced by a large body of work in gaze-based human-computer interaction, attentive user interfaces, and eye-based user modelling. However, current mobile eye trackers are still obtrusive, which not only makes them uncomfortable to wear and socially unacceptable in daily life, but also prevents them from being widely adopted in the social and behavioural sciences. To address these challenges we present InvisibleEye, a novel approach for mobile eye tracking that uses millimetre-size RGB cameras that can be fully embedded into normal glasses frames. To compensate for the cameras’ low image resolution of only a few pixels, our approach uses multiple cameras to capture different views of the eye, as well as learning-based gaze estimation to directly regress from eye images to gaze directions. We prototypically implement our system and characterise its performance on three large-scale, increasingly realistic, and thus challenging datasets: 1) eye images synthesised using a recent computer graphics eye region model, 2) real eye images recorded of 17 participants under controlled lighting, and 3) eye images recorded of four participants over the course of four recording sessions in a mobile setting. We show that InvisibleEye achieves a top person-specific gaze estimation accuracy of 1.79° using four cameras with a resolution of only 5 × 5 pixels. Our evaluations not only demonstrate the feasibility of this novel approach but, more importantly, underline its significant potential for finally realising the vision of invisible mobile eye tracking and pervasive attentive user interfaces.

場の盛り上がり推定モデルを用いた視線インタラクションを支援する音声駆動型身体引き込みキャラクタシステム

Line-of-sight such as gaze and eye-contract plays an important role to enhance the embodied interaction and communication through avatars. In addition, many eyeball movement models and communication systems using avatars with the line-of-sight have been proposed and developed. However, these eyeball movements by generating the above-mentioned models were not considered to enhance the embodied interaction such as activated communication, because the models stochastically generate the eyeball movements based on the human gaze behavior. Therefore, in order to enhance and promote the embodied interaction from the view point of line-of-sight, it is desired to design a line-of-sight interaction which relates to the interaction-activated communication. In this study, we analyze the interaction between human gaze behavior and the interaction-activated communication by using an embodied communication system with line-of-sight measurement devices. Then, we propose a line-of-sight model based on the above-mentioned analysis, and develop an advanced embodied communication system by applying the proposed model to the speech-driven embodied entrainment character. This system generates the eyeball movements such as gaze and looking away based on the proposed model, and provides a communication environment wherein the embodied interaction is promoted. The effectiveness of the system is demonstrated by means of sensory evaluations of 15 pairs of subjects involved in avatar-mediated communication.

Effects of color of book cover and typeface of title and author name on gaze duration and choice behavior for books: Evidence from an eye‐tracking experiment

ABSTRACTThis study explores the effect of color of book cover and typeface of title and author name on human gaze behavior while browsing bookshelves. An eye‐tracking experiment was performed with 23 participants, using a wearable eye‐tracker and 96 books. The difference of gaze duration time and the number of book subjects chosen was analyzed in two categories: between warm color book covers and cold color ones, and between Ming‐style typeface and Gothic typeface. Results showed no effect of book cover color on gaze duration and user choice. However, typeface of book title and author name had an effect. It was revealed that Gothic typeface was more attractive if book titles had meaning.

Preference bias of head orientation in choosing between two non-durables.

The goal of this study is to investigate how customers' gaze, head and body orientations reflect their choices. Although the relationship between human choice and gaze behavior has been well-studied, other behaviors such as head and body are unknown. We conducted a two-alternatives-forced-choice task to examine (1) whether preference bias, i.e., a positional bias in gaze, head and body toward the item that was later chosen, exists in choice, (2) when preference bias is observed and when prediction of the resulting choice becomes possible (3) whether human choice is affected when the body orientations are manipulated. We used real non-durable products (cheap snacks and clothing) on a shopping shelf. The results showed that there was a significant preference bias in head orientation at the beginning 1 s when the subjects stood straight toward the shelf, and that the head orientation was more biased toward the selected item than the gaze and the center of pressure at the ending 1 s. Manipulating body orientation did not affect the result of choice. The preference bias detected by observing the head orientation would be useful in marketing science for predicting customers' choice.

Impacts of Robot Head Gaze on Robot-to-Human Handovers

In this paper, we investigate the use of a robot's gazetoimprovethetimingandsubjectiveexperienceofface- to-face robot-to-human handovers. Based on observations of human gaze behaviors during face-to-face human-human handovers, we implement various gaze behaviors on a PR2 humanoid robot. We conducted two consecutive robot-to- human handover studies. Results show that when the robot continually gazes at a projected handover position while handing over an object, the human receivers reach for the object significantly earlier than when the robot looks down, away from the handover location; further, when the robot continually gazes at the receiver's face instead of the handover position, the receivers reach for the object even earlier. When the robot—instead of continually gazing at a location—transitions its gaze from the handover position to the receivers' face, or vice versa, the receivers' reach time did not improve; however, the receivers perceive these gaze transitions to better communicate handover timing than con- tinual gazes. Finally, the receivers perceive the robot to be more likeable and anthropomorphic when it looks at their

What/Where to Look Next? Modeling Top-Down Visual Attention in Complex Interactive Environments

Several visual attention models have been proposed for describing eye movements over simple stimuli and tasks such as free viewing or visual search. Yet, to date, there exists no computational framework that can reliably mimic human gaze behavior in more complex environments and tasks such as urban driving. In addition, benchmark datasets, scoring techniques, and top-down model architectures are not yet well understood. In this paper, we describe new task-dependent approaches for modeling top-down overt visual attention based on graphical models for probabilistic inference and reasoning. We describe a dynamic Bayesian network that infers probability distributions over attended objects and spatial locations directly from observed data. Probabilistic inference in our model is performed over object-related functions that are fed from manual annotations of objects in video scenes or by state-of-the-art object detection/recognition algorithms. Evaluating over approximately 3 h (approximately 315 000 eye fixations and 12 000 saccades) of observers playing three video games (time-scheduling, driving, and flight combat), we show that our approach is significantly more predictive of eye fixations compared to: 1) simpler classifier-based models also developed here that map a signature of a scene (multimodal information from gist, bottom-up saliency, physical actions, and events) to eye positions; 2) 14 state-of-the-art bottom-up saliency models; and 3) brute-force algorithms such as mean eye position. Our results show that the proposed model is more effective in employing and reasoning over spatio-temporal visual data compared with the state-of-the-art.

Designing and Evaluating a Social Gaze-Control System for a Humanoid Robot

This paper describes a context-dependent social gaze-control system implemented as part of a humanoid social robot. The system enables the robot to direct its gaze at multiple humans who are interacting with each other and with the robot. The attention mechanism of the gaze-control system is based on features that have been proven to guide human attention: nonverbal and verbal cues, proxemics, the visual field of view, and the habituation effect. Our gaze-control system uses Kinect skeleton tracking together with speech recognition and SHORE-based facial expression recognition to implement the same features. As part of a pilot evaluation, we collected the gaze behavior of 11 participants in an eye-tracking study. We showed participants videos of two-person interactions and tracked their gaze behavior. A comparison of the human gaze behavior with the behavior of our gaze-control system running on the same videos shows that it replicated human gaze behavior 89% of the time.