Computational Models Of Visual Attention Research Articles

Distinct eye movement patterns to complex scenes in Alzheimer's disease and Lewy body disease.

Alzheimer's disease (AD) and Lewy body disease (LBD), the two most common causes of neurodegenerative dementia with similar clinical manifestations, both show impaired visual attention and altered eye movements. However, prior studies have used structured tasks or restricted stimuli, limiting the insights into how eye movements alter and differ between AD and LBD in daily life. We aimed to comprehensively characterize eye movements of AD and LBD patients on naturalistic complex scenes with broad categories of objects, which would provide a context closer to real-world free viewing, and to identify disease-specific patterns of altered eye movements. We collected spontaneous viewing behaviors to 200 naturalistic complex scenes from patients with AD or LBD at the prodromal or dementia stage, as well as matched control participants. We then investigated eye movement patterns using a computational visual attention model with high-level image features of object properties and semantic information. Compared with matched controls, we identified two disease-specific altered patterns of eye movements: diminished visual exploration, which differentially correlates with cognitive impairment in AD and with motor impairment in LBD; and reduced gaze allocation to objects, attributed to a weaker attention bias toward high-level image features in AD and attributed to a greater image-center bias in LBD. Our findings may help differentiate AD and LBD patients and comprehend their real-world visual behaviors to mitigate the widespread impact of impaired visual attention on daily activities.

Interacting rhythms enhance sensitivity of target detection in a fronto-parietal computational model of visual attention.

Even during sustained attention, enhanced processing of attended stimuli waxes and wanes rhythmically, with periods of enhanced and relatively diminished visual processing (and subsequent target detection) alternating at 4 or 8 Hz in a sustained visual attention task. These alternating attentional states occur alongside alternating dynamical states, in which lateral intraparietal cortex (LIP), the frontal eye field (FEF), and the mediodorsal pulvinar (mdPul) exhibit different activity and functional connectivity at α, β, and γ frequencies-rhythms associated with visual processing, working memory, and motor suppression. To assess whether and how these multiple interacting rhythms contribute to periodicity in attention, we propose a detailed computational model of FEF and LIP. When driven by θ-rhythmic inputs simulating experimentally-observed mdPul activity, this model reproduced the rhythmic dynamics and behavioral consequences of observed attentional states, revealing that the frequencies and mechanisms of the observed rhythms allow for peak sensitivity in visual target detection while maintaining functional flexibility.

The Alleviation of Perceptual Blindness During Driving in Urban Areas Guided by Saccades Recommendation

In advanced industrial applications, computational visual attention models (CVAMs) could predict visual attention very similarly to actual human attention allocation. This has been used as a very important component of technology in advanced driver assistance systems (ADAS). Given that the biological inspiration of the driving-related CVAMs could be obtained from skilled drivers in complex driving conditions, in which the driver’s attention is constantly directed at various salient and informative visual stimuli by alternating the eye fixations via saccades to drive safely, this paper proposes a saccade recommendation strategy to enhance the driving safety under urban road environment, particularly when the driver’s vision is often impaired by the visual crowding. The altered and directed saccades are collected and optimized by extracting four innate features from human dynamic vision. A neural network isdesigned to classify preferable saccades to reduce perceptual blindness due to visual crowding under urban scenes. A state-of-the-art CVAM is firstly adopted to localize the predicted eye fixation locations (EFLs) in driving video clips. Besides, human subjects’ gaze at the recommended EFLs is measured via an eye-tracker. The time delays between the predicted EFLs and drivers’ EFLs are analyzed under different driving conditions, followed by the time delays between the predicted EFLs and the driver’s hand control. The visually safe margin is then measured by mediating the driving speed and the total delay. Experimental results demonstrate that the recommended saccades can effectively reduce the amount of perceptual blindness, which is known to be of help to further improve road driving safety.

Reclaiming saliency: Rhythmic precision-modulated action and perception.

Computational models of visual attention in artificial intelligence and robotics have been inspired by the concept of a saliency map. These models account for the mutual information between the (current) visual information and its estimated causes. However, they fail to consider the circular causality between perception and action. In other words, they do not consider where to sample next, given current beliefs. Here, we reclaim salience as an active inference process that relies on two basic principles: uncertainty minimization and rhythmic scheduling. For this, we make a distinction between attention and salience. Briefly, we associate attention with precision control, i.e., the confidence with which beliefs can be updated given sampled sensory data, and salience with uncertainty minimization that underwrites the selection of future sensory data. Using this, we propose a new account of attention based on rhythmic precision-modulation and discuss its potential in robotics, providing numerical experiments that showcase its advantages for state and noise estimation, system identification and action selection for informative path planning.

A Computational-Cognitive Model of Visual Attention in Dynamic Environments

Background and Objectives: Visual attention is a high order cognitive process of human brain which defines where a human observer attends. Dynamic computational visual attention models are modeled on the behavior of the human brain and can predict what areas a human will pay attention to when viewing a scene such as a video. However, several types of computational models have been proposed to provide a better understanding of saliency maps in static and dynamic environments, most of these models are used for specific scenes. In this paper, we propose a model that can generate saliency maps in a variety of dynamic environments with complex scenes.Methods: We used a deep learner as a mediating network to combine basic saliency maps with appropriate weighting. Each of these basic saliency maps covers an important feature of human visual attention, and ultimately the final saliency map is very similar to human visual behavior.Results: The proposed model is run on two datasets and the generated saliency maps are evaluated by different criteria such as ROC, CC, NSS, SIM and KLdiv. The results show that the proposed model has a good performance compared to other similar models.Conclusion: The proposed model consists of three main parts, including basic saliency maps, gating network, and combinator. This model was implemented on the ETMD dataset and the resulting saliency maps (visual attention areas) were compared with some other models in this field by evaluation criteria and their results were evaluated. The results obtained from the proposed model are acceptable and based on the accepted evaluation criteria in this area, it performs better than similar models.

A method for the analysis of the interaction between users and objects in 3D navigational space

Along with the improvement of eye-tracking technology, more and more distinct field of researches have introduced movements of the eye in relation to the head to understand user behavior. Most of current researches focus on the perception process of single 2-dimensional images by fixed eye-tracking devices or the head-mount devices. A method of applying eye-tracking on the analysis of the interaction between users and objects in 3D navigational space is proposed in this article. It aims to understand the visual stimulation of 3D objects and the user’s spatial navigational reactions while receiving the stimulation, and proposes the concept of 3D object attention heat map. It also proposes to construct a computational visual attention model for different geometric featured 3D objects by applying the method of feature curves. The VR results of this study also provide future assistance in the incoming immersive world. This study sets to promote eye-tracking from the mainstream of 2D field to 3D spaces and points to a deeper understanding between human and artificial product or natural objects. It would also serve an important role in the field of human-computer interaction, product usability, aids devices for cognition degenerative individuals, and even the field of visual recognition of daily human behavior.

The Improvement of Road Driving Safety Guided by Visual Inattentional Blindness

The computational modeling of human visual attention has received much attention in recent decades. In advanced industrial applications, it has been demonstrated that computational visual attention models (CVAMs) can predict visual attention very similarly to human visual attention. However, it is controversial whether the driver’s eye fixation location (EFL) or the predicted eye fixation location of computational visual attention models is more reliable and helpful for actual driving. To address this issue, an open database of videos taken under the most common 18 driving conditions in everyday driving has been established. In experiments using this database, expert drivers found that it was not sufficient for drivers to rely on only one of the two EFLs. Based on this finding, a hybrid EFL recommendation strategy is proposed for improving driving safety. By extracting visual characteristics from human dynamic vision, the performance of the proposed recommendation method demonstrates its potential value in these collected driving tasks. In addition, the visual comfort of driving is further addressed to enhance the safety of driving. From the results of experiments on 108 driving video clips taken of the most common 18 real driving conditions, it is confirmed that the proposed EFL recommendation achieves an experience rating of driving comfort between 88.1 and 92.7 out of 100.

Bottom-up visual attention model for still image: a preliminary study

The philosophy of human visual attention is scientifically explained in the field of cognitive psychology and neuroscience then computationally modeled in the field of computer science and engineering. Visual attention models have been applied in computer vision systems such as object detection, object recognition, image segmentation, image and video compression, action recognition, visual tracking, and so on. This work studies bottom-up visual attention, namely human fixation prediction and salient object detection models. The preliminary study briefly covers from the biological perspective of visual attention, including visual pathway, the theory of visual attention, to the computational model of bottom-up visual attention that generates saliency map. The study compares some models at each stage and observes whether the stage is inspired by biological architecture, concept, or behavior of human visual attention. From the study, the use of low-level features, center-surround mechanism, sparse representation, and higher-level guidance with intrinsic cues dominate the bottom-up visual attention approaches. The study also highlights the correlation between bottom-up visual attention and curiosity.

A Temporally Irreversible Visual Attention Model Inspired by Motion Sensitive Neurons

When a human perceives videos composed of the same images in various orders, such as normal order, reverse order, and random order, the human visual attention system perceives them as different visual inputs. This means that the temporal change in the image sequence exerts a considerable influence on the human visual system. However, most state-of-the-art computational visual attention models have not considered the temporal cues adequately. Motivated by this deficiency, we propose a novel temporally irreversible visual attention model considering the following three aspects. First, the central bias of human dynamic vision is incorporated into the model to manifest this tendency. Second, the depth and directional motion-sensitive neurons are fused to discern different motion patterns. Third, the rarity factor is integrated into the model to mimic the attention shift when human observer perceives new emerging motion cues. The proposed model demonstrates its competitiveness to select attentive events in our experiments, in both laboratory setting and real driving video clips. When compared with recent visual attention models, the proposed model achieves the highest score in similarity with human dynamic vision. The proposed model could be one of the fundamental building blocks for any visual attention systems coping with dynamic scenes.

Video saliency detection by gestalt theory

Image saliency detection has been widely explored in recent decades, but computational modeling of visual attention for video sequences is limited due to complicated temporal saliency extraction and fusion of spatial and temporal saliency. Inspired by Gestalt theory, we introduce a novel spatiotemporal saliency detection model in this study. First, we compute spatial and temporal saliency maps by low-level visual features. And then we merge these two saliency maps for spatiotemporal saliency prediction of video sequences. The spatial saliency map is calculated by extracting three kinds of features including color, luminance, and texture, while the temporal saliency map is computed by extracting motion features estimated from video sequences. A novel adaptive entropy-based uncertainty weighting method is designed to fuse spatial and temporal saliency maps to predict the final spatiotemporal saliency map by Gestalt theory. The Gestalt principle of similarity is used to estimate spatial uncertainty from spatial saliency, while temporal uncertainty is computed from temporal saliency by the Gestalt principle of common fate. Experimental results on three large-scale databases show that our method can predict visual saliency more accurately than the state-of-art spatiotemporal saliency detection algorithms.

Attention Based Visual Analysis for Fast Grasp Planning With a Multi-Fingered Robotic Hand.

We present an attention based visual analysis framework to compute grasp-relevant information which helps to guide grasp planning using a multi-fingered robotic hand. Our approach uses a computational visual attention model to locate regions of interest in a scene and employ a deep convolutional neural network to detect grasp type and grasp attention point for a sub-region of the object in a region of interest. We demonstrate the proposed framework with object grasping tasks, in which the information generated from the proposed framework is used as prior information to guide grasp planning. The effectiveness of the proposed approach is evaluated in both simulation experiments and real-world experiments. Experimental results show that the proposed framework can not only speed up grasp planning with more stable configurations, but also handle unknown objects. Furthermore, our framework can handle cluttered scenarios. A new Grasp Type Dataset (GTD) which includes six commonly used grasp types and covers 12 household objects is also presented.

Hierarchical visual attention model for saliency detection inspired by avian visual pathways

Visual attention is a mechanism that enables the visual system to detect potentially important objects in complex environment. Most computational visual attention models are designed with inspirations from mammalian visual systems. However, electrophysiological and behavioral evidences indicate that avian species are animals with high visual capability that can process complex information accurately in real time. Therefore, the visual system of the avian species, especially the nuclei related to the visual attention mechanism, are investigated in this paper. Afterwards, a hierarchical visual attention model is proposed for saliency detection. The optic tectum neuron responses are computed and the self-information is used to compute primary saliency maps in the first hierarchy. The &#x201C winner-take-all &#x201D network in the tecto-isthmal projection is simulated and final saliency maps are estimated with the regularized random walks ranking in the second hierarchy. Comparison results verify that the proposed model, which can define the focus of attention accurately, outperforms several state-of-the-art models. This study provides insights into the relationship between the visual attention mechanism and the avian visual pathways. The computational visual attention model may reveal the underlying neural mechanism of the nuclei for biological visual attention.

Using Human Attention to Address Human–Robot Motion

Let human–robot motion (HRM) denote the study of how robots should move among people, the work presented herein explores to what extent human attention can be useful to address the HRM. To that end, a computational model of the human visual attention is proposed to estimate how a person's attentional resources are distributed among the elements in their environment. Based on this model, the concept of attention field for a robot is used to define different attentional properties for the robot's motions such as distraction or surprise. The relevance of the attentional properties for HRM is demonstrated on a proof-of-concept acceptable motion planner on various case studies where a robot is assigned different tasks. It is shown how to compute motions that are non-distracting and non-surprising, but also motions that convey the robot's intention to interact with a person.

Eye Fixation Location Recommendation in Advanced Driver Assistance System

Recent research progress on the approach of visual attention modeling for mediated perception to advanced driver assistance system (ADAS) has drawn the attention of computer and human vision researchers. However, it is still debatable whether the actual driver’s eye fixation locations (EFLs) or the predicted EFLs which are calculated by computational visual attention models (CVAMs) are more reliable for safe driving under real-life driving conditions. We analyzed the suitability of the following two EFLs using ten typical categories of natural driving video clips: the EFLs of human drivers and the EFLs predicted by CVAMs. In the suitability analysis, we used the EFLs confirmed by two experienced drivers as the reference EFLs. We found that both approaches alone are not suitable for safe driving and EFL suitable for safe driving depends on the driving conditions. Based on this finding, we propose a novel strategy for recommending one of the EFLs to the driver for ADAS under predefined 10 real-life driving conditions. We propose to recommend one of the following 3 EFL modes for different driving conditions: driver’s EFL only, CVAM’s EFL only, and interchangeable EFL. In interchangeable EFL mode, driver’s EFL and CVAM’s EFL are interchangeable. The selection of two EFLs is a typical binary classification problem, so we apply support vector machines (SVMs) to solve this problem. We also provide a quantitative evaluation of the classifiers. The performance evaluation of the proposed recommendation method indicates that it is potentially useful to ADAS for future safe driving.

A bio-inspired motion sensitive model and its application to estimating human gaze positions under classified driving conditions

Human visual attention performs the spatio-temporal regularity, an inherent regularity in dynamic vision, for the natural visual tasks. However, recent computational visual attention models still have deficiency to reflect the spatio-temporal regularity thoroughly. Motivated by this, we propose a bio-inspired motion sensitive model to estimate human gaze positions in driving. The proposed model has four key advantages. First, inspired by two types of motion sensitive neurons, this model can perceive the motion cues from different directions spatially and temporally. Second, compared to conventional deep learning based models, this model does not rely on expensive training samples with gaze annotations. Third, the proposed model is based upon the visual signal processing without constructing a complex deep neural network architecture, which enables this model to be implemented with low-cost hardware. Fourth, inspired by the visual pathway in drosophila motion vision, the visual signal processing on directional and depth motion sensitive map largely enhances this model's competence in a similar way of human gaze positions in driving. To test this proposed model, we collect a driving scene dataset from the perspective of egocentric vision that aims to systematically evaluate the performance of spatio-temporal visual attentional models. The video clips in the dataset are categorized into ten popular driving conditions. The proposed model is evaluated by comparing with the human baseline of the gaze positions. Experimental results demonstrate that the proposed model can effectively estimate the human gaze positions in driving and consistently outperforms traditional visual attention models as well as the deep learning based model.

Top-down contributions to attention shifting and disengagement: A template model of visual attention.

Two separate systems are involved in the control of spatial attention; one that is driven by a goal, and the other that is driven by stimuli. While the goal- and stimulus-driven systems follow different general principles, they also interplay with each other. However, the mechanism by which the goal-driven system influences the stimulus-driven system is still debated. The present study examined top-down contributions to two components of attention orienting, shifting and disengagement, with an experimental paradigm in which participants held a visual item in short-term memory (STM) and performed a prosaccade task with a manipulation of the gap between fixation offset and target onset. Four experiments showed that the STM content accelerated shifting and impaired disengagement, but the influence on disengagement depended on the utility of STM in guiding attention toward the target. Thus, the use of STM was strategic. Computational models of visual attention were fitted to the experimental data, which suggested that the top-down contributions to shifting was more prominent than those to disengagement. The results indicate that the current modeling framework was particularly useful when examining the contributions of theoretical constructs for the control of visual attention, but it also suggests limitations. (PsycINFO Database Record

Saliency in VR: How Do People Explore Virtual Environments?

Understanding how people explore immersive virtual environments is crucial for many applications, such as designing virtual reality (VR) content, developing new compression algorithms, or learning computational models of saliency or visual attention. Whereas a body of recent work has focused on modeling saliency in desktop viewing conditions, VR is very different from these conditions in that viewing behavior is governed by stereoscopic vision and by the complex interaction of head orientation, gaze, and other kinematic constraints. To further our understanding of viewing behavior and saliency in VR, we capture and analyze gaze and head orientation data of 169 users exploring stereoscopic, static omni-directional panoramas, for a total of 1980 head and gaze trajectories for three different viewing conditions. We provide a thorough analysis of our data, which leads to several important insights, such as the existence of a particular fixation bias, which we then use to adapt existing saliency predictors to immersive VR conditions. In addition, we explore other applications of our data and analysis, including automatic alignment of VR video cuts, panorama thumbnails, panorama video synopsis, and saliency-basedcompression.

Remote Sensing Scene Classification Based on Convolutional Neural Networks Pre-Trained Using Attention-Guided Sparse Filters

Semantic-level land-use scene classification is a challenging problem, in which deep learning methods, e.g., convolutional neural networks (CNNs), have shown remarkable capacity. However, a lack of sufficient labeled images has proved a hindrance to increasing the land-use scene classification accuracy of CNNs. Aiming at this problem, this paper proposes a CNN pre-training method under the guidance of a human visual attention mechanism. Specifically, a computational visual attention model is used to automatically extract salient regions in unlabeled images. Then, sparse filters are adopted to learn features from these salient regions, with the learnt parameters used to initialize the convolutional layers of the CNN. Finally, the CNN is further fine-tuned on labeled images. Experiments are performed on the UCMerced and AID datasets, which show that when combined with a demonstrative CNN, our method can achieve 2.24% higher accuracy than a plain CNN and can obtain an overall accuracy of 92.43% when combined with AlexNet. The results indicate that the proposed method can effectively improve CNN performance using easy-to-access unlabeled images and thus will enhance the performance of land-use scene classification especially when a large-scale labeled dataset is unavailable.

Visual Attention Saccadic Models Learn to Emulate Gaze Patterns From Childhood to Adulthood.

How people look at visual information reveals fundamental information about themselves, their interests and their state of mind. While previous visual attention models output static 2D saliency maps, saccadic models aim to predict not only where observers look at but also how they move their eyes to explore the scene. In this paper, we demonstrate that saccadic models are a flexible framework that can be tailored to emulate observer's viewing tendencies. More specifically, we use fixation data from 101 observers split into five age groups (adults, 8-10 y.o., 6-8 y.o., 4-6 y.o., and 2 y.o.) to train our saccadic model for different stages of the development of human visual system. We show that the joint distribution of saccade amplitude and orientation is a visual signature specific to each age group, and can be used to generate age-dependent scan paths. Our age-dependent saccadic model does not only output human-like, age-specific visual scan paths, but also significantly outperforms other state-of-the-art saliency models. We demonstrate that the computational modeling of visual attention, through the use of saccadic model, can be efficiently adapted to emulate the gaze behavior of a specific group of observers.

Visual Attention Modeling for Stereoscopic Video: A Benchmark and Computational Model

In this paper, we investigate the visual attention modeling for stereoscopic video from the following two aspects. First, we build one large-scale eye tracking database as the benchmark of visual attention modeling for stereoscopic video. The database includes 47 video sequences and their corresponding eye fixation data. Second, we propose a novel computational model of visual attention for stereoscopic video based on Gestalt theory. In the proposed model, we extract the low-level features, including luminance, color, texture, and depth, from discrete cosine transform coefficients, which are used to calculate feature contrast for the spatial saliency computation. The temporal saliency is calculated by the motion contrast from the planar and depth motion features in the stereoscopic video sequences. The final saliency is estimated by fusing the spatial and temporal saliency with uncertainty weighting, which is estimated by the laws of proximity, continuity, and common fate in Gestalt theory. Experimental results show that the proposed method outperforms the state-of-the-art stereoscopic video saliency detection models on our built large-scale eye tracking database and one other database (DML-ITRACK-3D).