Human Visual Attention Model Research Articles

Attention Guided Imitation Learning and Reinforcement Learning


 
 
 We propose a framework that uses learned human visual attention model to guide the learning process of an imitation learning or reinforcement learning agent. We have collected high-quality human action and eye-tracking data while playing Atari games in a carefully controlled experimental setting. We have shown that incorporating a learned human gaze model into deep imitation learning yields promising results.
 
 

Top-Down Neural Attention by Excitation Backprop

We aim to model the top-down attention of a Convolutional Neural Network (CNN) classifier for generating task-specific attention maps. Inspired by a top-down human visual attention model, we propose a new backpropagation scheme, called Excitation Backprop, to pass along top-down signals downwards in the network hierarchy via a probabilistic Winner-Take-All process. Furthermore, we introduce the concept of contrastive attention to make the top-down attention maps more discriminative. In experiments, we demonstrate the accuracy and generalizability of our method in weakly supervised localization tasks on the MS COCO, PASCAL VOC07 and ImageNet datasets. The usefulness of our method is further validated in the text-to-region association task. On the Flickr30k Entities dataset, we achieve promising performance in phrase localization by leveraging the top-down attention of a CNN model that has been trained on weakly labeled web images.

An efficient visual saliency detection model based on Ripplet transform

Even though there have been great advancements in computer vision tasks, the development of human visual attention models is still not well investigated. In day-to-day life, one can find ample applications of saliency detection in image and video processing. This paper presents an efficient visual saliency detection model based on Ripplet transform, which aims at detecting the salient region and achieving higher Receiver Operating Characteristics (ROC). Initially the feature maps are obtained from Ripplet transform in different scales and different directions of the image. The global and local saliency maps are computed based on the global probability density distribution and feature distribution of local areas, which are combined together to get the final saliency map. Ripplet-transform-based visual saliency detection is the novel approach carried out in this paper. Experimental results indicate that the proposed method based on Ripplet transformation can give excellent performance in terms of precision, recall, F measure and Mean Absolute Error (MAE), and is compared with 10 state-of-the-art methods on five benchmark datasets.

Temporal Binding and Segmentation in Visual Search: A Computational Neuroscience Analysis.

Human visual search operates not only over space but also over time, as old items remain in the visual field and new items appear. Preview search (where one set of distractors appears before the onset of a second set) has been used as a paradigm to study search over time and space [Watson, D. G., & Humphreys, G. W. Visual marking: Prioritizing selection for new objects by top-down attentional inhibition of old objects. Psychological Review, 104, 90-122, 1997], with participants showing efficient search when old distractors can be ignored and new targets prioritized. The benefits of preview search are lost, however, if a temporal gap is introduced between a first presentation of the old items and the re-presentation of all the items in the search display [Kunar, M. A., Humphreys, G. W., & Smith, K. J. History matters: The preview benefit in search is not onset capture. Psychological Science, 14, 181-185, 2003a], consistent with the old items being bound by temporal onset to the new stimuli. This effect of temporal binding can be eliminated if the old items reappear briefly before the new items, indicating also a role for the memory of the old items. Here we simulate these effects of temporal coding in search using the spiking search over time and space model [Mavritsaki, E., Heinke, D., Allen, H., Deco, G., & Humphreys, G. W. Bridging the gap between physiology and behavior: Evidence from the sSoTS model of human visual attention. Psychological Review, 118, 3-41, 2011]. We show that a form of temporal binding by new onsets has to be introduced to the model to simulate the effects of a temporal gap, but that effects of the memory of the old item can stem from continued neural suppression across a temporal gap. We also show that the model can capture the effects of brain lesion on preview search under the different temporal conditions. The study provides a proof-of-principle analysis that neural suppression and temporal binding can be sufficient to account for human search over time and space.

A matter of hand: Causal links between hand dominance, structural organization of fronto-parietal attention networks, and variability in behavioural responses to transcranial magnetic stimulation

Considerable evidence suggests that, on a group level, human visuospatial attention is asymmetrically organized, with a right-hemispheric dominance. The asymmetrical organization of the superior longitudinal fasciculus (SLF) has been shown to account for the right-hemispheric dominance in visual attention. However, such account is by no means universal, and large individual differences in asymmetrical performance on visuospatial tasks have been reported. Furthermore, the variability in the SLF lateralization has been shown to correlate with behavioural asymmetries. Continuous theta burst stimulation (cTBS) enables to temporarily interfere with cortical activity. cTBS applied over the posterior parietal cortex (PPC) has been previously used to systematically study attentional asymmetries. Interestingly, large individual differences in the effectiveness of stimulation have been reported. In accordance with earlier both animal and human studies, one possible cause underlying these striking individual differences might lie in the structural organization of frontoparietal pathways subserving visuospatial attention. Thus, the current study employed diffusion tractography to examine the relationship between the variability in the structural organization of the SLF and the individual differences in attentional shifts induced by a modified cTBS (cTBSmod; triplets of pulses at 30 Hz, repeated at 6 Hz) applied over the IPS, as measured by a line bisection task. Consistent with previous studies, on a group level, cTBSmod applied over the right intraparietal sulcus (IPS) triggered a rightward bisection bias shift, and there were no significant effects of cTBSmod applied over the left IPS. However, further analyses demonstrated that both handedness and structural variability (as assessed based on hindrance modulated orientational anisotropy) within the middle and the ventral branches of the SLF predicted individual differences in the cTBSmod-induced attentional shifts. Our study thus suggests that the effects of cTBSmod over the IPS may depend on intra-hemispheric interactions between cortical loci controlling visual attention. To conclude, our findings provide converging evidence for the notion put forward previously that inter-individual variability in the structural organization of intra-hemispheric frontoparietal connections has important implications for the functional models of human visual attention. Moreover, we hypothesize that this may also be relevant for the understanding of attentional disorders and their rehabilitation.

Human Visual Attention Model Based on Analysis of Magic for Smooth Human–Robot Interaction

In order to smoothly interact with humans, it is desirable that a robot can guide human attention and behaviors. In this study, we developed a model of human visual attention for guiding human attention based on an analysis of a magic trick performance. We measured human gaze points of people watching a video of a magic trick performance and compared them with the area where the magician intended to draw a spectator’s attention. The analysis showed that the relationship between the magician’s face, hands, and gaze plays an important role in guiding the spectator’s attention. On the basis of the preliminary user studies on watching the magic video, we integrated a saliency map and a manipulation map that describes the relationship between gaze and hands to develop a novel human attention model. The evaluation using the observed gaze points demonstrated that the proposed model can better explain human visual attention than the saliency map while people are watching a video of a magic trick performance.

An application of a bio-inspired visual attention model for the localization of vehicle parts

The automated servicing of vehicles is becoming more and more a reality in today's world. While certain operations, such as car washing, require only a rough model of the surface of a vehicle, other operations, such as changing of a wheel or filling the gas tank, require a correct localization of the different parts of the vehicle on which operations are to be performed. The paper describes a two-step approach to localize vehicle parts over the surface of a vehicle in front, rear and lateral views capitalizing on a novel approach based on bio-inspired visual attention. First, bounding-boxes are determined based on a model of human visual attention to roughly locate parts of interest. Second, the bounding-boxes are further searched to finely tune and better capture the boundaries of each vehicle part by means of active contour models. The proposed method obtains average bounding-box localization rates over 99.8% for different vehicle parts on a dataset of 120 vehicles belonging to sedan, SUV and wagon categories. Moreover, it allows, with the addition of the active contour models, for a more complete and accurate description of vehicle parts contours than other state-of-the-art solutions. This research work is contributing to the implementation of an automated industrial system for vehicle inspection.

Image retrieval based on query by saliency content

In this research study we propose the query by saliency content retrieval (QSCR) image retrieval method which is based on human visual attention models. The proposed methodology represents a bottom-up approach considering the human attention, both locally at the level of image regions as well as globally, for the entire image. In the proposed retrieval system, each image is segmented and a set of characteristic features is calculated for each region. A weight is assigned to each region according to its perceived saliency. The proposed methodology considers not only the salient regions but also their context in the image. Localized image similarity is evaluated using an optimization method called the Earth Mover's Distance (EMD). EMD calculates inter-region distances between images and in our algorithm it considers the local saliency as well. A global image saliency, evaluated in terms of spatially-invariant salient edges, is used as an additional term in the optimization search besides the EMD distance. Experimental results are provided for three different image databases containing a wide variety of semantic categories.

A neuromorphic system for visual object recognition

Abstract Autonomous object recognition in images and videos is a topic of emerging importance in commercial and defense applications. Recent advances in the field of visual neuroscience are increasingly being leveraged to develop biologically-plausible models and algorithms for visual cognition with the ultimate goal of improving object recognition accuracy, speed, and robustness. In this paper, we describe our feasibility attempt at building one such end-to-end neuromorphic visual object recognition architecture and system that is inspired by neuroscience theories of human visual attention and recognition. Our system, called the NVRS (Neuromorphic Visual Recognition System), processes images and videos to first detect and localize objects and then classify them into one of several pre-defined object classes. The NVRS uses modularized algorithms inspired by the Form-and-Color-and-Depth (FACADE) and Laminar-ART (LAMINART) models of human visual attention, search, and object recognition. The NVRS is primarily composed of two encapsulated modules: attention and object recognition. The system was evaluated on simulated scenes, Caltech-101 and COIL-100 datasets, and aerial imagery, with results that demonstrate the efficacy of the NVRS. Lessons learnt from the NVRS can be useful in building practical autonomous visual systems for real-world applications and towards addressing longer-term BICA challenges.

Towards Understanding and Improving Geoscientific Data Interpretation

Geoscientific data interpretation is a highly subjective and complex task as human intuition and biases play a significant role. Based on these interpretations, however, mining and petroleum industries make decisions with paramount of financial implications. As a first step towards understanding and improving the interpretation process, we carried out two experiments to monitor the human-data interactions during the process of identifying â??targetsâ?? (porphyry-style intrusive systems) within the aeromagnetic imagery. This is achieved by capturing the eye gaze position using an eye tracker system and the brain responses using electroencephalography (EEG). The first experiment was intended to analyse the target spotting performance and the data observation patterns. For this experiment participants performed exercises, where the same magnetic image was presented in different orientations. Some key findings include: inconsistencies in target spotting performance within and between the interpreters; an improvement performance when the data were viewed in multiple orientations; and a strong correlation between the target spotting performance and efficient (systematic) data observation pattern. There was no correlation between success in identifying targets and the participantsâ?? perception of their expertise. The second experiment was designed to identify the characteristics of the targets that are easier to detect using EEG. For this experiment images with targets and without targets were presented in a rapid visual display. The analysis on the image characteristics based on the human visual attention model show a strong correlation between target spotting difficulty and dispersion of the visual attention.

Predicting Visual Discomfort of Stereoscopic Images Using Human Attention Model

We introduce a new objective assessment method for visual discomfort of stereoscopic images that makes effective use of the human visual attention model. The proposed method takes into account visual importance regions that play an important role in determining the overall degree of visual discomfort of a stereoscopic image. After obtaining a saliency-based visual importance map for an image, perceptually significant disparity features are extracted to predict the overall degree of visual discomfort. Experimental results show that the proposed method can achieve significantly higher prediction accuracy than the state-of-the-art methods.

Computational Models of Human Visual Attention and Their Implementations: A Survey

We humans are easily able to instantaneously detect the regions in a visual scene that are most likely to contain something of interest. Exploiting this pre-selection mechanism called visual attention for image and video processing systems would make them more sophisticated and therefore more useful. This paper briefly describes various computational models of human visual attention and their development, as well as related psychophysical findings. In particular, our objective is to carefully distinguish several types of studies related to human visual attention and saliency as a measure of attentiveness, and to provide a taxonomy from several viewpoints such as the main objective, the use of additional cues and mathematical principles. This survey finally discusses possible future directions for research into human visual attention and saliency computation.

Visual comfort assessment metric based on salient object motion information in stereoscopic video

Objective assessment of visual comfort for stereoscopic video is of great importance for stereoscopic image safety issue. We propose a novel visual comfort assessment metric framework that systematically exploits human visual attention models. In a stereoscopic video shot, perceptually significant regions where human subjects pay more attention are likely to play an essential role in determining the overall level of visual comfort. As a specific example of this concept, we develop a visual comfort metric that quantifies the level of visual discomfort caused by fast salient object motion. The performance of the proposed visual comfort metric has been evaluated using natural stereoscopic videos. The experimental results show that the proposed visual comfort metric significantly improves the correlations with subjective judgment.

Feature Description and Image Retrieval Based on Visual Attention Model

B ased on the bottom-up human visual attention model, a procedure for feature description and image retrieval is proposed. In the procedure, an integrated saliency map for a given image , which reflects the significance of regions in the image, is generated first, then two region segmentation methods, the image segmentation using the maximum entropy of the one-dimensional histogram and the region growing based on the focus of attention, are defined and used to separate the image into different regions. Here, the focus of attention is obtained from the saliency map and the secondary or uncorrelated regions can be filtered then . Hereafter, combining the saliency map and the segmented image, the first three most salient regions, defined as the salient region set, are obtained in term s of the average saliency of the regions. Radius entropy and angle entropy are also defined and performed to describe the global feature s and a shape feature for saliency regions is used to describe the local feature. Finally, the distance s between the features are calculated to measure the similarity between images. Experimental results show ed that our method can reflect the semantic feature of a given image to a certain extent, and the method is beneficial for content-based image retrieval.

Bridging the gap between physiology and behavior: Evidence from the sSoTS model of human visual attention.

We present the case for a role of biologically plausible neural network modeling in bridging the gap between physiology and behavior. We argue that spiking-level networks can allow "vertical" translation between physiological properties of neural systems and emergent "whole-system" performance-enabling psychological results to be simulated from implemented networks and also inferences to be made from simulations concerning processing at a neural level. These models also emphasize particular factors (e.g., the dynamics of performance in relation to real-time neuronal processing) that are not highlighted in other approaches and that can be tested empirically. We illustrate our argument from neural-level models that select stimuli by biased competition. We show that a model with biased competition dynamics can simulate data ranging from physiological studies of single-cell activity (Study 1) to whole-system behavior in human visual search (Study 2), while also capturing effects at an intermediate level, including performance breakdown after neural lesion (Study 3) and data from brain imaging (Study 4). We also show that, at each level of analysis, novel predictions can be derived from the biologically plausible parameters adopted, which we proceed to test (Study 5). We argue that, at least for studying the dynamics of visual attention, the approach productively links single-cell to psychological data.

A Perception-based Color Correction Method for Multi-view Images

Three-dimensional (3D) video technologies are becoming increasingly popular, as it can provide users with high quality and immersive experiences. However, color inconsistency between the camera views is an urgent problem to be solved in multi-view imaging. In this paper, a perception-based color correction method for multi-view images is proposed. In the proposed method, human visual sensitivity (VS) and visual attention (VA) models are incorporated into the correction process. Firstly, the VS property is used to reduce the computational complexity by removing these visual insensitive regions. Secondly, the VA property is used to improve the perceptual quality of local VA regions by performing VA-dependent color correction. Experimental results show that compared with other color correction methods, the proposed method can greatly promote the perceptual quality of local VA regions greatly and reduce the computational complexity, and obtain higher coding performance.

Influence of Low-Level Stimulus Features, Task Dependent Factors, and Spatial Biases on Overt Visual Attention

Visual attention is thought to be driven by the interplay between low-level visual features and task dependent information content of local image regions, as well as by spatial viewing biases. Though dependent on experimental paradigms and model assumptions, this idea has given rise to varying claims that either bottom-up or top-down mechanisms dominate visual attention. To contribute toward a resolution of this discussion, here we quantify the influence of these factors and their relative importance in a set of classification tasks. Our stimuli consist of individual image patches (bubbles). For each bubble we derive three measures: a measure of salience based on low-level stimulus features, a measure of salience based on the task dependent information content derived from our subjects' classification responses and a measure of salience based on spatial viewing biases. Furthermore, we measure the empirical salience of each bubble based on our subjects' measured eye gazes thus characterizing the overt visual attention each bubble receives. A multivariate linear model relates the three salience measures to overt visual attention. It reveals that all three salience measures contribute significantly. The effect of spatial viewing biases is highest and rather constant in different tasks. The contribution of task dependent information is a close runner-up. Specifically, in a standardized task of judging facial expressions it scores highly. The contribution of low-level features is, on average, somewhat lower. However, in a prototypical search task, without an available template, it makes a strong contribution on par with the two other measures. Finally, the contributions of the three factors are only slightly redundant, and the semi-partial correlation coefficients are only slightly lower than the coefficients for full correlations. These data provide evidence that all three measures make significant and independent contributions and that none can be neglected in a model of human overt visual attention.

Free viewing of dynamic stimuli by humans and monkeys

Due to extensive homologies, monkeys provide a sophisticated animal model of human visual attention. However, for electrophysiological recording in behaving animals simplified stimuli and controlled eye position are traditionally used. To validate monkeys as a model for human attention during realistic free viewing, we contrasted human (n = 5) and monkey (n = 5) gaze behavior using 115 natural and artificial video clips. Monkeys exhibited broader ranges of saccadic endpoints and amplitudes and showed differences in fixation and intersaccadic intervals. We compared tendencies of both species to gaze toward scene elements with similar low-level visual attributes using two computational models--luminance contrast and saliency. Saliency was more predictive of both human and monkey gaze, predicting human saccades better than monkey saccades overall. Quantifying interobserver gaze consistency revealed that while humans were highly consistent, monkeys were more heterogeneous and were best predicted by the saliency model. To address these discrepancies, we further analyzed high-interest gaze targets--those locations simultaneously chosen by at least two monkeys. These were on average very similar to human gaze targets, both in terms of specific locations and saliency values. Although substantial quantitative differences were revealed, strong similarities existed between both species, especially when focusing analysis onto high-interest targets.

The Image Retrieval and Relevance Feedback Methods Based on Region

Traditional image retrieval method based on global features can only extract low-level features, which are far from the semantics that human expects. So there is a huge gap between low-level features and high-level semantics. In order to overcome this gap, two approaches have been widely used: high-level semantic image representation and relevance feedback. The paper is based on region representation that comes closely to the semantics. Region-based image retrieval (RBIR) can effectively exclude the affection of backgrounds. The main work in the paper is summarized as follows: Firstly in order to solve the problems of image segment and similarity measure, an algorithm about detecting visual attended region with respect to image segment is proposed in the paper. This algorithm combined human visual attention model can effective detect the meaningful regions. And the paper proposes a similarity measure algorithm named Modified Integrated Region Matching (MIRM), which is more robust for over segmented images. Secondly this paper focuses on the applications of relevance feedback in RBIR. By studying three cases that may occur in relevance feedback respectively, this paper introduces some relevance feedback algorithms. Those are image coding and cluster based relevance feedback algorithm, Mapping Convergence (MC) based relevance feedback algorithm, Adaptive Boosting (AdaBoost) Support Vector Machines (SVM) based relevance feedback algorithm, Asymmetric Bagging SVM based relevance feedback algorithm and region representation based SVM relevance feedback algorithm. A large number of experimental results demonstrate that all the algorithms, both region based image representation and relevance feedback, proposed in this paper can improve the performance and efficiency of image retrieval.

Models of human visual attention should consider trial-by-trial variability in preparatory neural signals

Models of human visual attention should consider trial-by-trial variability in preparatory neural signals