Computational Models Of Visual Attention Research Articles

Computational Model of Stereoscopic 3D Visual Saliency

Many computational models of visual attention performing well in predicting salient areas of 2D images have been proposed in the literature. The emerging applications of stereoscopic 3D display bring an additional depth of information affecting the human viewing behavior, and require extensions of the efforts made in 2D visual modeling. In this paper, we propose a new computational model of visual attention for stereoscopic 3D still images. Apart from detecting salient areas based on 2D visual features, the proposed model takes depth as an additional visual dimension. The measure of depth saliency is derived from the eye movement data obtained from an eye-tracking experiment using synthetic stimuli. Two different ways of integrating depth information in the modeling of 3D visual attention are then proposed and examined. For the performance evaluation of 3D visual attention models, we have created an eye-tracking database, which contains stereoscopic images of natural content and is publicly available, along with this paper. The proposed model gives a good performance, compared to that of state-of-the-art 2D models on 2D images. The results also suggest that a better performance is obtained when depth information is taken into account through the creation of a depth saliency map, rather than when it is integrated by a weighting method.

A Visual-Attention Model Using Earth Mover's Distance-Based Saliency Measurement and Nonlinear Feature Combination

This paper introduces a new computational visual-attention model for static and dynamic saliency maps. First, we use the Earth Mover's Distance (EMD) to measure the center-surround difference in the receptive field, instead of using the Difference-of-Gaussian filter that is widely used in many previous visual-attention models. Second, we propose to take two steps of biologically inspired nonlinear operations for combining different features: combining subsets of basic features into a set of super features using the Lm-norm and then combining the super features using the Winner-Take-All mechanism. Third, we extend the proposed model to construct dynamic saliency maps from videos by using EMD for computing the center-surround difference in the spatiotemporal receptive field. We evaluate the performance of the proposed model on both static image data and video data. Comparison results show that the proposed model outperforms several existing models under a unified evaluation setting.

RETRACTED ARTICLE: From the human visual system to the computational models of visual attention: a survey

This article has been retracted by the authors. The article included text and ideas taken by the first author, without acknowledgement, from the following published article: “State-ofthe-art in visual attention modeling”, Ali Borji, Laurent Itti, IEEE Transactions on Pattern Analysis and Machine Intelligence 35(1) (2013) 185–207, published online 05/04/12. Most notably: • In Sect. 3.1 (Biological plausible methods) the following paragraphs or sentences largely derive from the Borji and Itti article: “Rosenholtz (1999), Rosenholtz et al. (2004) designed a model ...”; “In Gu et al. (2005), a saliency map ... ”; “Le Meur et al. (2006) proposed ...”; “Kootstra et al. (2008) developed ...”; “Marat et al. (2009) proposed ...”; “Chikkerur et al. (2010) proposed ...”; and “Murray et al. (2011) introduced ...”. • In Sect. 3.2 (Computational methods) the following paragraphs or sentences largely derive from the Borji and Itti article: “Salah et al. (2002) proposed ...”; “Ramstrom and Christensen (2002) introduced ...”; “In Rao et al. (2002) and Rao (2005), they proposed ...”; “Jodogne andPiater (2007) introduced ...”; “Boccignone (2008) presented ...”; “Rosin (2009) proposed ...”; “Mahadevan and Vasconcelos (2010) presented ...”; and “Wang et al. (2011) introduced ...”. • In Sect. 3.3 (Hybrid methods) the following paragraphs or sentences largely derive from theBorji and Itti article: “Lee andYu (1999) proposed ...”; “Peters et al. (2005), Peters and Itti (2007a,b, 2008) trained ...”; “Weights between two nodes ...”; “The model consists of a nonlinear ...”; “Zhang et al. (2007, 2008) proposed ...”; “Pang et al. (2008) presented ...”; “Zhang et al. (2009) extended ...”; and “Li et al. (2010a) presented ...”. • Section 6 (Discussion) largely derives from, or summarizes ideas presented in, Sects. 2.1, 2.2, 2.4, 2.6 and 3.1–3.8 of the Borji and Itti article. The first author apologizes for his action.

Computational Models of Human Visual Attention and Their Implementations: A Survey

We humans are easily able to instantaneously detect the regions in a visual scene that are most likely to contain something of interest. Exploiting this pre-selection mechanism called visual attention for image and video processing systems would make them more sophisticated and therefore more useful. This paper briefly describes various computational models of human visual attention and their development, as well as related psychophysical findings. In particular, our objective is to carefully distinguish several types of studies related to human visual attention and saliency as a measure of attentiveness, and to provide a taxonomy from several viewpoints such as the main objective, the use of additional cues and mathematical principles. This survey finally discusses possible future directions for research into human visual attention and saliency computation.

Salient Region Detection and Analysis Based on the Weighted Band-Pass Features

Researches on visual attention mechanism have revealed that the human visual system (HVS) is sensitive to the higher frequency components where they are distinctive from their surroundings by popping out. These attentive components of the scene can be in any form such as edge to texture differences based on the focus of attention of HVS. There are several visual attention computational models that can yield saliency values of attentive regions on the image. Some of these models take advantage of band-pass filter regions on spatial domain by computing center-surround differences with difference of low pass filters. They use either down-sampling that may cause loss of information or constant scale of the filters that may not contain all the necessary saliency information from the image. Therefore, we proposed an efficient and simple saliency detection model with full resolution and high perceptual quality, which outputs several band-pass regions by utilizing Fourier transform to obtain attentive regions edges to textures from the color image. All these detected important information with different bandwidth, then, were fused in a weighted manner by giving more priority to the texture compared to edge based salient regions. Experimental analysis was made for different color spaces and the model was compared with some relevant state of the art algorithms. As a result, the proposed saliency detection model has promising results based on the area under curve (AUC) performance evaluation metric.

High-throughput classification of clinical populations from natural viewing eye movements

Many high-prevalence neurological disorders involve dysfunctions of oculomotor control and attention, including attention deficit hyperactivity disorder (ADHD), fetal alcohol spectrum disorder (FASD), and Parkinson's disease (PD). Previous studies have examined these deficits with clinical neurological evaluation, structured behavioral tasks, and neuroimaging. Yet, time and monetary costs prevent deploying these evaluations to large at-risk populations, which is critically important for earlier detection and better treatment. We devised a high-throughput, low-cost method where participants simply watched television while we recorded their eye movements. We combined eye-tracking data from patients and controls with a computational model of visual attention to extract 224 quantitative features. Using machine learning in a workflow inspired by microarray analysis, we identified critical features that differentiate patients from control subjects. With eye movement traces recorded from only 15 min of videos, we classified PD versus age-matched controls with 89.6 % accuracy (chance 63.2 %), and ADHD versus FASD versus control children with 77.3 % accuracy (chance 40.4 %). Our technique provides new quantitative insights into which aspects of attention and gaze control are affected by specific disorders. There is considerable promise in using this approach as a potential screening tool that is easily deployed, low-cost, and high-throughput for clinical disorders, especially in young children and elderly populations who may be less compliant to traditional evaluation tests.

Diagnostic Features of Emotional Expressions Are Processed Preferentially

Diagnostic features of emotional expressions are differentially distributed across the face. The current study examined whether these diagnostic features are preferentially attended to even when they are irrelevant for the task at hand or when faces appear at different locations in the visual field. To this aim, fearful, happy and neutral faces were presented to healthy individuals in two experiments while measuring eye movements. In Experiment 1, participants had to accomplish an emotion classification, a gender discrimination or a passive viewing task. To differentiate fast, potentially reflexive, eye movements from a more elaborate scanning of faces, stimuli were either presented for 150 or 2000 ms. In Experiment 2, similar faces were presented at different spatial positions to rule out the possibility that eye movements only reflect a general bias for certain visual field locations. In both experiments, participants fixated the eye region much longer than any other region in the face. Furthermore, the eye region was attended to more pronouncedly when fearful or neutral faces were shown whereas more attention was directed toward the mouth of happy facial expressions. Since these results were similar across the other experimental manipulations, they indicate that diagnostic features of emotional expressions are preferentially processed irrespective of task demands and spatial locations. Saliency analyses revealed that a computational model of bottom-up visual attention could not explain these results. Furthermore, as these gaze preferences were evident very early after stimulus onset and occurred even when saccades did not allow for extracting further information from these stimuli, they may reflect a preattentive mechanism that automatically detects relevant facial features in the visual field and facilitates the orientation of attention towards them. This mechanism might crucially depend on amygdala functioning and it is potentially impaired in a number of clinical conditions such as autism or social anxiety disorders.

Methods for comparing scanpaths and saliency maps: strengths and weaknesses

In this article, we are interested in the computational modeling of visual attention. We report methods commonly used to assess the performance of these kinds of models. We survey the strengths and weaknesses of common assessment methods based on diachronic eye-tracking data. We then illustrate the use of some methods to benchmark computational models of visual attention.

A computational model of vision attention for inspection of surface quality in production line

There is need to detect regions of small defects in a large background, when product surface quality in line is inspected by machine vision systems. A computational model of visual attention was developed for solving the problem, inspired by the behavior and the neuronal architecture of human visual attention. Firstly, the global feature was extracted from input image by law’s rules, then local features were extracted and evaluated with an improved saliency map model of Itti. The local features were fused into a single topographical saliency map by a multi-feature fusion operator differenced from Itti model, in which the better feature has the higher weighting coefficient and more contribution to fusion of the feature’s images. Finally, the regions were “popped out” in the map. Experimental results show that the model can locate regions of interest and exclude the most background regions.

Automatic generation of accentuated pencil drawing with saliency map and LIC

An artist usually does not draw all the areas in a picture homogeneously but tries to make the work more expressive by emphasizing what is important while eliminating irrelevant details. Creating expressive painterly images with such accentuation effect remains to be a challenge because of the subjectivity of information selection. This paper presents a novel technique for automatically converting an input image into a pencil drawing with such emphasis and elimination effect. The proposed technique utilizes saliency map, a computational model for visual attention, to predict the focus of attention in the input image. A new level of detail controlling algorithm using multi-resolution pyramid is also developed for locally adapting the rendering parameters, such as the density, orientation and width of pencil strokes, to the degree of attention defined by saliency map. Experimental results show that the images generated with the proposed method present the visual effect similar to that of the real pencil drawing and can successfully direct the viewer’s attention toward the focus.

When viewing natural scenes, do abnormal colors impact on spatial or temporal parameters of eye movements?

Since Treisman's theory, it has been generally accepted that color is an elementary feature that guides eye movements when looking at natural scenes. Hence, most computational models of visual attention predict eye movements using color as an important visual feature. In this paper, using experimental data, we show that color does not affect where observers look when viewing natural scene images. Neither colors nor abnormal colors modify observers' fixation locations when compared to the same scenes in grayscale. In the same way, we did not find any significant difference between the scanpaths under grayscale, color, or abnormal color viewing conditions. However, we observed a decrease in fixation duration for color and abnormal color, and this was particularly true at the beginning of scene exploration. Finally, we found that abnormal color modifies saccade amplitude distribution.

Visual attention modeling based on short-term environmental adaption

Visual attention modeling is crucial for interpreting the structure and functionality of human vision system. A typical computational model of visual attention includes two basic elements: visual representation and saliency measurement. Most existing models left two phases unmodifiable without explicit adaption to the statistics of their corresponding visual environment. Inspired by neural adaption of biological neural systems, we proposed a novel principle for modeling visual attention mechanism named short-term environmental adaption. Given the statistics of a specified short-term visual environment, the proposed model adaptively extract sparse features and treats saliency as the features’ conditional self-information, which is more accurate in saliency measurement and more sparse with respect to visual signal representation.We have demonstrated our superior effectiveness and robustness over state-of-the-arts by carrying out dense experiments on human eye fixation benchmarks as well as psychological patterns.

On performance of image quality metrics enhanced with visual attention computational models

The benefits of incorporating saliency maps obtained with visual attention computational models into three image quality metrics are investigated. In particular, comparison is made of the performance of simple quality metrics with quality metrics that incorporate saliency maps obtained using three popular visual attention computational models. Results show that the performance of simple quality metrics can be improved by adding visual attention information. Nevertheless, gains in performance depend on the precision of the visual attention model, the type of distortion, and the characteristics of the quality metric.

Visual Attention Model with Adaptive Weighting of Conspicuity Maps for Building Detection in Satellite Images

Abstract The lack of automation and the limited performance of current image processing techniques pose critical challenges to the efficient and timely use of the large amount of data made available by aerial and space based assets. The imitation of fast adaptation and inference capability of human visual system appears to be a promising research direction for the development of computational algorithms able to deal with large variations in image content, characteristics and scale as those encountered in satellite imaging. The paper explores the potential use of an improved computational model of visual attention for the complex task of building identification in satellite images. It contributes to extend the envelope of application areas of such models and also to expand their current use from single object to multiple object detection. A set of original weighting schemes based on the contribution of different features to the identification of building and non-building areas is first proposed and evaluated against existing solutions in the literature. A novel adaptive algorithm then chooses the best weighting scheme based on a similarity error to ensure the best performance of the attention model in a given context. Finally, a neural network is trained to predict the set of weights provided by the best weighting scheme for the context of the image in which buildings are to be detected. The solution provides encouraging results on a set of 50 satellite images.

Saliency based on cortex-like mechanisms

A visual attention system should preferentially locate the most informative spots in complex environments. Feature-integration theory of attention plays an important role in bottom-up computational model for visual attention. This point extremely decreases the detection accuracy and also impacts the performance of the automatic visual attention model. To improve the accuracy of saliency detection in human visual attention, we propose a model based on cortex-like mechanisms. Saliency Criteria are obtained from two pathways: Shannon's entropy and sparse representation. And our model not only substantiates the bottom-up model proposed by Itti and HMAX model by Paggio, but also enriches the theory of saliency detection. We demonstrate that the proposed model achieves superior accuracy in comparison to the classical approach in static saliency map generation on real data of natural scenes and psychology stimuli patterns.

Improved saliency toolbox/Itti model for region of interest extraction

The toolbox (STB)/Itti model is an outstanding computational selective visual attention model. In this paper, we propose an improved STB/Itti model to overcome the drawback of STB/Itti-its output saliency map is not large enough for region of interest (ROI) extraction. First, we employ a simplified pulse coupled neural network (PCNN) with a special input image, and more importantly, the PCNN does not require iterations. Subsequently, the PCNN takes the place of the winner-take-all network in STB/Itti. Experimental results show that the improved STB/Itti model works well for ROI extraction, with the mean area under the curve value of 0.8306 and robustness against noise and geometric attacks. The proposed model can greatly enhance the performances of both STB/Itti and PCNN model in image processing.

A Computational Model for Object-based Visual Attention

Computational model for object-based visual attention is a hot research field. A computational model of object-based visual attention is proposed in this paper. This model mainly includes three key technologies which are perceptual objects extraction, attention value calculation of perceptual objects and focus of attention shift. Firstly spatial visual saliency of each pixel is calculated and a saliency map is generated. The most salient homogeneity regions are extracted based on their spatial visual saliency. The extracted salient regions can be considered as perceptual objects. Attention value of each perceptual object is computed according to its area, location and saliency. Then a hierarchical attention shift method is used in the proposed model. The proposed model has been tested on many natural images. Experiment results show that the proposed model is valid and the attention results are consistent with human visual system.

Vassilis Cutsuridis, Amir Hussain and John G. Taylor (eds): Perception–action cycle. Models, architectures, and hardware

Vassilis Cutsuridis, Amir Hussain and John G. Taylor (eds): Perception–action cycle. Models, architectures, and hardware

Visual Attention in Objective Image Quality Assessment: Based on Eye-Tracking Data

Since the human visual system (HVS) is the ultimate assessor of image quality, current research on the design of objective image quality metrics tends to include an important feature of the HVS, namely, visual attention. Different metrics for image quality prediction have been extended with a computational model of visual attention, but the resulting gain in reliability of the metrics so far was variable. To better understand the basic added value of including visual attention in the design of objective metrics, we used measured data of visual attention. To this end, we performed two eye-tracking experiments: one with a free-looking task and one with a quality assessment task. In the first experiment, 20 observers looked freely to 29 unimpaired original images, yielding us so-called natural scene saliency (NSS). In the second experiment, 20 different observers assessed the quality of distorted versions of the original images. The resulting saliency maps showed some differences with the NSS, and therefore, we applied both types of saliency to four different objective metrics predicting the quality of JPEG compressed images. For both types of saliency the performance gain of the metrics improved, but to a larger extent when adding the NSS. As a consequence, we further integrated NSS in several state-of-the-art quality metrics, including three full-reference metrics and two no-reference metrics, and evaluated their prediction performance for a larger set of distortions. By doing so, we evaluated whether and to what extent the addition of NSS is beneficial to objective quality prediction in general terms. In addition, we address some practical issues in the design of an attention-based metric. The eye-tracking data are made available to the research community .

Bottom-up attention: pulsed PCA transform and pulsed cosine transform.

In this paper we propose a computational model of bottom-up visual attention based on a pulsed principal component analysis (PCA) transform, which simply exploits the signs of the PCA coefficients to generate spatial and motional saliency. We further extend the pulsed PCA transform to a pulsed cosine transform that is not only data-independent but also very fast in computation. The proposed model has the following biological plausibilities. First, the PCA projection vectors in the model can be obtained by using the Hebbian rule in neural networks. Second, the outputs of the pulsed PCA transform, which are inherently binary, simulate the neuronal pulses in the human brain. Third, like many Fourier transform-based approaches, our model also accomplishes the cortical center-surround suppression in frequency domain. Experimental results on psychophysical patterns and natural images show that the proposed model is more effective in saliency detection and predict human eye fixations better than the state-of-the-art attention models.