Saliency Computation Research Articles

Neural activities in V1 create the bottom-up saliency map of natural scenes.

A saliency map is the bottom-up contribution to the deployment of exogenous attention. It, as well as its underlying neural mechanism, is hard to identify because of the influence of top-down signals. A recent study showed that neural activities in V1 could create a bottom-up saliency map (Zhang et al. in Neuron 73(1):183-192, 2012). In this paper, we tested whether their conclusion can generalize to complex natural scenes. In order to avoid top-down influences, each image was presented with a low contrast for only 50ms and was followed by a high contrast mask, which rendered the whole image invisible to participants (confirmed by a forced-choice test). The Posner cueing paradigm was adopted to measure the spatial cueing effect (i.e., saliency) by an orientation discrimination task. A positive cueing effect was found, and the magnitude of the cueing effect was consistent with the saliency prediction of a computational saliency model. In a following fMRI experiment, we used the same masked natural scenes as stimuli and measured BOLD signals responding to the predicted salient region (relative to the background). We found that the BOLD signal in V1, but not in other cortical areas, could well predict the cueing effect. These results suggest that the bottom-up saliency map of natural scenes could be created in V1, providing further evidence for the V1 saliency theory (Li in Trends Cogn Sci 6(1):9-16, 2002).

A Computational Model for Object-Based Visual Saliency: Spreading Attention Along Gestalt Cues

The past few years have witnessed impressive progress on the research of salient object detection. Nevertheless , existing approaches still cannot perform satisfactorily in the case of complex scenes, particularly when the salient objects have non- uniform appearance or complicated shapes, and the background is complexly structured. One important reason for such limitations may be that these approaches commonly ignore the factor of perceptual grouping in saliency modeling. To address this issue, this paper presents a novel computational model for object -based visual saliency, which explicitly takes into consideration the connections between attention and perceptual grouping, and incorporates Gestalt grouping cues into saliency computation. Inspired by the sensory enhancement theory, we suggest a paradigm for object-based saliency modeling, that is, object-based saliency stems from spreading attention along Gestalt grouping cues. Computationally , three typical Gestalt cues, including proximity, similarity, and closure, are respectively extracted from the given image, which are then integrated by constructing a unified Gestalt graph. A new algorithm named personalized power iteration clustering is developed to effectively fulfill the spreading of attention information across the Gestalt graph. Intensive experiments have been carried out to demonstrate the superior performance of the proposed model in comparison to the state-of-the-art.

Saliency computation via whitened frequency band selection.

Many saliency computational models have been proposed to simulate bottom-up visual attention mechanism of human visual system. However, most of them only deal with certain kinds of images or aim at specific applications. In fact, human beings have the ability to correctly select attentive focuses of objects with arbitrary sizes within any scenes. This paper proposes a new bottom-up computational model from the perspective of frequency domain based on the biological discovery of non-Classical Receptive Field (nCRF) in the retina. A saliency map can be obtained according to the idea of Extended Classical Receptive Field. The model is composed of three major steps: firstly decompose the input image into several feature maps representing different frequency bands that cover the whole frequency domain by utilizing Gabor wavelet. Secondly, whiten the feature maps to highlight the embedded saliency information. Thirdly, select some optimal maps, simulating the response of receptive field especially nCRF, to generate the saliency map. Experimental results show that the proposed algorithm is able to work with stable effect and outstanding performance in a variety of situations as human beings do and is adaptive to both psychological patterns and natural images. Beyond that, biological plausibility of nCRF and Gabor wavelet transform make this approach reliable.

DISC: Deep Image Saliency Computing via Progressive Representation Learning.

Salient object detection increasingly receives attention as an important component or step in several pattern recognition and image processing tasks. Although a variety of powerful saliency models have been intensively proposed, they usually involve heavy feature (or model) engineering based on priors (or assumptions) about the properties of objects and backgrounds. Inspired by the effectiveness of recently developed feature learning, we provide a novel deep image saliency computing (DISC) framework for fine-grained image saliency computing. In particular, we model the image saliency from both the coarse-and fine-level observations, and utilize the deep convolutional neural network (CNN) to learn the saliency representation in a progressive manner. In particular, our saliency model is built upon two stacked CNNs. The first CNN generates a coarse-level saliency map by taking the overall image as the input, roughly identifying saliency regions in the global context. Furthermore, we integrate superpixel-based local context information in the first CNN to refine the coarse-level saliency map. Guided by the coarse saliency map, the second CNN focuses on the local context to produce fine-grained and accurate saliency map while preserving object details. For a testing image, the two CNNs collaboratively conduct the saliency computing in one shot. Our DISC framework is capable of uniformly highlighting the objects of interest from complex background while preserving well object details. Extensive experiments on several standard benchmarks suggest that DISC outperforms other state-of-the-art methods and it also generalizes well across data sets without additional training. The executable version of DISC is available online: http://vision.sysu.edu.cn/projects/DISC.

Night-Time Vehicle Detection Algorithm Based on Visual Saliency and Deep Learning

Night vision systems get more and more attention in the field of automotive active safety field. In this area, a number of researchers have proposed far-infrared sensor based night-time vehicle detection algorithm. However, existing algorithms have low performance in some indicators such as the detection rate and processing time. To solve this problem, we propose a far-infrared image vehicle detection algorithm based on visual saliency and deep learning. Firstly, most of the nonvehicle pixels will be removed with visual saliency computation. Then, vehicle candidate will be generated by using prior information such as camera parameters and vehicle size. Finally, classifier trained with deep belief networks will be applied to verify the candidates generated in last step. The proposed algorithm is tested in around 6000 images and achieves detection rate of 92.3% and processing time of 25 Hz which is better than existing methods.

A dense subgraph based algorithm for compact salient image region detection

We present an algorithm for graph based saliency computation that utilizes the underlying dense subgraphs in finding visually salient regions in an image. To compute the salient regions, the model first obtains a saliency map using random walks on a Markov chain. Next, k-dense subgraphs are detected to further enhance the salient regions in the image. Dense subgraphs convey more information about local graph structure than simple centrality measures. To generate the Markov chain, intensity and color features of an image in addition to region compactness is used. For evaluating the proposed model, we do extensive experiments on benchmark image data sets. The proposed method performs comparable to well-known algorithms in salient region detection.

Salient object detection via spectral matting

A number of pro-superpixel based saliency models have recently been proposed, which segment the image into small perceptually homogeneous regions before saliency computation. Such approaches ignore important object properties, resulting in inappropriate object annotations and considerably different saliency assignment to the various regions of an object. Although previous techniques employ multi-scale saliency maps in an attempt to rectify this problem, it becomes difficult to retain the characteristics of proto-objects after the first stage of processing. We introduce matting components based saliency to address the problems of inappropriate object annotations and inappropriate saliency assignment to object regions. The matting components account for proto-object properties by employing object aware spectral segmentation. To complement the matting component based saliency, we also employ the smallest eigenvectors of a matting Laplacian matrix. Color spatial distribution features are employed to capture global relationships at the pixel-level and assist the process of matting components based saliency computation. A novel joint optimization framework is introduced to fuse the features and learn important associated parameters. The contributions of the proposed approach are two-fold. The first contribution is the introduction of proto-objects aware spectral segmentation to obtain an accurate foreground saliency. The second contribution is the joint optimization of important parameters in conjunction with learning feature importance. In contrast to superpixel based approaches, the proposed model is able to completely annotate salient objects and assign similar saliency to various regions of the salient object. Moreover, the proposed approach shows robust and efficient performance across five challenging benchmark datasets when compared with 10 recently proposed state-of-the-art saliency detection models.

Small object detection in forward-looking infrared images with sea clutter using context-driven Bayesian saliency model

There are two common challenges for small object detection in forward-looking infrared (FLIR) images with sea clutter, namely, detection ambiguity and scale variance. This paper presents a context-driven Bayesian saliency model to deal with these two issues. By inspecting the camera geometry of the FLIR imaging under the background of sea and sky, we observed that there exists dependency relationship between the locations and scales at which objects may occur, and the context which is defined to be the location of horizon line. Based on this observation, we propose to incorporate contextual information into the basic bottom-up saliency computation, and a unified Bayesian model is developed to achieve this goal. The proposed model is generic and can be potentially applied to other circumstances where context is available for facilitating object detection. Experimental results have demonstrated the effectiveness of our method.

Visual Saliency Prediction and Evaluation across Different Perceptual Tasks.

Saliency maps produced by different algorithms are often evaluated by comparing output to fixated image locations appearing in human eye tracking data. There are challenges in evaluation based on fixation data due to bias in the data. Properties of eye movement patterns that are independent of image content may limit the validity of evaluation results, including spatial bias in fixation data. To address this problem, we present modeling and evaluation results for data derived from different perceptual tasks related to the concept of saliency. We also present a novel approach to benchmarking to deal with some of the challenges posed by spatial bias. The results presented establish the value of alternatives to fixation data to drive improvement and development of models. We also demonstrate an approach to approximate the output of alternative perceptual tasks based on computational saliency and/or eye gaze data. As a whole, this work presents novel benchmarking results and methods, establishes a new performance baseline for perceptual tasks that provide an alternative window into visual saliency, and demonstrates the capacity for saliency to serve in approximating human behaviour for one visual task given data from another.

The Application of Visual Saliency Models in Objective Image Quality Assessment: A Statistical Evaluation

Advances in image quality assessment have shown the potential added value of including visual attention aspects in its objective assessment. Numerous models of visual saliency are implemented and integrated in different image quality metrics (IQMs), but the gain in reliability of the resulting IQMs varies to a large extent. The causes and the trends of this variation would be highly beneficial for further improvement of IQMs, but are not fully understood. In this paper, an exhaustive statistical evaluation is conducted to justify the added value of computational saliency in objective image quality assessment, using 20 state-of-the-art saliency models and 12 best-known IQMs. Quantitative results show that the difference in predicting human fixations between saliency models is sufficient to yield a significant difference in performance gain when adding these saliency models to IQMs. However, surprisingly, the extent to which an IQM can profit from adding a saliency model does not appear to have direct relevance to how well this saliency model can predict human fixations. Our statistical analysis provides useful guidance for applying saliency models in IQMs, in terms of the effect of saliency model dependence, IQM dependence, and image distortion dependence. The testbed and software are made publicly available to the research community.

Ray Saliency: Bottom-Up Visual Saliency for a Rotating and Zooming Camera

We extend the classical notion of computational visual saliency to multi-image data collected using a stationary pan-tilt-zoom (PTZ) camera by introducing the concept of consistency: the requirement that the set of generated saliency maps should each assign the same saliency value to unique regions of the environment that appear in more than one image. We show that processing each image independently will often fail to provide a consistent measure of saliency, and that using an image mosaic to quantify saliency suffers from several drawbacks. We then propose ray saliency and an immediate extension, approximate ray saliency: a mosaic-free method for calculating a consistent measure of bottom-up saliency. Experimental results demonstrating the effectiveness of the proposed approach are presented.

Visible watermarking based on importance and just noticeable distortion of image regions

Visible watermarking is the process of embedding data (watermark) into a multimedia object (video/image) such that the embedded watermark is perceptible to a human observer. Many times, visible watermarks occlude important portion of multimedia objects. This paper introduces a visible watermarking algorithm to embed a binary logo watermark at N non-overlapping positions in an image such that important portions of the image are not occluded. The important portions are found through visual saliency computation or available human eye fixation density maps. In the proposed visible watermarking, just noticeable distortion is used to adaptively filter the watermark embedding energy based on the image content. A mathematical model in terms of information-content-weighted-structural-similarity-index and visual importance is proposed to find optimal watermark embedding strength. We tested the algorithm on several color images of different sizes and on several binary watermarks of different sizes and found the results to be very promising as per the requirements in visible watermarking. When compared to the state-of-the-art, we also found that the proposed technique does better in not hiding the details of any test image.

A bio-inspired center-surround model for salience computation in images

A center-surround model inspired by photoreceptor interactions and visual receptive field organization is presented in this paper for salience computation that predicts human eye fixation locations in images. The essence of photoreceptor interactions is implemented considering different nonlinear combinations of responses to stimuli given by values at nearby image pixels. These combinations are then fed to difference of Gaussian filtered outputs operation and Gabor filter based processes simulating visual receptive field organization. The proposed center-surround model is used in Itti et al.’s bio-inspired framework to perform salience computation. Analysis is carried out to present the information-theoretic aspect of the nonlinear combinations. Significance of the proposed center-surround model is shown both qualitatively and quantitatively by comparing its use in salience computation with the use of existing models considering different psychological patterns, and synthetic and real-life images. Quantitative and qualitative performance of salience computation using the novel center-surround model for three well-known datasets of images are also compared to that of relevant existing salience computation approaches to demonstrate the effectiveness of the proposed approach in generating salience maps closer to human eye fixation density maps.

A depth perception and visual comfort guided computational model for stereoscopic 3D visual saliency

With the emerging development of three-dimensional (3D) related technologies, 3D visual saliency modeling is becoming particularly important and challenging. This paper presents a new depth perception and visual comfort guided saliency computational model for stereoscopic 3D images. The prominent advantage of the proposed model is that we incorporate the influence of depth perception and visual comfort on 3D visual saliency computation. The proposed saliency model is composed of three components: 2D image saliency, depth saliency and visual comfort based saliency. In the model, color saliency, texture saliency and spatial compactness are computed respectively and fused to derive 2D image saliency. Global disparity contrast is considered to compute depth saliency. Particularly, we train a visual comfort prediction function to distinguish stereoscopic image pair as high comfortable stereo viewing (HCSV) or low comfortable stereo viewing (LCSV), and devise different computational rules to generate a visual comfort based saliency map. The final 3D saliency map is obtained by using a linear combination and enhanced by a “saliency-center bias” model. Experimental results show that the proposed 3D saliency model outperforms the state-of-the-art models on predicting human eye fixations and visual comfort assessment.

WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians

We introduce in this paper a novel polyp localization method for colonoscopy videos. Our method is based on a model of appearance for polyps which defines polyp boundaries in terms of valley information. We propose the integration of valley information in a robust way fostering complete, concave and continuous boundaries typically associated to polyps. This integration is done by using a window of radial sectors which accumulate valley information to create WM-DOVA (Window Median Depth of Valleys Accumulation) energy maps related with the likelihood of polyp presence. We perform a double validation of our maps, which include the introduction of two new databases, including the first, up to our knowledge, fully annotated database with clinical metadata associated. First we assess that the highest value corresponds with the location of the polyp in the image. Second, we show that WM-DOVA energy maps can be comparable with saliency maps obtained from physicians’ fixations obtained via an eye-tracker. Finally, we prove that our method outperforms state-of-the-art computational saliency results. Our method shows good performance, particularly for small polyps which are reported to be the main sources of polyp miss-rate, which indicates the potential applicability of our method in clinical practice.

Database Saliency for Fast Image Retrieval

The bag-of-visual-words (BoW) model is effective for representing images and videos in many computer vision problems, and achieves promising performance in image retrieval. Nevertheless, the level of retrieval efficiency in a large-scale database is not acceptable for practical usage. Considering that the relevant images in the database of a given query are more likely to be distinctive than ambiguous, this paper defines “database saliency” as the distinctiveness score calculated for every image to measure its overall “saliency” in the database. By taking advantage of database saliency, we propose a saliency- inspired fast image retrieval scheme, S-sim, which significantly improves efficiency while retains state-of-the-art accuracy in image retrieval . There are two stages in S-sim: the bottom-up saliency mechanism computes the database saliency value of each image by hierarchically decomposing a posterior probability into local patches and visual words, the concurrent information of visual words is then bottom-up propagated to estimate the distinctiveness, and the top-down saliency mechanism discriminatively expands the query via a very low-dimensional linear SVM trained on the top-ranked images after initial search, ranking images are then sorted on their distances to the decision boundary as well as the database saliency values. We comprehensively evaluate S-sim on common retrieval benchmarks, e.g., Oxford and Paris datasets. Thorough experiments suggest that, because of the offline database saliency computation and online low-dimensional SVM, our approach significantly speeds up online retrieval and outperforms the state-of-the-art BoW-based image retrieval schemes.

Salience computation in images based on perceptual distinctness

A salience computation approach based on perceptual distinctness is proposed in this paper that predicts human eye fixation locations in images. The approach uses a novel representation of pattern in a patch around a pixel such that discrimination between patterns is in compliance with distinctness between them in-terms of a few well-known perceptual cues. Image patches of different sizes (resolutions) are considered. The representations of the value at and pattern around a pixel in different features such as lightness, color and orientation are discriminated from all those in the rest of the image to compute salience at that pixel. The effectiveness of the proposed salience computation approach in generating salience maps closer to human eye fixation density maps is demonstrated by comparing it both qualitatively and quantitatively to other existing state-of-the-art salience computation approaches. Several synthetic images and psychological patterns, and real-life images from five well-known datasets are considered for this purpose.

Spatial-Aware Object-Level Saliency Prediction by Learning Graphlet Hierarchies

To fill the semantic gap between the predictive power of computational saliency models and human behavior, this paper proposes to predict where people look at using spatial-aware object-level cues. While object-level saliency has been recently suggested by psychophysics experiments and shown effective with a few computational models, the spatial relationship between the objects has not yet been explored in this context. We in this work for the first time explicitly model such spatial relationship, as well as leveraging semantic information of an image to enhance object-level saliency modeling. The core computational module is a graphlet-based (i.e., graphlets are moderate-sized connected subgraphs) deep architecture, which hierarchically learns a saliency map from raw image pixels to object-level graphlets (oGLs) and further to spatial-level graphlets (sGLs). Eye tracking data are also used to leverage human experience in saliency prediction. Experimental results demonstrate that the proposed oGLs and sGLs well capture object-level and spatial-level cues relating to saliency, and the resulting saliency model performs competitively compared with the state-of-the-art.

Inferring Visual Perceptual Object by Adaptive Fusion of Image Salient Features

Saliency computational model with active environment perception can be useful for many applications including image retrieval, object recognition, and image segmentation. Previous work on bottom-up saliency computation typically relies on hand-crafted low-level image features. However, the adaptation of saliency computational model towards different kinds of scenes remains a challenge. For a low-level image feature, it can contribute greatly on some images but may be detrimental for saliency computation on other images. In this work, a novel data driven approach is proposed to adaptively select proper features for different kinds of images. This method exploits low-level features containing the most distinguishable salient information per image. Then the image saliency can be calculated based on the adaptive weight selection scheme. A large number of experiments are conducted on the MSRA database to compare the performance of the proposed method with the state-of-the-art saliency computational models.

Attentional Scene-Exploration and Object Discovery in Image and RGB-D Data

In this paper, we summarize our project work of the last two years, where we addressed the tasks of visually exploring a scene with visual attention mechanisms based on saliency computation, and of locating unknown objects in the environment. The latter is also called object discovery and consists in finding candidate objects without previous knowledge about the objects themselves or the scene. We follow an approach motivated from human perception and combine saliency and segmentation to generate object candidates. We show results on 2D images as well as on 3D sequences obtained from an RGB-D camera.