Object-level Saliency Research Articles

Stage-wise Salient Object Detection in 360° Omnidirectional Image via Object-level Semantical Saliency Ranking.

The 2D image based salient object detection (SOD) has been extensively explored, while the 360° omnidirectional image based SOD has received less research attention and there exist three major bottlenecks that are limiting its performance. Firstly, the currently available training data is insufficient for the training of 360° SOD deep model. Secondly, the visual distortions in 360° omnidirectional images usually result in large feature gap between 360° images and 2D images; consequently, the widely used stage-wise training-a widely-used solution to alleviate the training data shortage problem, becomes infeasible when conducing SOD in 360° omnidirectional images. Thirdly, the existing 360° SOD approach has followed a multi-task methodology that performs salient object localization and segmentation-like saliency refinement at the same time, being faced with extremely large problem domain, making the training data shortage dilemma even worse. To tackle all these issues, this paper divides the 360° SOD into a multi-staqe task, the key rationale of which is to decompose the original complex problem domain into sequential easy sub problems that only demand for small-scale training data. Meanwhile, we learn how to rank the "object-level semantical saliency", aiming to locate salient viewpoints and objects accurately. Specifically, to alleviate the training data shortage problem, we have released a novel dataset named 360-SSOD, containing 1,105 360° omnidirectional images with manually annotated object-level saliency ground truth, whose semantical distribution is more balanced than that of the existing dataset. Also, we have compared the proposed method with 13 SOTA methods, and all quantitative results have demonstrated the performance superiority.

50 FPS Object-Level Saliency Detection via Maximally Stable Region.

The human visual system tends to consider saliency of an object as a whole. Some object-level saliency detection methods have been proposed by leveraging object proposals in bounding boxes, and regarding the entire bounding box as one candidate salient region. However, the bounding boxes can not provide exact object position and a lot of pixels in bounding boxes belong to the background. Consequently, background pixels in bounding box also show high saliency. Besides, acquiring object proposals needs high time cost. In order to compute object-level saliency, we consider region growing from some seed superpixels, to find one surrounding region which probably represents the whole object. The desired surrounding region has similar appearance inside and obvious difference with the outside, which is proposed as maximally stable region (MSR) in this paper. In addition, one effective seed superpixel selection strategy is presented to improve speed. MSR based saliency detection is more robust than pixel or superpixel level methods and object proposal based methods. The proposed method significantly outperforms the state-of-the-art unsupervised methods at 50 FPS. Compared with deep learning based methods, we show worse performance, but with about 1200-1600 times faster, which means better trade-off between performance and speed.

Inferring Salient Objects from Human Fixations.

Previous research in visual saliency has been focused on two major types of models namely fixation prediction and salient object detection. The relationship between the two, however, has been less explored. In this work, we propose to employ the former model type to identify salient objects. We build a novel Attentive Saliency Network (ASNet)1 1.Available at: https://github.com/wenguanwang/ASNet. that learns to detect salient objects from fixations. The fixation map, derived at the upper network layers, mimics human visual attention mechanisms and captures a high-level understanding of the scene from a global view. Salient object detection is then viewed as fine-grained object-level saliency segmentation and is progressively optimized with the guidance of the fixation map in a top-down manner. ASNet is based on a hierarchy of convLSTMs that offers an efficient recurrent mechanism to sequentially refine the saliency features over multiple steps. Several loss functions, derived from existing saliency evaluation metrics, are incorporated to further boost the performance. Extensive experiments on several challenging datasets show that our ASNet outperforms existing methods and is capable of generating accurate segmentation maps with the help of the computed fixation prior. Our work offers a deeper insight into the mechanisms of attention and narrows the gap between salient object detection and fixation prediction.

Unsupervised Salient Object Detection by Aggregating Multi-Level Cues

In this paper, we present a novel method to detect salient object based on multi-level cues. First, a proposal processing scheme is developed by various object-level saliency cues to generate an initial saliency map. For the sake of more accurate object boundaries, a two-stage optimization mechanism is then proposed upon superpixel-level. Finally, the superpixel-level saliency map is further improved to construct the final saliency map by applying superpixel-to-pixel mapping. Extensive experimental results demonstrate that the proposed algorithm performs favorably against the state-of-art saliency detection methods in terms of different evaluation metrics on several benchmark datasets.

Video co-segmentation based on directed graph

This paper proposes a novel video co-segmentation method, which aims to extract multi-class objects from a group of videos. A set of tracklets are first generated based on object proposals, and then a novel directed graph is constructed to connect object tracklets. The directed graph is transformed to an undirected graph, and the extraction of common object tracklets is solved by using maximum weighted clique. The obtained common object tracklets are used as seed regions to perform manifold ranking and to generate the object-level saliency maps. Based on common object tracklets and object-level saliency maps, GrabCut is exploited to get the refined co-segmentation results. Experimental results on a public video dataset show that the proposed video co-segmentation method consistently outperforms the state-of-the-art methods.

Salient Object Detection via Recursive Sparse Representation

Object-level saliency detection is an attractive research field which is useful for many content-based computer vision and remote-sensing tasks. This paper introduces an efficient unsupervised approach to salient object detection from the perspective of recursive sparse representation. The reconstruction error determined by foreground and background dictionaries other than common local and global contrasts is used as the saliency indication, by which the shortcomings of the object integrity can be effectively improved. The proposed method consists of the following four steps: (1) regional feature extraction; (2) background and foreground dictionaries extraction according to the initial saliency map and image boundary constraints; (3) sparse representation and saliency measurement; and (4) recursive processing with a current saliency map updating the initial saliency map in step 2 and repeating step 3. This paper also presents the experimental results of the proposed method compared with seven state-of-the-art saliency detection methods using three benchmark datasets, as well as some satellite and unmanned aerial vehicle remote-sensing images, which confirmed that the proposed method was more effective than current methods and could achieve more favorable performance in the detection of multiple objects as well as maintaining the integrity of the object area.

Object-level saliency: Fusing objectness estimation and saliency detection into a uniform framework

Most existing saliency detection algorithms concentrate on obtaining good results for images with single salient object, while it produces poor generalization power when tested on more realistic images. In this paper, we present a novel framework to detect saliency in object-level through fusing objectness estimation into the process of salient object detection. Different from most existing methods that evaluate saliency via aggregation of adjacent pixels or regions, our approach peels background regions step by step via evaluating each region’s saliency, objectness and background, until all the independent foreground objects are left. Instead of extracting from saliency map, the proposed method can obtain salient objects directly, and different salient scores can be assigned to different salient objects. Experimental results show that the proposed method is effective and achieves state-of-the-art performance in several benchmark datasets, especially on PASCAL_S and SED2 that offer salient objects in more complicated scenes.

Video Saliency Detection Using Object Proposals.

In this paper, we introduce a novel approach to identify salient object regions in videos via object proposals. The core idea is to solve the saliency detection problem by ranking and selecting the salient proposals based on object-level saliency cues. Object proposals offer a more complete and high-level representation, which naturally caters to the needs of salient object detection. As well as introducing this novel solution for video salient object detection, we reorganize various discriminative saliency cues and traditional saliency assumptions on object proposals. With object candidates, a proposal ranking and voting scheme, based on various object-level saliency cues, is designed to screen out nonsalient parts, select salient object regions, and to infer an initial saliency estimate. Then a saliency optimization process that considers temporal consistency and appearance differences between salient and nonsalient regions is used to refine the initial saliency estimates. Our experiments on public datasets (SegTrackV2, Freiburg-Berkeley Motion Segmentation Dataset, and Densely Annotated Video Segmentation) validate the effectiveness, and the proposed method produces significant improvements over state-of-the-art algorithms.

Spectral salient object detection

Many salient object detection methods first apply pre-segmentation on image to obtain over-segmented regions to facilitate subsequent saliency computation. However, these pre-segmentation methods often ignore the holistic issue of objects and could degrade object detection performance. This paper proposes a novel method, spectral salient object detection, that aims at maintaining objects holistically during pre-segmentation in order to provide more reliable feature extraction from a complete object region and to facilitate object-level saliency estimation. In the proposed method, a hierarchical spectral partition method based on the normalized graph cut (Ncut) is proposed for image segmentation phase in saliency detection, where a superpixel graph that captures the intrinsic color and edge information of an image is constructed and then hierarchically partitioned. In each hierarchy level, a region constituted by superpixels is evaluated by criteria based on figure-ground principles and statistical prior to obtain a regional saliency score. The coarse salient region is obtained by integrating multiple saliency maps from successive hierarchies. The final saliency map is derived by minimizing the graph-based semi-supervised learning energy function on the synthetic coarse saliency map. Despite the simple intuition of maintaining object holism, experimental results on 5 benchmark datasets including ASD, ECSSD, MSRA, PASCAL-S, DUT-OMRON demonstrate encouraging performance of the proposed method, along with the comparisons to 13 state-of-the-art methods. The proposed method is shown to be effective on emphasizing large/medium-sized salient objects uniformly due to the employment of Ncut. Besides, we conduct thorough analysis and evaluation on parameters and individual modules.

Salient object detection via boosting object-level distinctiveness and saliency refinement

Many salient object detection approaches share the common drawback that they cannot uniformly highlight heterogeneous regions of salient objects, and thus, parts of the salient objects are not discriminated from background regions in a saliency map. In this paper, we focus on this drawback and accordingly propose a novel algorithm that more uniformly highlights the entire salient object as compared to many approaches. Our method consists of two stages: boosting the object-level distinctiveness and saliency refinement. In the first stage, a coarse object-level saliency map is generated based on boosting the distinctiveness of the object proposals in the test images, using a set of object-level features and the Modest AdaBoost algorithm. In the second stage, several saliency refinement steps are executed to obtain a final saliency map in which the boundaries of salient objects are preserved. Quantitative and qualitative comparisons with state-of-the-art approaches demonstrate the superior performance of our approach.

Detecting Salient Objects via Color and Texture Compactness Hypotheses.

In recent years, the object-level saliency detection has attracted much research attention, due to its usefulness in many high-level tasks. Existing methods are mostly based on the contrast hypothesis, which regards the regions with high contrast in a certain context as salient objects. Although the contrast hypothesis is effective in many scenarios, it cannot handle some difficult cases. As a remedy to address the weakness of contrast hypothesis, we propose a novel compactness hypothesis, which assumes salient regions are more compact than background from the perspectives of both color layout and texture layout. Based on the compactness hypotheses, we implement an effective object-level saliency detection method. In the proposed method, we first construct a weak saliency map based on the compact hypotheses, then collect samples from the weak saliency map to train a dedicated classifier. This classifier is applied on each individual pixel of the input image to produce a confidence score. Finally, the confidence scores are used to form a saliency map. This process is carried out at different scales, and the corresponding results are integrated into the formation of the final saliency map. The proposed approach is evaluated on eight benchmark data sets, where it delivers the competitive performance compared with the state-of-the-art methods.

Visual saliency detection based on homology similarity and an experimental evaluation

In recent years, visual saliency detection has become a popular research topic. It can provide useful prior knowledge for high-level vision tasks, such as object detection and image classification. In this paper, a graph-based superpixel-wise similarity called “homology similarity” is proposed, which describes how likely two superpixels belong to the same object or background region. A saliency detection model is then developed based on the combination of homology distribution and improved color contrast. The homology distribution represents spatial compactness, while the color contrast characterizes color conspicuity. By combining these two saliency cues, the proposed model obtains more uniformly highlighted object-level saliency maps with fewer false positive noises. In the experiments, we evaluate our model and 14 competing models (including traditional and state-of-the-art models) on the most popular dataset MSRA-1000 and 4 other publicly available datasets. Experimental results show that, compared with these competing models, our model yields promising results.

Object-level saliency detection with color attributes

Recently, saliency detection has been becoming a popular topic in computer vision. In this paper we propose an object-level saliency detection algorithm which explicitly explores bottom-up visual attention and objectness cues. Firstly, some category-independent object candidates are segmented from the image by the quantized color attributes. Then two metrics, global cues and candidate objectness, are developed to estimate the saliency in the whole image and the object candidates respectively. We use global cues to describe the focusness and spatial distribution of color attributes in the entire image. As supplementary, candidate objectness can reveal the objectness of the object candidates. Finally, we integrate the two saliency cues to derive a saliency map of the image. By explicitly fusing candidate objectness and global cues, our proposed method is more suitable for processing images with complex background. Experimental results on three large benchmark databases demonstrate that the proposed method achieves more favorable performance than 15 recent state-of-the-art methods in terms of three evaluation criterions.

Spatial-Aware Object-Level Saliency Prediction by Learning Graphlet Hierarchies

To fill the semantic gap between the predictive power of computational saliency models and human behavior, this paper proposes to predict where people look at using spatial-aware object-level cues. While object-level saliency has been recently suggested by psychophysics experiments and shown effective with a few computational models, the spatial relationship between the objects has not yet been explored in this context. We in this work for the first time explicitly model such spatial relationship, as well as leveraging semantic information of an image to enhance object-level saliency modeling. The core computational module is a graphlet-based (i.e., graphlets are moderate-sized connected subgraphs) deep architecture, which hierarchically learns a saliency map from raw image pixels to object-level graphlets (oGLs) and further to spatial-level graphlets (sGLs). Eye tracking data are also used to leverage human experience in saliency prediction. Experimental results demonstrate that the proposed oGLs and sGLs well capture object-level and spatial-level cues relating to saliency, and the resulting saliency model performs competitively compared with the state-of-the-art.

Saliency detection with color contrast based on boundary information and neighbors

Object-level saliency detection is significant in many computer vision tasks. In this paper, we propose a novel saliency detection model based on color contrast and image boundaries. The saliency of an image is defined as the contrast between the image elements (regions) and image boundaries elements (regions). We consider the saliency in two-stage procedure rather than in one stage. First of all, according to the definition of saliency, we take four boundaries of image into consideration respectively to obtain a combination coarse saliency map. Furthermore, a new energy function based on the coarse saliency map is proposed, which takes the coarse saliency map as input to yield the final full resolution saliency map. Experimental results on two public datasets demonstrate that the proposed model performs better than the state-of-the-art methods.

A Unified Approach of Object-Level Saliency Detection in Images and Videos

Object-level saliency detection is an important branch of visual saliency. In this paper, we propose a novel method which can conduct object-level saliency detection in both images and videos in a unified way. We employ a more effective spatial compactness assumption to measure saliency instead of the popular contrast assumption. In addition, we present a combination framework which integrates multiple saliency maps generated in different feature maps. The proposed algorithm can automatically select saliency maps of high quality according to the quality evaluation score we define. The experimental results demonstrate that the proposed method outperforms all state-of-the-art methods on both of the datasets of still images and video sequences.

Graph-Regularized Saliency Detection With Convex-Hull-Based Center Prior

Object level saliency detection is useful for many content-based computer vision tasks. In this letter, we present a novel bottom-up salient object detection approach by exploiting contrast, center and smoothness priors. First, we compute an initial saliency map using contrast and center priors. Unlike most existing center prior based methods, we apply the convex hull of interest points to estimate the center of the salient object rather than directly use the image center. This strategy makes the saliency result more robust to the location of objects. Second, we refine the initial saliency map through minimizing a continuous pairwise saliency energy function with graph regularization which encourages adjacent pixels or segments to take the similar saliency value (i.e., smoothness prior). The smoothness prior enables the proposed method to uniformly highlight the salient object and simultaneously suppress the background effectively. Extensive experiments on a large dataset demonstrate that the proposed method performs favorably against the state-of-the-art methods in terms of accuracy and efficiency.