Saliency Ranking Research Articles

HyperSOR: Context-Aware Graph Hypernetwork for Salient Object Ranking.

Salient object ranking (SOR) aims to segment salient objects in an image and simultaneously predict their saliency rankings, according to the shifted human attention over different objects. The existing SOR approaches mainly focus on object-based attention, e.g., the semantic and appearance of object. However, we find that the scene context plays a vital role in SOR, in which the saliency ranking of the same object varies a lot at different scenes. In this paper, we thus make the first attempt towards explicitly learning scene context for SOR. Specifically, we establish a large-scale SOR dataset of 24,373 images with rich context annotations, i.e., scene graphs, segmentation, and saliency rankings. Inspired by the data analysis on our dataset, we propose a novel graph hypernetwork, named HyperSOR, for context-aware SOR. In HyperSOR, an initial graph module is developed to segment objects and construct an initial graph by considering both geometry and semantic information. Then, a scene graph generation module with multi-path graph attention mechanism is designed to learn semantic relationships among objects based on the initial graph. Finally, a saliency ranking prediction module dynamically adopts the learned scene context through a novel graph hypernetwork, for inferring the saliency rankings. Experimental results show that our HyperSOR can significantly improve the performance of SOR.

Semantic-Consistency-guided Learning on Deep Features for Unsupervised Salient Object Detection

Unsupervised salient object detection is an important task in many real-world scenarios where pixel-wise label information is of scarce availability. Despite its significance, this problem remains rarely explored, with a few works that consider unsupervised salient object detection methods based on the fused graph from the sum fusion of multiple deep feature similarity matrices. However, these methods ignore the interrelation of the low-level feature similarity matrices and the high-level semantic similarity matrice, which degrades the quality of the fused graph. In this article, we propose a semantic-consistency-guided multi-graph fusion learning algorithm for unsupervised saliency detection, where the consistency and inconsistency between multiple low-level feature similarity matrices and the high-level semantic similarity matrice are explored to promote the robustness and quality of the fused graph. In the first stage, a semantic-consistency-guided multi-graph fusion learning method is proposed to exploit consistency and inconsistency of multiple low-level deep features and the high-level semantic feature. The semantic-consistency-guided similarity matrices are computed for preliminary saliency ranking. In the following saliency refinement stage, the semantic-enhanced similarity matrices are built by the cross diffusion to fuse the multiple low-level deep features and the high semantic deep feature. Based on the semantic-enhanced similarity matrices, the refinement saliency maps are calculated in a semantic-enhanced cellular automata manner. Furthermore, the final ensemble stage of the large margin semi-supervised classification views the preliminary ranking results and refinement results as features, adopts the large margin graphs for saliency ensemble. Extensive evaluations over four benchmark datasets show that the proposed unsupervised method performs favorably against the state-of-the-art approaches and is competitive with some supervised deep learning-based methods.

Rethinking Object Saliency Ranking: A Novel Whole-Flow Processing Paradigm.

Existing salient object detection methods are capable of predicting binary maps that highlight visually salient regions. However, these methods are limited in their ability to differentiate the relative importance of multiple objects and the relationships among them, which can lead to errors and reduced accuracy in downstream tasks that depend on the relative importance of multiple objects. To conquer, this paper proposes a new paradigm for saliency ranking, which aims to completely focus on ranking salient objects by their "importance order". While previous works have shown promising performance, they still face ill-posed problems. First, the saliency ranking ground truth (GT) orders generation methods are unreasonable since determining the correct ranking order is not well-defined, resulting in false alarms. Second, training a ranking model remains challenging because most saliency ranking methods follow the multi-task paradigm, leading to conflicts and trade-offs among different tasks. Third, existing regression-based saliency ranking methods are complex for saliency ranking models due to their reliance on instance mask-based saliency ranking orders. These methods require a significant amount of data to perform accurately and can be challenging to implement effectively. To solve these problems, this paper conducts an in-depth analysis of the causes and proposes a whole-flow processing paradigm of saliency ranking task from the perspective of "GT data generation", "network structure design" and "training protocol". The proposed approach outperforms existing state-of-the-art methods on the widely-used SALICON set, as demonstrated by extensive experiments with fair and reasonable comparisons. The saliency ranking task is still in its infancy, and our proposed unified framework can serve as a fundamental strategy to guide future work. The code and data will be available at https://github.com/MengkeSong/Saliency-Ranking-Paradigm.

RGB-D salient object ranking based on depth stack and truth stack for complex indoor scenes

RGB-D salient object detection has achieved a great development in recent years due to its extensive applications. Previous studies mainly focus on simple scene images with one single object. These models usually become overwhelmed by complex scenes with multiple objects. Moreover, these methods model salient object detection as a binary segmentation problem. However, psychology studies show that humans shift their visual attention from one object to another and rank salient objects, especially in complex indoor scenes. Following the psychological studies, we propose to rank salient objects in RGB-D images of complex indoor scenes. Due to the lack of such data, we first construct a RGB-D salient object ranking dataset containing complex indoor scenes with multiple objects. The saliency ranking of different objects is defined based on the order that an observer notices these objects. The final salient object ranking result is an average across the saliency rankings of 13 observers. This RGB-D salient object ranking dataset is also analyzed with current mainstream RGB-D salient object detection dataset for comparison. Since location information provided by depth images can help to determine the saliency ranking of objects, we further propose an end-to-end network exploiting depth stack and ground truth stack to predict the order of salient objects in complex scenes. The quantitative and qualitative comparisons demonstrate the effectiveness of the proposed method.

Inferentialist semantics for lexicalized social meanings

This paper offers a general model of the semantics of lexicalized social meanings, i.e. semiotic properties of certain expressions in a socio-political context. Examples include slurs, problematically charged expressions such as inner city, as well as terms such as mother, which also carry implicit ideological associations. Insofar as their linguistic properties are concerned, social meanings can be construed as context-structuring devices: without introducing specific at-issue contents, they evoke background assumptions which shape the context of conversation. An inferentialist model of discourse is developed to account for this effect, in which the discursive significance of an utterance is defined as the set of inferences it licenses relative to a discursive context. A discursive context is a set of propositions that can serve as auxiliary premises in material inferences, together with a salience ranking that makes some of these propositions more readily available and therefore more relevant to determining discursive significance. Social meanings are defined as functions on discursive contexts that modify the salience ranking, increasing the salience of certain assumptions and stereotypes. As a result, they impact the discursive significance of utterances indirectly and independently of at-issue contents. They are also largely independent of speaker intentions in virtue of the ideological nature of discursive contexts.

Salient object detection via joint perception of region-level spatial distribution and color contrast

Nowadays, many RGB-D saliency detection models have been proposed, and they are conducted based on the pixel-level depth data obtained by active depth sensors or dense stereo matching algorithms in advance. However, the pixel-level depth maps may have a negative impact on saliency ranking, especially when they are inconsistent or invalid within the object area. It can be found that people tend to observe an object as a whole and ignore the details of the target surface. Inspired by this characteristic, we propose a salient object detection method via joint perception of region-level spatial distribution and color contrast. First, a twice segmentation strategy based on multifeature fusion is used to compute the region-level information. Then, the region-level spatial distribution maps are constructed instead of pixel-level depth maps, which is helpful for avoiding the interference of dense depth information. To improve the integrity of object detection, color saliency maps are also computed based on regional segmentation information. After that, we adopt a fusion strategy to achieve the effective complementarity of the two kinds of information. Two optimization strategies are employed to further improve the results of saliency ranking. Experimental results on two benchmark datasets demonstrate that the proposed method has better performance than most of the state-of-the-art methods, and it also shows competitive capability compared with deep learning-based methods.

Multi-Graph Fusion and Learning for RGBT Image Saliency Detection

RGB and thermal infrared (RGBT) image saliency detection is a relatively new direction in the field of computer vision. Combining the advantages of RGB images and T images can significantly improve detection performance. Currently, there are only a few methods to work on RGBT saliency detection, and the number of image samples cannot meet the training requirements for deep learning, so it remains valuable to propose an effective unsupervised method. In this paper, we present an unsupervised RGBT saliency detection method based on multi-graph fusion and learning. Firstly, RGB images and T images are adaptively fused based on boundary information to produce more accurate superpixels. Next, a multi-graph fusion model is proposed to selectively learn useful information from multi-modal images. Finally, we implement the theory of finding good neighbors in the graph affinity and propose different algorithms for two stages of saliency ranking. Experimental results on three RGBT datasets show that the proposed method is effective compared with the state-of-the-art algorithms.

Instance-Level Relative Saliency Ranking With Graph Reasoning.

Conventional salient object detection models cannot differentiate the importance of different salient objects. Recently, two works have been proposed to detect saliency ranking by assigning different degrees of saliency to different objects. However, one of these models cannot differentiate object instances and the other focuses more on sequential attention shift order inference. In this paper, we investigate a practical problem setting that requires simultaneously segment salient instances and infer their relative saliency rank order. We present a novel unified model as the first end-to-end solution, where an improved Mask R-CNN is first used to segment salient instances and a saliency ranking branch is then added to infer the relative saliency. For relative saliency ranking, we build a new graph reasoning module by combining four graphs to incorporate the instance interaction relation, local contrast, global contrast, and a high-level semantic prior, respectively. A novel loss function is also proposed to effectively train the saliency ranking branch. Besides, a new dataset and an evaluation metric are proposed for this task, aiming at pushing forward this field of research. Finally, experimental results demonstrate that our proposed model is more effective than previous methods. We also show an example of its practical usage on adaptive image retargeting.

Stage-wise Salient Object Detection in 360° Omnidirectional Image via Object-level Semantical Saliency Ranking.

The 2D image based salient object detection (SOD) has been extensively explored, while the 360° omnidirectional image based SOD has received less research attention and there exist three major bottlenecks that are limiting its performance. Firstly, the currently available training data is insufficient for the training of 360° SOD deep model. Secondly, the visual distortions in 360° omnidirectional images usually result in large feature gap between 360° images and 2D images; consequently, the widely used stage-wise training-a widely-used solution to alleviate the training data shortage problem, becomes infeasible when conducing SOD in 360° omnidirectional images. Thirdly, the existing 360° SOD approach has followed a multi-task methodology that performs salient object localization and segmentation-like saliency refinement at the same time, being faced with extremely large problem domain, making the training data shortage dilemma even worse. To tackle all these issues, this paper divides the 360° SOD into a multi-staqe task, the key rationale of which is to decompose the original complex problem domain into sequential easy sub problems that only demand for small-scale training data. Meanwhile, we learn how to rank the "object-level semantical saliency", aiming to locate salient viewpoints and objects accurately. Specifically, to alleviate the training data shortage problem, we have released a novel dataset named 360-SSOD, containing 1,105 360° omnidirectional images with manually annotated object-level saliency ground truth, whose semantical distribution is more balanced than that of the existing dataset. Also, we have compared the proposed method with 13 SOTA methods, and all quantitative results have demonstrated the performance superiority.

Local Regression Ranking for Saliency Detection.

Saliency detection is an important and challenging research topic due to the variety and complex of the background and saliency regions. In this paper, we present a novel unsupervised saliency detection approach by exploiting a learning-based ranking framework. First, the local linear regression model is adopted to simulate the local manifold structure of every image element, which is approximately linear. Using the background queries from the boundary prior, we construct a unified objective function to globally minimize all the errors of the local models for the whole image element points. The Laplacian matrix is learned via optimizing the unified objective function. Low-level image features as well as high-level semantic information extracted from deep neural networks are used for the Laplacian matrix learning. Based on the learnt Laplacian matrix, the saliency of the image element is measured as the relevance ranking to the background queries. The foreground queries are obtained from the background-based saliency and the relevance ranking to the foreground queries is calculated in the same way as the background-based saliency. Second, we calculate an enhanced similarity matrix by fusing two different-level deep feature metrics through cross diffusion. A propagation algorithm uses this enhanced similarity matrix to better exploit the intrinsic relevance of similar regions and improve the saliency ranking results effectively. Results on four benchmark datasets with pixel-wise accurate labelling demonstrate that the proposed unsupervised method shows better performance compared with the newest state-of-the-art methods and is competitive with deep learning-based methods.

Rapid Ocular Responses Are Modulated by Bottom-up-Driven Auditory Salience.

Despite the prevalent use of alerting sounds in alarms and human–machine interface systems and the long-hypothesized role of the auditory system as the brain's “early warning system,” we have only a rudimentary understanding of what determines auditory salience—the automatic attraction of attention by sound—and which brain mechanisms underlie this process. A major roadblock has been the lack of a robust, objective means of quantifying sound-driven attentional capture. Here we demonstrate that: (1) a reliable salience scale can be obtained from crowd-sourcing (N = 911), (2) acoustic roughness appears to be a driving feature behind this scaling, consistent with previous reports implicating roughness in the perceptual distinctiveness of sounds, and (3) crowd-sourced auditory salience correlates with objective autonomic measures. Specifically, we show that a salience ranking obtained from online raters correlated robustly with the superior colliculus-mediated ocular freezing response, microsaccadic inhibition (MSI), measured in naive, passively listening human participants (of either sex). More salient sounds evoked earlier and larger MSI, consistent with a faster orienting response. These results are consistent with the hypothesis that MSI reflects a general reorienting response that is evoked by potentially behaviorally important events regardless of their modality.SIGNIFICANCE STATEMENT Microsaccades are small, rapid, fixational eye movements that are measurable with sensitive eye-tracking equipment. We reveal a novel, robust link between microsaccade dynamics and the subjective salience of brief sounds (salience rankings obtained from a large number of participants in an online experiment): Within 300 ms of sound onset, the eyes of naive, passively listening participants demonstrate different microsaccade patterns as a function of the sound's crowd-sourced salience. These results position the superior colliculus (hypothesized to underlie microsaccade generation) as an important brain area to investigate in the context of a putative multimodal salience hub. They also demonstrate an objective means for quantifying auditory salience.

Saliency detection via affinity graph learning and weighted manifold ranking

Abstract Graph-based saliency detection approaches have gained great popularity due to the simplicity and efficiency of graph algorithms. In these approaches, the saliency values of image elements are ranked by the similarity of image elements with foreground or background cues via graph-based ranking. However, in previous methods, the similarity between any two image elements on the affinity graph is computed by manually set functions which are sensitive to function parameters, and the constructed graph may not reveal the essentially relevance between feature vectors extracted from different image elements. In addition, during the saliency ranking process, all the initial labels contribute equally to the ranking function while the global saliency confidence of each image element is not taken into consideration. In order to address these two issues, we propose a bottom-up saliency detection approach by affinity graph learning and weighted manifold ranking. An unsupervised learning approach is introduced to learn the affinity graph based on image data self-representation. By setting image boundary superpixels as background seeds, the global saliency confidence prior implied in the affinity matrix is utilized to weight the saliency ranking. In such a manner, the superpixels with higher saliency confidences will be assigned higher saliency values in the final saliency map and the background superpixels can be efficiently suppressed. Comprehensive evaluations on three challenge datasets indicate that our algorithm universally surpasses other unsupervised graph based saliency detection methods.

Saliency ranker: A new salient object detection method

Recently, saliency detection has become an active research topic in learning from labeled image, where various supervised methods were designed. Many existing methods usually cast saliency detection as a binary classification or regression problem, in which saliency detection performance relies heavily on the expensive pixel-wise annotations of salient objects. This paper addresses the issue by developing a novel learning-to-rank model with a limited number of training data, which combines the strength of cost-sensitive label ranking methods with the power of low-rank matrix recovery theories. Rather than using a binary decision for each saliency value, our approach ranks saliency values in a descending order with the estimated relevance to the given saliency. Additionally, we also aggregate the prediction models for different saliency labels into a matrix, and solve saliency ranking via a low-rank matrix recovery problem. Extensive experiments over challenging benchmarks clearly validate advantage of our method.

Visual Saliency Predicts Fixations in Low Clutter Web Pages

Previous research has shown a computational model of visual saliency can predict where people fixate in cluttered web pages (Masciocchi & Still, 2013). Over time, web site designers are moving towards simpler, less cluttered webpages to improve aesthetics and to make searches more efficient. Even with simpler interfaces, determining a saliency ranking among interface elements is a difficult task. Also, it is unclear whether the traditionally employed saliency model (Itti, Koch, & Niebur, 1998) can be applied to simpler interfaces. To examine the model’s ability to predict fixations in simple web pages we compared a distribution of observed fixations to a conservative measure of chance performance (a shuffled distribution). Simplicity was determined by using two visual clutter models (Rosenholz, Li, & Nakano, 2007). We found under free-viewing conditions that the saliency model was able to predict fixations within less cluttered web pages.

Automatic tag saliency ranking for stereo images

With the rapid advances in 3D capture and display technology, tag ranking for stereo images will be a potential application on web image retrieval. Directly extending the existing approaches to stereo images is problematic as it may neglect the representative 3D elements. In this paper, a novel automatic tag saliency ranking algorithm for stereo images is presented. Specifically, a novel method of interacting with stereo images is proposed to segment the two images into regions simultaneously. Then tags annotated on the image-level are propagated to the region-level via an improved multi-instance learning algorithm. In the next, a new 3D saliency detection algorithm is proposed using occlusion cues along with the contrast in color and disparity. And finally, tags are re-ranked according to the saliency values of the corresponding regions. Moreover, to evaluate the performance of our approach, an annotated stereo image dataset is set up. The experimental results on this dataset demonstrate the effectiveness of our approach.

Adaptive all-season image tag ranking by saliency-driven image pre-classification

Social image tag ranking has emerged as an important research topic due to its application on web image search. This paper presents an adaptive all-season tag ranking algorithm which can handle the images with and without distinct object(s) using different tag ranking strategies. Firstly, based on saliency map derived from the visual attention model, a linear SVM is trained to pre-classify an image as attentive or non-attentive category by using the gray histogram descriptor on the corresponding saliency map. Then, an image with distinct object is processed by the tag saliency ranking algorithm emphasizing distinct object, which combines image saliency map with sparse representation based multi-instance learning algorithm. On the other hand, an image without distinct object can be processed by the tag relevance ranking algorithm via the sparse representation based neighbor-voting strategy. Such adaptive all-season tag ranking strategy can be regarded as taking full advantage of existing tag ranking paradigms. Experiments conducted on well-known image data sets demonstrate the effectiveness of the proposed framework.

Alert Response to Motion Onset in the Retina

Previous studies have shown that motion onset is very effective at capturing attention and is more salient than smooth motion. Here, we find that this salience ranking is present already in the firing rate of retinal ganglion cells. By stimulating the retina with a bar that appears, stays still, and then starts moving, we demonstrate that a subset of salamander retinal ganglion cells, fast OFF cells, responds significantly more strongly to motion onset than to smooth motion. We refer to this phenomenon as an alert response to motion onset. We develop a computational model that predicts the time-varying firing rate of ganglion cells responding to the appearance, onset, and smooth motion of a bar. This model, termed the adaptive cascade model, consists of a ganglion cell that receives input from a layer of bipolar cells, represented by individual rectified subunits. Additionally, both the bipolar and ganglion cells have separate contrast gain control mechanisms. This model captured the responses to our different motion stimuli over a wide range of contrasts, speeds, and locations. The alert response to motion onset, together with its computational model, introduces a new mechanism of sophisticated motion processing that occurs early in the visual system.

Combining visual attention model with multi-instance learning for tag ranking

Tag ranking has emerged as an important research topic recently due to its potential application on web image search. Existing tag relevance ranking approaches mainly rank the tags according to their relevance levels with respect to a given image. Nonetheless, such algorithms heavily rely on the large-scale image dataset and the proper similarity measurement to retrieve semantic relevant images with multi-labels. In contrast to the existing tag relevance ranking algorithms, in this paper, we propose a novel tag saliency ranking scheme, which aims to automatically rank the tags associated with a given image according to their saliency to the image content. To this end, this paper presents an integrated framework for tag saliency ranking, which combines both visual attention model and multi-instance learning to investigate the saliency ranking order information of tags with respect to the given image. Specifically, tags annotated on the image-level are propagated to the region-level via an efficient multi-instance learning algorithm firstly; then, visual attention model is employed to measure the importance of regions in the given image. Finally, tags are ranked according to the saliency values of the corresponding regions. Experiments conducted on the COREL and MSRC image datasets demonstrate the effectiveness and efficiency of the proposed framework.

Extracting salient contours for target recognition: algorithm and performance evaluation

To achieve robust and efficient object recognition, particularly from real outdoor images, we must develop methods to reduce clutter and extract salient information of objects. Toward this end, we present a technique to rank and extract salient contours from a 2-D image acquired by a passive sensor. The goal is to find important contours corresponding to possible objects. Our method starts with edge pixels, or edgels, from an edge detector and assigns a saliency measure to linked edgels (contours) based on length, smoothness, and contrast. For length we use the number of edgels in the contour, for smoothness we use average change of curvature, and for contrast we use the edge magnitude. Contours are ranked by saliency, and the more salient contours are selected. We test this method on several real outdoor images of objects in cluttered and occluded conditions and obtain excellent results. We evaluate the performance of this technique in the context of a recognition system that matches 2-D image corners with 3-D model vertices. We present graphs, using corners on the object of interest and clutter, to demonstrate the appropriateness of saliency ranking. We plot curves displaying the percentage of object corners to all image corners for the top few salient contours. We conclude by observing that extracting the more salient contours increases the ratio of image corners on the object to all image corners, reducing the search space for the corner matching step in recognition.