Co-saliency Detection Methods Research Articles

Co-Saliency Detection Guided by Group Weakly Supervised Learning

The detection results of many existing co-saliency detection methods are easily interfered by the unrelated salient objects, which have similar appearance characteristics to co-salient objects. Therefore, mining the inter-saliency cues which contain the common category information of multiple related images is the core of co-saliency detection. To address above concern, a novel group weakly supervised learning induced co-saliency detection (GWSCoSal) model is proposed in this paper. First of all, a novel group class activation maps (GCAM) network is constructed and trained through a group weakly supervised learning scheme, which adopts the common category of a group of related images as the ground truth. The GCAM produced by the trained GCAM network are considered as the inter-saliency cues, which can only highlight the regions covered by the objects with common category. Afterwards, the GCAM are integrated into a feature pyramid networks (FPN) based backbone trained by the pixel-level labels to infer the co-saliency maps. The group weakly supervised and the pixel-level learning are jointly implemented for end-to-end training of GWSCoSal model. The comprehensive comparisons with 13 state-of-the-art methods demonstrate that, our GWSCoSal model can detect the co-salient objects more accurately under the condition of being interfered by the similar unrelated salient objects, and the overall performance of which has achieved the level of state-of-the-art methods. The ablation study of our GWSCoSal model validates the effectiveness of proposed GCAM network.

Co-Saliency Detection of RGBD Image Based on Superpixel and Hypergraph

For the co-saliency detection algorithm of an RGBD image that may have incomplete detection of common salient regions and unclear boundaries, we proposed an improved co-saliency detection method of RGBD images based on superpixels and hypergraphs. First, we optimized the depth map based on edge consistency, and introduced the optimized depth map into the SLIC algorithm to obtain the better superpixel segmentation results of RGBD images. Second, the color features, optimized depth features and global spatial features of superpixels were extracted to construct a weighted hypergraph model to generate saliency maps. Finally, we constructed a weighted hypergraph model for co-saliency detection based on the relationship of color features, global spatial features, optimized depth features and saliency features among images. In addition, in order to verify the impact of the symmetry of the optimized depth information on the co-saliency detection results, we compared the proposed method with two types of models, which included considering depth information and not considering depth information. The experimental results on Cosal150 and Coseg183 datasets showed that our improved algorithm had the advantages of suppressing the background and detecting the integrity of the common salient region, and outperformed other algorithms on the metrics of P-R curve, F-measure and MAE.

Object-Oriented Change Detection Method Based on Spectral–Spatial–Saliency Change Information and Fuzzy Integral Decision Fusion for HR Remote Sensing Images

Spectral features in remote sensing images are extensively utilized to detect land cover changes. However, detection noise appearing in the changing maps due to the abundant spatial details in the high-resolution images makes it difficult to acquire an accurate interpretation result. In this paper, an object-oriented change detection approach is proposed which integrates spectral–spatial–saliency change information and fuzzy integral decision fusion for high-resolution remote sensing images with the purpose of eliminating the impact of detection noise. First, to reduce the influence of feature uncertainty, spectral feature change is generated by three independent methods, and spatial change information is obtained by spatial feature set construction and the optimal feature selection strategy. Secondly, the saliency change map of bi-temporal images is obtained with the co-saliency detection method to complement the insufficiency of image features. Then, the image objects are acquired by multi-scale segmentation based on the staking images. Finally, different pixel-level image change information and the segmentation result are fused using the fuzzy integral decision theory to determine the object change probability. Three high-resolution remote sensing image datasets and three comparative experiments were carried out to evaluate the performance of the proposed algorithm. Spectral–spatial–saliency change information was found to play a major role in the change detection of high-resolution remote sensing images, and the fuzzy integral decision strategy was found to effectively obtain reliable changed objects to improve the accuracy and robustness of change detection.

Co-saliency-regularized correlation filter for object tracking

Correlation filtering (CF) has emerged as one of the best object-tracking frameworks, achieving a good balance between tracking accuracy and speed. Based on this framework, many successful trackers have been developed. We propose a new co-saliency regularization method for visual object tracking based on the correlation filter, called the CRCF. To the best our knowledge, this is the first application of co-saliency regularization to CF-based tracking, in the context of widely used co-saliency detection and regularization methods. In the CRCF, the co-saliency information is extracted and introduced into the regularization component in the relevant tracking framework. The model first calculates the salient regions based on the feature differences between salient objects and background information. A co-saliency map is generated for the global correspondence of implicit learning between multiple images. To avoid traversing all the pixels, we use SLIC superpixel clustering to extract the objects saliency information, thus significantly reducing the computational overhead in the image processing step while retaining the image features and structural information. Finally, during the tracking process, it is possible to dynamically learn the co-saliency map configuration, highlight relevant object areas, and mitigate the impact of the lack of discriminatory representations on the performance. Quantitative and qualitative experimental results on OTB-2015, UAV123, LaSOT, GOT-10k and VOT-2018 datasets show that our CRCF outperforms the latest trackers.

Multi-scale Graph Fusion for Co-saliency Detection

The key challenge of co-saliency detection is to extract discriminative features to distinguish the common salient foregrounds from backgrounds in a group of relevant images. In this paper, we propose a new co-saliency detection framework which includes two strategies to improve the discriminative ability of the features. Specifically, on one hand, we segment each image to semantic superpixel clusters as well as generate different scales/sizes of images for each input image by the VGG-16 model. Different scales capture different patterns of the images. As a result, multi-scale images can capture various patterns among all images by many kinds of perspectives. Second, we propose a new method of Graph Convolutional Network (GCN) to fine-tune the multi-scale features, aiming at capturing the common information among the features from all scales and the private or complementary information for the feature of each scale. Moreover, the proposed GCN method jointly conducts multi-scale feature fine-tune, graph learning, and feature learning in a unified framework. We evaluated our method on three benchmark data sets, compared to state-of-the-art co-saliency detection methods. Experimental results showed that our method outperformed all comparison methods in terms of different evaluation metrics.

Co-Saliency Detection via a General Optimization Model and Adaptive Graph Learning

Co-saliency detection is an important research problem, and has been widely used in computer vision area. One main challenge for co-saliency detection problem is how to explore both interactive information among different images and individual salient information within each image simultaneously in co-saliency estimation. In this paper, we propose a novel general optimization framework with adaptive graph learning for co-saliency estimation problem. The proposed model integrates multiple cues including background, and foreground priors, structural information of images, and image feature representation together to obtain a uniform, and accurate co-saliency estimation. One main benefit of the proposed co-saliency method is that it conducts co-saliency propagation, and prediction across different images while maintains the individual salient information of each image, which ensures the consistency, and communication across different images effectively in co-saliency estimation. To improve the accuracy of co-saliency estimation, we adaptively learn a neighborhood, and structured graph to conduct co-saliency propagation among superpixels. An effective optimization algorithm has been designed to seek the optimal solution for the proposed co-saliency optimization model. Experimental results on several widely used datasets show that our method outperforms some other related co-saliency detection methods.

A model of co-saliency based audio attention

Inspired by biological perceptual characteristics in human auditory systems and the mechanisms of saliency detection, we study the relevance constraint between time-frequency characteristics of sound signals and the multiple spectrogram and propose a co-saliency detection method for multiple sound signals in this paper. Then, according to the auditory characteristics of the human ear, the distinctive saliency features from the acoustic channel and the image channel are fused. Finally, an auditory saliency map is obtained to complete the detection of significant sounds. The saliency features of the acoustic channel include the features calculated in the in the temporal and spectral domains of signal, which the temporal saliency features could be represented by the local maximum points in the Power Spectral Density (PSD) curve, and the spectral features could be represented by local maximum points in Mel Frequency Cepstrum Coefficient (MFCC) curve of sound signal. The saliency features of acoustic channel and cross-scale fusion with the contrast cue of spectrogram, whose result is more in line with the human auditory attention mechanism. Finally, combined with the corresponding cue which could reflect the distribution between multiple spectrograms, it could reflect the characteristics of global repeatability, and reflect high frequency of occurrence. Experimentally, the auditory Co-Saliency map verifies the accuracy and robustness of proposed method in this paper. It shows that the proposed method is superior to other traditional detection methods for auditory saliency, and can implement intelligent automatic detection to sound signals.

Trustful Internet of Surveillance Things Based on Deeply Represented Visual Co-Saliency Detection

Trustful Internet of Things (IoT) plays an important role in smart cities. The trust information in surveillance data motivates the analysis of images from numerous IoT devices. Saliency detection is a fundamental step in surveillance data analysis for providing help to the subsequent tasks, but unsuitable to IoT applications owing to the neglect of image similarity and difference from diverse IoT devices. To solve this problem, we enable the co-saliency detection in IoT, which detects the common and salient foreground regions in the group surveillance images. The main contributions include: 1) enable a multistage context perception scheme to efficiently extract the contextual information corresponding to different-size receptive fields in the single image; 2) construct a two-path information propagation to extract the interimage similarity and difference from the high-level image feature representations of the group images; and 3) propose the stage-wise refinement to allocate the label information to different parts of the network for helping the network to learn the enriched semantically common knowledge. The extensive experiments performed on three public data sets can demonstrate the effectiveness of our approach and its superiority to four state-of-the-art co-saliency detection methods.

HSCS: Hierarchical Sparsity Based Co-saliency Detection for RGBD Images

Co-saliency detection aims to discover common and salient objects in an image group containing more than two relevant images. Moreover, depth information has been demonstrated to be effective for many computer vision tasks. In this paper, we propose a novel co-saliency detection method for RGBD images based on hierarchical sparsity reconstruction and energy function refinement. With the assistance of the intrasaliency map, the inter-image correspondence is formulated as a hierarchical sparsity reconstruction framework. The global sparsity reconstruction model with a ranking scheme focuses on capturing the global characteristics among the whole image group through a common foreground dictionary. The pairwise sparsity reconstruction model aims to explore the corresponding relationship between pairwise images through a set of pairwise dictionaries. In order to improve the intra-image smoothness and inter-image consistency, an energy function refinement model is proposed, which includes the unary data term, spatial smooth term, and holistic consistency term. Experiments on two RGBD co-saliency detection benchmarks demonstrate that the proposed method outperforms the state-of-the-art algorithms both qualitatively and quantitatively.

Co-saliency Detection with Graph Matching

Recently, co-saliency detection, which aims to automatically discover common and salient objects appeared in several relevant images, has attracted increased interest in the computer vision community. In this article, we present a novel graph-matching based model for co-saliency detection in image pairs. A solution of graph matching is proposed to integrate the visual appearance, saliency coherence, and spatial structural continuity for detecting co-saliency collaboratively. Since the saliency and the visual similarity have been seamlessly integrated, such a joint inference schema is able to produce more accurate and reliable results. More concretely, the proposed model first computes the intra-saliency for each image by aggregating multiple saliency cues. The common and salient regions across multiple images are thus discovered via a graph matching procedure. Then, a graph reconstruction scheme is proposed to refine the intra-saliency iteratively. Compared to existing co-saliency detection methods that only utilize visual appearance cues, our proposed model can effectively exploit both visual appearance and structure information to better guide co-saliency detection. Extensive experiments on several challenging image pair databases demonstrate that our model outperforms state-of-the-art baselines significantly.

A Unified Metric Learning-Based Framework for Co-Saliency Detection

Co-saliency detection, which focuses on extracting commonly salient objects in a group of relevant images, has been attracting research interest because of its broad applications. In practice, the relevant images in a group may have a wide range of variations, and the salient objects may also have large appearance changes. Such wide variations usually bring about large intra-co-salient objects (intra-COs) diversity and high similarity between COs and background, which makes the co-saliency detection task more difficult. To address these problems, we make the earliest effort to introduce metric learning to co-saliency detection. Specifically, we propose a unified metric learning-based framework to jointly learn discriminative feature representation and co-salient object detector. This is achieved by optimizing a new objective function that explicitly embeds a metric learning regularization term into support vector machine (SVM) training. Here, the metric learning regularization term is used to learn a powerful feature representation that has small intra-COs scatter, but big separation between background and COs and the SVM classifier is used for subsequent co-saliency detection. In the experiments, we comprehensively evaluate the proposed method on two commonly used benchmark data sets. The state-of-the-art results are achieved in comparison with the existing co-saliency detection methods.

Co-Salient Object Detection Based on Deep Saliency Networks and Seed Propagation Over an Integrated Graph.

This paper presents a co-salient object detection method to find common salient regions in a set of images. We utilize deep saliency networks to transfer co-saliency prior knowledge and better capture high-level semantic information. The resulting initial co-saliency maps are enhanced by seed propagation steps over an integrated graph. The deep saliency networks are trained in a supervised manner to avoid weakly supervised online learning and exploit them not only to extract high-level features but also to produce both intra- and inter-image saliency maps. Through a refinement step, the initial co-saliency maps can uniformly highlight co-salient regions and locate accurate object boundaries. To handle input image groups inconsistent in size, we propose to pool multi-regional descriptors including both within-segment and within-group information. In addition, the integrated multilayer graph is constructed to find the regions that the previous steps may not detect by seed propagation with low-level descriptors. In this paper, we utilize the useful complementary components of high- and low-level information and several learning-based steps. Our experiments have demonstrated that the proposed approach outperforms comparable co-saliency detection methods on widely used public databases and can also be directly applied to co-segmentation tasks.

Revisiting Co-Saliency Detection: A Novel Approach Based on Two-Stage Multi-View Spectral Rotation Co-clustering.

With the goal of discovering the common and salient objects from the given image group, co-saliency detection has received tremendous research interest in recent years. However, as most of the existing co-saliency detection methods are performed based on the assumption that all the images in the given image group should contain co-salient objects in only one category, they can hardly be applied in practice, particularly for the large-scale image set obtained from the Internet. To address this problem, this paper revisits the co-saliency detection task and advances its development into a new phase, where the problem setting is generalized to allow the image group to contain objects in arbitrary number of categories and the algorithms need to simultaneously detect multi-class co-salient objects from such complex data. To solve this new challenge, we decompose it into two sub-problems, i.e., how to identify subgroups of relevant images and how to discover relevant co-salient objects from each subgroup, and propose a novel co-saliency detection framework to correspondingly address the two sub-problems via two-stage multi-view spectral rotation co-clustering. Comprehensive experiments on two publically available benchmarks demonstrate the effectiveness of the proposed approach. Notably, it can even outperform the state-of-the-art co-saliency detection methods, which are performed based on the image subgroups carefully separated by the human labor.

BSFCoS: Block and Sparse Principal Component Analysis-Based Fast Co-Saliency Detection Method

Co-saliency detection, an emerging research area in saliency detection, aims to extract the common saliency from the multi images. The extracted co-saliency map has been utilized in various applications, such as in co-segmentation, co-recognition and so on. With the rapid development of image acquisition technology, the original digital images are becoming more and more clearly. The existing co-saliency detection methods processing these images need enormous computer memory along with high computational complexity. These limitations made it hard to satisfy the demand of real-time user interaction. This paper proposes a fast co-saliency detection method based on the image block partition and sparse feature extraction method (BSFCoS). Firstly, the images are divided into several uniform blocks, and the low-level features are extracted from Lab and RGB color spaces. In order to maintain the characteristics of the original images and reduce the number of feature points as well as possible, Truncated Power for sparse principal components method are employed to extract sparse features. Furthermore, K-Means method is adopted to cluster the extracted sparse features, and calculate the three salient feature weights. Finally, the co-saliency map was acquired from the feature fusion of the saliency map for single image and multi images. The proposed method has been tested and simulated on two benchmark datasets: Co-saliency Pairs and CMU Cornell iCoseg datasets. Compared with the existing co-saliency methods, BSFCoS has a significant running time improvement in multi images processing while ensuring detection results. Lastly, the co-segmentation method based on BSFCoS is also given and has a better co-segmentation performance.

Cosaliency Detection Based on Intrasaliency Prior Transfer and Deep Intersaliency Mining.

As an interesting and emerging topic, cosaliency detection aims at simultaneously extracting common salient objects in multiple related images. It differs from the conventional saliency detection paradigm in which saliency detection for each image is determined one by one independently without taking advantage of the homogeneity in the data pool of multiple related images. In this paper, we propose a novel cosaliency detection approach using deep learning models. Two new concepts, called intrasaliency prior transfer and deep intersaliency mining, are introduced and explored in the proposed work. For the intrasaliency prior transfer, we build a stacked denoising autoencoder (SDAE) to learn the saliency prior knowledge from auxiliary annotated data sets and then transfer the learned knowledge to estimate the intrasaliency for each image in cosaliency data sets. For the deep intersaliency mining, we formulate it by using the deep reconstruction residual obtained in the highest hidden layer of a self-trained SDAE. The obtained deep intersaliency can extract more intrinsic and general hidden patterns to discover the homogeneity of cosalient objects in terms of some higher level concepts. Finally, the cosaliency maps are generated by weighted integration of the proposed intrasaliency prior, deep intersaliency, and traditional shallow intersaliency. Comprehensive experiments over diverse publicly available benchmark data sets demonstrate consistent performance gains of the proposed method over the state-of-the-art cosaliency detection methods.

A Review on Co-Saliency Detection Methods for Complex Background Images

Human eye is perceptually more sensitive to certain colors and intensities and objects with such features are considered more salient. Co-saliency detection aims at discovering the common salient objects existing in multiple images, which is a relatively under-explored area. A review of various techniques used in co-saliency detection like cluster based co-saliency detection, Hierarchical segmentation, Base reconstruction, Low rank matrix recovery etc. are done in this paper. Detection of Salient image regions is useful in applications such as co-segmentation, common pattern discovery, co recognition, Image retrieval etc. The survey is represented in a tabular form for quick reference and also describes the advantages and disadvantages of papers.

Detecting Co-Salient Objects in Large Image Sets

Co-salient object detection has attracted much more attention recently as it is useful for many problems in vision computing. However, most of existing methods emphasize detecting the common salient objects in a small group of images and the objects of interest in those images have clear borders with respect to the backgrounds. In this work, we propose a novel co-saliency detection method, which aims at discovering the common objects in a large and diverse image set composed of hundreds of images. First, we search a group of similar images for each image in the set. Our method is based on the overlapped groups. We handle each group with an unsupervised random forest to extract the rough contours of the common objects. Then a contrast-based measure is utilized to produce the saliency map for an individual image. For each image in the set, we collect all the maps from the groups that contain the image and fuse them together as the inter-saliency map for the image. The final co-saliency map is computed by combining the inter-saliency map with the single saliency map of this image. Experimental evaluation on an established large dataset demonstrates that our approach attains superior results and outperforms the state-of-the-art methods.