Co-salient Objects Research Articles

Group-wise co-salient object detection via multi-view self-labeling novel class discovery

Group-wise co-salient object detection via multi-view self-labeling novel class discovery

An Information Calibration and Sliding Mining Network for Co-Saliency Object Detection

An Information Calibration and Sliding Mining Network for Co-Saliency Object Detection

Co-saliency detection with two-stage co-attention mining and individual calibration

Learning-based methods have become popular in co-salient object detection (CoSOD). However, existing methods suffer from two challenging issues, including mining the inter-image co-attention and calibrating the intra-image salient objects. Moreover, the training data is insufficient. To address these challenges, we propose an end-to-end network using Two-stage Co-attention mining and Individual Calibration (TCIC) to predict the co-salient objects. Firstly, a two-stage co-attention mining architecture (TCM), including a classified co-attention module (CCM) and a focal co-attention module (FCM), is designed to model inter-image relationships. In the first stage, a CCM is applied to capture the classification interactions of multiple images, tentatively extracting the co-attention. In the second stage, we propose an FCM to adaptively suppress and aggregate multiple salient features, aiming to recalibrate the co-attention in the first stage. Secondly, considering the shape features and location information offered by the boundary features, an edge guidance module (EGM) is embedded into the individual calibration architecture (ICA) to calibrate individuals. Besides, we also adopt a co-attention transfer strategy (CTS) to keep the consistency of the co-attention during feature transfer in the decoder. Finally, TCM and ICA are integrated into a unified end-to-end framework to predict fine-grained boundary-preserving results. Besides, an image fusion algorithm (IFA) is tailored without extra pixel-level annotations for automatic generation of the composite images, aiming to supplement the training dataset. Experimental results on three prevailing testing datasets show the superiority of the proposed method in terms of various evaluation metrics.

3D salient object detection based on light field integral imaging.

Potent usage of the multi-scale light field information for salient object detection (SOD) is the essential requirement of three-dimensional (3D) SOD. On this basis, a light field 3D-SOD scheme is proposed that employs the pixel mapping algorithm to achieve a more distinct representation of spatial and angular information in the four-dimensional (4D) light field, collaboratively mining the global saliency cues via the co-salient object detection (CoSOD) network. Compared with the previous method, our scheme filters out most of the noise by thoroughly leveraging the global dependence of the 4D light field, offering significant enhancements in saliency extraction performance and efficiency. Additionally, the 3D reconstruction results demonstrate the integral retention of the spatial and angular information of the original light field.

GCoNet+: A Stronger Group Collaborative Co-Salient Object Detector.

In this paper, we present a novel end-to-end group collaborative learning network, termed GCoNet+, which can effectively and efficiently (250 fps) identify co-salient objects in natural scenes. The proposed GCoNet+ achieves the new state-of-the-art performance for co-salient object detection (CoSOD) through mining consensus representations based on the following two essential criteria: 1) intra-group compactness to better formulate the consistency among co-salient objects by capturing their inherent shared attributes using our novel group affinity module (GAM); 2) inter-group separability to effectively suppress the influence of noisy objects on the output by introducing our new group collaborating module (GCM) conditioning on the inconsistent consensus. To further improve the accuracy, we design a series of simple yet effective components as follows: i) a recurrent auxiliary classification module (RACM) promoting model learning at the semantic level; ii) a confidence enhancement module (CEM) assisting the model in improving the quality of the final predictions; and iii) a group-based symmetric triplet (GST) loss guiding the model to learn more discriminative features. Extensive experiments on three challenging benchmarks, i.e., CoCA, CoSOD3k, and CoSal2015, demonstrate that our GCoNet+ outperforms the existing 12 cutting-edge models. Code has been released at https://github.com/ZhengPeng7/GCoNet_plus.

Memory-Aided Contrastive Consensus Learning for Co-salient Object Detection

Co-salient object detection (CoSOD) aims at detecting common salient objects within a group of relevant source images. Most of the latest works employ the attention mechanism for finding common objects. To achieve accurate CoSOD results with high-quality maps and high efficiency, we propose a novel Memory-aided Contrastive Consensus Learning (MCCL) framework, which is capable of effectively detecting co-salient objects in real time (∼150 fps). To learn better group consensus, we propose the Group Consensus Aggregation Module (GCAM) to abstract the common features of each image group; meanwhile, to make the consensus representation more discriminative, we introduce the Memory-based Contrastive Module (MCM), which saves and updates the consensus of images from different groups in a queue of memories. Finally, to improve the quality and integrity of the predicted maps, we develop an Adversarial Integrity Learning (AIL) strategy to make the segmented regions more likely composed of complete objects with less surrounding noise. Extensive experiments on all the latest CoSOD benchmarks demonstrate that our lite MCCL outperforms 13 cutting-edge models, achieving the new state of the art (∼5.9% and ∼6.2% improvement in S-measure on CoSOD3k and CoSal2015, respectively). Our source codes, saliency maps, and online demos are publicly available at https://github.com/ZhengPeng7/MCCL.

TCNet: Co-Salient Object Detection via Parallel Interaction of Transformers and CNNs

The purpose of co-salient object detection (CoSOD) is to detect the salient objects that co-occur in a group of relevant images. CoSOD has been significantly prospered by recent advances in <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">convolutional neural networks</i> (CNNs). However, it shows general limitations in modeling long-range feature dependencies, which is crucial for CoSOD. In the vision transformer, the self-attention mechanism is utilized to capture global dependencies but unfortunately destroy local spatial details, which are also essential for CoSOD. To address the above issues, we propose a dual network structure, called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TCNet</i> , which can efficiently excavate both local information and global representations for co-saliency learning via the parallel interaction of Transformers and CNNs. Specifically, it contains three critical components, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> ., the mutual consensus module (MCM), the consensus complementary module (CCM), and the group consistent progressive decoder (GCPD). MCM aims to capture the global consensus from high-level features of these two branches as a guide for the following integration of consensus cues of both branches at each level. Next, CCM is designed to effectively fuse the consensus of local information and global contexts from different levels of the two branches. Finally, GCPD is developed to maintain group feature consistency and predict accurate co-saliency maps. The proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TCNet</i> is evaluated on five challenging CoSOD benchmark datasets using six widely used metrics, showing that our proposed method is superior to other existing cutting-edge methods for co-salient object detection.

Hierarchical interaction and pooling network for co-salient object detection

Co-salient object detection (CoSOD) aims to detect common and salient objects across the given image group. Due to the particularity of CoSOD, images in the given group are processed synergistically to excavate the relevance between them. Inspired by the tracking methods, previous works tend to utilize pixel-wise correspondences to measure the relevance. However, because of the complexity of the image groups, the obtained feature maps could be easily affected by the common interference. Moreover, current works tend to utilize classification labels to ensure intra-group coherence and inter-group separability, which may cause some overfitting problems. In this paper, we propose the hierarchical interaction and pooling network to alleviate the above problems. We first design a pyramid pooling interaction module and perform convolution with dimension permutation, making full use of convolution and multi-receptive field information. We further propose the coherence confirmation module along with the four-branch architecture. Without the classification labels, the module achieves comparable or even better performance. Extensive experiments demonstrate that the proposed method can detect common and salient objects more accurately and achieves the new state-of-the-art.

Global-and-Local Collaborative Learning for Co-Salient Object Detection.

The goal of co-salient object detection (CoSOD) is to discover salient objects that commonly appear in a query group containing two or more relevant images. Therefore, how to effectively extract interimage correspondence is crucial for the CoSOD task. In this article, we propose a global-and-local collaborative learning (GLNet) architecture, which includes a global correspondence modeling (GCM) and a local correspondence modeling (LCM) to capture the comprehensive interimage corresponding relationship among different images from the global and local perspectives. First, we treat different images as different time slices and use 3-D convolution to integrate all intrafeatures intuitively, which can more fully extract the global group semantics. Second, we design a pairwise correlation transformation (PCT) to explore similarity correspondence between pairwise images and combine the multiple local pairwise correspondences to generate the local interimage relationship. Third, the interimage relationships of the GCM and LCM are integrated through a global-and-local correspondence aggregation (GLA) module to explore more comprehensive interimage collaboration cues. Finally, the intra and inter features are adaptively integrated by an intra-and-inter weighting fusion (AEWF) module to learn co-saliency features and predict the co-saliency map. The proposed GLNet is evaluated on three prevailing CoSOD benchmark datasets, demonstrating that our model trained on a small dataset (about 3k images) still outperforms 11 state-of-the-art competitors trained on some large datasets (about 8k-200k images).

Co-Salient Object Detection With Co-Representation Purification.

Co-salient object detection (Co-SOD) aims at discovering the common objects in a group of relevant images. Mining a co-representation is essential for locating co-salient objects. Unfortunately, the current Co-SOD method does not pay enough attention that the information not related to the co-salient object is included in the co-representation. Such irrelevant information in the co-representation interferes with its locating of co-salient objects. In this paper, we propose a Co-Representation Purification (CoRP) method aiming at searching noise-free co-representation. We search a few pixel-wise embeddings probably belonging to co-salient regions. These embeddings constitute our co-representation and guide our prediction. For obtaining purer co-representation, we use the prediction to iteratively reduce irrelevant embeddings in our co-representation. Experiments on three datasets demonstrate that our CoRP achieves state-of-the-art performances on the benchmark datasets. Our source code is available at https://github.com/ZZY816/CoRP.

Co-Saliency Detection Guided by Group Weakly Supervised Learning

The detection results of many existing co-saliency detection methods are easily interfered by the unrelated salient objects, which have similar appearance characteristics to co-salient objects. Therefore, mining the inter-saliency cues which contain the common category information of multiple related images is the core of co-saliency detection. To address above concern, a novel group weakly supervised learning induced co-saliency detection (GWSCoSal) model is proposed in this paper. First of all, a novel group class activation maps (GCAM) network is constructed and trained through a group weakly supervised learning scheme, which adopts the common category of a group of related images as the ground truth. The GCAM produced by the trained GCAM network are considered as the inter-saliency cues, which can only highlight the regions covered by the objects with common category. Afterwards, the GCAM are integrated into a feature pyramid networks (FPN) based backbone trained by the pixel-level labels to infer the co-saliency maps. The group weakly supervised and the pixel-level learning are jointly implemented for end-to-end training of GWSCoSal model. The comprehensive comparisons with 13 state-of-the-art methods demonstrate that, our GWSCoSal model can detect the co-salient objects more accurately under the condition of being interfered by the similar unrelated salient objects, and the overall performance of which has achieved the level of state-of-the-art methods. The ablation study of our GWSCoSal model validates the effectiveness of proposed GCAM network.

Modified ResNet50 model and semantic segmentation based image co-saliency detection

Co-salient object identification is a new and emerging branch of the visual saliency detection technique that tries to find salient pattern appearing in several image groups. The proposed work has the potential to benefit a wide variety of important applications including the detection of objects of interest, more robust object recognition and animation synthesis, handling input image query, 3D object reconstruction, object co-segmentation etc. To build modified ResNet50 model, the hyperparameters are adjusted in the current work to increase accuracy while minimizing loss. The modified network is trained on HOG features to mine more significant features along with their corresponding ground truth images. For a more streamlined outcome, the proposed system was built using the SGDM optimizer. During testing among the relevant and irrelevant image the network generates appropriate co-saliency map of relevant images. Integrating the associated and prominent characteristics of the image yields the appropriate ground truth for each image. The proposed method reports better F1 value 98.7% and MAE score 0.089 value when compared with SOTA model.

GSNNet: Group semantic-guided neighbor interaction network for co-salient object detection

The critical challenge for co-salient object detection (CoSOD) is to extract common saliency information from a group of relevant images. Most of the existing CoSOD methods do not fully explore the semantic commonality of co-salient objects, which can be strong guidance for collaborative feature learning, and do not take full advantage of rich hierarchical features of different layers, resulting in inferior performance. To this end, we propose a Group Semantic-guided Neighbor interaction network (GSNNet) for co-salient object detection. Specifically, the proposed network contains the group semantic module (GSM), neighbor interaction module (NIM), and feature enhancement module (FEM). The network first learns semantic consensus from a group of relevant images by the GSM, which uses the reverse guidance strategy and the group-wise combination strategy to distill the group semantic cues from the forward and complementary features. With the powerful guidance of group semantic, the NIM is employed to conduct the neighbor feature interaction of adjacent layers to excavate the contextual information and enhance feature representation. Then the FEM is adopted to refine the critical cues by the attention mechanism, which enhances the compactness of feature representation. The proposed GSNNet is evaluated on three challenging CoSOD benchmark datasets using four widely-used metrics, which demonstrates that our proposed method is superior to the other twelve cutting-edge methods for co-salient object detection.

Similarity activation map for co-salient object detection

• A novel yet efficient model (SimiNet) is proposed for the Co-SOD task. • The similarity activation module is designed as the substitution of the pixel-wise similarity. • Two lightweight modules (EEM and FFM) are proposed to introduce the edge information into Co-SOD thoroughly. • The EEM and FFM can be easily applied to other methods and improve their performance. • Empirical evaluations show that SimiNet achieves state-of-the-art performance. Co-salient object detection aims to detect the common objects within a group of relevant images, to which the spatial similarity contributes a lot. Existing methods utilize the inner product to compute the pixel-wise correlations, imitating the tracking methods. We present a novel yet effective module (Similarity Activation Module, SAM) to generate the similarity activation maps as the spatial modulator. The similarity activation maps are learned to highlight the common objects across the multiple images while suppressing other objects and the background. Moreover, we propose the Edge Extraction Module (EEM) and Feature Fusion Module (FFM) which can be easily applied to any existing methods without requiring architectural changes. Extensive experiments on different co-salient detection datasets demonstrate that our method (SimiNet) achieves state-of-the-art performance under various evaluation metrics .

Re-Thinking the Relations in Co-Saliency Detection

Co-salient object detection (CoSOD) aims to detect common salient objects sharing the same attributes in an image group. The key issue of CoSOD is how to model the inter-saliency relations within an image group. The major limitation of previous methods is that they pre-define the group-to-one relations within an image group. In this paper, we propose a new concept of structural inter-saliency relations and solve the CoSOD with deep reinforcement learning framework. Firstly, we design a semantic relation graph (SRG) to model the structural inter-saliency relations. Then the feature selecting agent (FS-agent) aims to select the informative features, which can help the SRG effectively model structural inter-saliency relations. Finally, relation updating agent (RU-agent) progressively updates the SRG to focus on the co-salient relations like human decision-making process. Extensive experiments on co-saliency datasets show that because of well modeling inter-saliency relations in image group, our proposed method achieves superior performance compared to the state-of-the-art methods. We hope that this paper can motivate future research for visual co-analysis tasks.

Global-guided cross-reference network for co-salient object detection

Global-guided cross-reference network for co-salient object detection

Co-saliency detection with intra-group two-stage group semantics propagation and inter-group contrastive learning

Co-salient object detection (CoSOD) aims at discovering the salient objects with repeatability from multiple relevant images and suppressing non-repetitively appearing salient and non-salient ones. Recently, learning-based methods have shown the advantages in the CoSOD tasks. However, modeling the intra-group semantics and suppressing the inter-group noise objects are still two challenges. In this paper, we propose a unified Two-stage grOup semantics PropagatIon and Contrastive learning NETwork (TopicNet) for CoSOD. TopicNet consists of an intra-group two-stage group semantics propagation module (TGSP) and an inter-group contrastive learning module (CLM). On one hand, in the first stage of TGSP, we put forward an image-to-group propagation module (IGP) to capture the inter-image similarity feature interactions. In the second stage of TGSP, a group-to-pixel propagation module (GPP) is designed to construct the connection between inter-image interactions and inter-pixel correspondences to distill consensus representation. GPP redistributes the interactions with different attention weights to each pixel so as to improve the robustness of group semantics. On the other hand, in CLM, using the design of positive samples enhances the semantic consistency of group semantics. Using the design of negative samples explores the interactive suppression relationships of co-salient objects and noise objects, which suppresses non-repetitively appearing salient and non-salient objects. TGSP and CLM are jointly integrated into an end-to-end unified network predicting the co-salient objects. Experimental results on three prevailing benchmarks reveal that TopicNet outperforms other competitors under various evaluation metrics.

A benchmark dataset and baseline model for co-salient object detection within RGB-D images

A benchmark dataset and baseline model for co-salient object detection within RGB-D images

Learning Implicit Class Knowledge for RGB-D Co-Salient Object Detection With Transformers.

RGB-D co-salient object detection aims to segment co-occurring salient objects when given a group of relevant images and depth maps. Previous methods often adopt separate pipeline and use hand-crafted features, being hard to capture the patterns of co-occurring salient objects and leading to unsatisfactory results. Using end-to-end CNN models is a straightforward idea, but they are less effective in exploiting global cues due to the intrinsic limitation. Thus, in this paper, we alternatively propose an end-to-end transformer-based model which uses class tokens to explicitly capture implicit class knowledge to perform RGB-D co-salient object detection, denoted as CTNet. Specifically, we first design adaptive class tokens for individual images to explore intra-saliency cues and then develop common class tokens for the whole group to explore inter-saliency cues. Besides, we also leverage the complementary cues between RGB images and depth maps to promote the learning of the above two types of class tokens. In addition, to promote model evaluation, we construct a challenging and large-scale benchmark dataset, named RGBD CoSal1k, which collects 106 groups containing 1000 pairs of RGB-D images with complex scenarios and diverse appearances. Experimental results on three benchmark datasets demonstrate the effectiveness of our proposed method.

CoU2Net and CoLDF: Two Novel Methods Built on Basis of Double-Branch Co-Salient Object Detection Framework

In this paper, we propose a framework for evaluation of co-salient object detection algorithms along with two novel CoSOD methods. The processing pipeline of this framework is based on the CoEGNet algorithm, where the saliency detection part can be easily replaced by any saliency detector to be evaluated. By leveraging the proposed framework, we developed two new algorithms: one based on U2Net and the other based on the label decoupling framework (LDF). They are called in this paper CoU2Net and CoLDF, respectively. The proposed solutions were tested on three datasets: CoCA, CoSal2015, and CoSOD3k, and compared with some of the best algorithms in co-silent object detection: GICD and CoEGNet. The advantages and disadvantages of the proposed methods are highlighted and discussed. As a generalization of the aforementioned methods, a framework called Sal.Co has been also proposed. It is a modification of the CoEGNet method and it works on a saliency mask obtained from a saliency detector and attempts to indicate salient objects coexisting in a group of images. Both CoLDF and CoU2Net achieved better results than CoEGNet on the CoCA dataset. On the CoSal2015 and CoSOD3k datasets, they performed similarly to state-of-the-art methods, while maintaining a highly customizable structure. The source code can be found at: https://github.com/jakubkorczakowski/cosal_sal_testing.