Co-saliency Detection Research Articles

A Unified Transformer Framework for Group-Based Segmentation: Co-Segmentation, Co-Saliency Detection and Video Salient Object Detection

Humans tend to mine objects by learning from a group of images or several frames of video since we live in a dynamic world. In the computer vision area, many researchers focus on co-segmentation (CoS), co-saliency detection (CoSD) and video salient object detection (VSOD) to discover the co-occurrent objects. However, previous approaches design different networks for these similar tasks separately, and they are difficult to apply to each other. Besides, they fail to take full advantage of the cues among inter- and intra-feature within a group of images. In this paper, we introduce a unified framework to tackle these issues from a unified view, term as UFGS (Unified Framework for Group-based Segmentation). Specifically, we first introduce a transformer block, which views the image feature as a patch token and then captures their long-range dependencies through the self-attention mechanism. This can help the network to excavate the patch-structured similarities among the relevant objects. Furthermore, we propose an intra-MLP learning module to produce self-mask to enhance the network to avoid partial activation. Extensive experiments on four CoS benchmarks (PASCAL, iCoseg, Internet and MSRC), three CoSD benchmarks (Cosal2015, CoSOD3k, and CocA) and five VSOD benchmarks (DAVIS <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$_{16}$</tex-math></inline-formula> , FBMS, ViSal, SegV2 and DAVSOD) show that our method outperforms other state-of-the-arts on three different tasks in both accuracy and speed by using the same network architecture, which can reach 140 FPS in real-time. Code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/suyukun666/UFO</uri>

Co-saliency detection with two-stage co-attention mining and individual calibration

Learning-based methods have become popular in co-salient object detection (CoSOD). However, existing methods suffer from two challenging issues, including mining the inter-image co-attention and calibrating the intra-image salient objects. Moreover, the training data is insufficient. To address these challenges, we propose an end-to-end network using Two-stage Co-attention mining and Individual Calibration (TCIC) to predict the co-salient objects. Firstly, a two-stage co-attention mining architecture (TCM), including a classified co-attention module (CCM) and a focal co-attention module (FCM), is designed to model inter-image relationships. In the first stage, a CCM is applied to capture the classification interactions of multiple images, tentatively extracting the co-attention. In the second stage, we propose an FCM to adaptively suppress and aggregate multiple salient features, aiming to recalibrate the co-attention in the first stage. Secondly, considering the shape features and location information offered by the boundary features, an edge guidance module (EGM) is embedded into the individual calibration architecture (ICA) to calibrate individuals. Besides, we also adopt a co-attention transfer strategy (CTS) to keep the consistency of the co-attention during feature transfer in the decoder. Finally, TCM and ICA are integrated into a unified end-to-end framework to predict fine-grained boundary-preserving results. Besides, an image fusion algorithm (IFA) is tailored without extra pixel-level annotations for automatic generation of the composite images, aiming to supplement the training dataset. Experimental results on three prevailing testing datasets show the superiority of the proposed method in terms of various evaluation metrics.

An end-to-end network for co-saliency detection in one single image

Co-saliency detection within a single image is a common vision problem that has received little attention and has not yet been well addressed. Existing methods often used a bottom-up strategy to infer co-saliency in an image in which salient regions are firstly detected using visual primitives such as color and shape and then grouped and merged into a co-saliency map. However, co-saliency is intrinsically perceived complexly with bottom-up and top-down strategies combined in human vision. To address this problem, this study proposes a novel end-to-end trainable network comprising a backbone net and two branch nets. The backbone net uses ground-truth masks as top-down guidance for saliency prediction, whereas the two branch nets construct triplet proposals for regional feature mapping and clustering, which drives the network to be bottom-up sensitive to co-salient regions. We construct a new dataset of 2,019 natural images with co-saliency in each image to evaluate the proposed method. Experimental results show that the proposed method achieves state-of-the-art accuracy with a running speed of 28 fps.

Co-Saliency Detection Guided by Group Weakly Supervised Learning

The detection results of many existing co-saliency detection methods are easily interfered by the unrelated salient objects, which have similar appearance characteristics to co-salient objects. Therefore, mining the inter-saliency cues which contain the common category information of multiple related images is the core of co-saliency detection. To address above concern, a novel group weakly supervised learning induced co-saliency detection (GWSCoSal) model is proposed in this paper. First of all, a novel group class activation maps (GCAM) network is constructed and trained through a group weakly supervised learning scheme, which adopts the common category of a group of related images as the ground truth. The GCAM produced by the trained GCAM network are considered as the inter-saliency cues, which can only highlight the regions covered by the objects with common category. Afterwards, the GCAM are integrated into a feature pyramid networks (FPN) based backbone trained by the pixel-level labels to infer the co-saliency maps. The group weakly supervised and the pixel-level learning are jointly implemented for end-to-end training of GWSCoSal model. The comprehensive comparisons with 13 state-of-the-art methods demonstrate that, our GWSCoSal model can detect the co-salient objects more accurately under the condition of being interfered by the similar unrelated salient objects, and the overall performance of which has achieved the level of state-of-the-art methods. The ablation study of our GWSCoSal model validates the effectiveness of proposed GCAM network.

Deep Object Co-Segmentation and Co-Saliency Detection via High-Order Spatial-Semantic Network Modulation

Deep Object Co-Segmentation and Co-Saliency Detection via High-Order Spatial-Semantic Network Modulation

Modified ResNet50 model and semantic segmentation based image co-saliency detection

Co-salient object identification is a new and emerging branch of the visual saliency detection technique that tries to find salient pattern appearing in several image groups. The proposed work has the potential to benefit a wide variety of important applications including the detection of objects of interest, more robust object recognition and animation synthesis, handling input image query, 3D object reconstruction, object co-segmentation etc. To build modified ResNet50 model, the hyperparameters are adjusted in the current work to increase accuracy while minimizing loss. The modified network is trained on HOG features to mine more significant features along with their corresponding ground truth images. For a more streamlined outcome, the proposed system was built using the SGDM optimizer. During testing among the relevant and irrelevant image the network generates appropriate co-saliency map of relevant images. Integrating the associated and prominent characteristics of the image yields the appropriate ground truth for each image. The proposed method reports better F1 value 98.7% and MAE score 0.089 value when compared with SOTA model.

Adaptive Group-Wise Consistency Network for Co-Saliency Detection

Adaptive Group-Wise Consistency Network for Co-Saliency Detection

Self-supervised image co-saliency detection

Self-supervised learning has been shown to be highly effective to avoid the expensive cost of collecting and annotating large-scale datasets. In this paper, we explore its use in image co-saliency detection task which detects and segments the co-concurrent and conspicuous patterns in one image group, and a self-supervised framework is presented that contains three novel module designs. Specifically, we first design an unsupervised graph clustering algorithm to generate an initial detection result from extracted semantical features of convolutional networks. Then, the post-processed output is taken as the pseudo labels of the second module that learns a neural network for more accurate sample affinity estimation, followed by a manifold ranking based label smoothing. Finally, with the smoothed labels as supervision, we design a saliency network that aggregates a hierarchy of multi-scale features to predict final optimized co-saliency maps. In addition, the proposed framework can be readily extended to image saliency object detection task when giving a single image as input. Extensive evaluations on three co-saliency and five saliency object detection benchmark datasets demonstrate that the proposed framework achieves favorable performance against a variety of supervised and unsupervised state-of-the-arts.

Co-Saliency Detection of RGBD Image Based on Superpixel and Hypergraph

For the co-saliency detection algorithm of an RGBD image that may have incomplete detection of common salient regions and unclear boundaries, we proposed an improved co-saliency detection method of RGBD images based on superpixels and hypergraphs. First, we optimized the depth map based on edge consistency, and introduced the optimized depth map into the SLIC algorithm to obtain the better superpixel segmentation results of RGBD images. Second, the color features, optimized depth features and global spatial features of superpixels were extracted to construct a weighted hypergraph model to generate saliency maps. Finally, we constructed a weighted hypergraph model for co-saliency detection based on the relationship of color features, global spatial features, optimized depth features and saliency features among images. In addition, in order to verify the impact of the symmetry of the optimized depth information on the co-saliency detection results, we compared the proposed method with two types of models, which included considering depth information and not considering depth information. The experimental results on Cosal150 and Coseg183 datasets showed that our improved algorithm had the advantages of suppressing the background and detecting the integrity of the common salient region, and outperformed other algorithms on the metrics of P-R curve, F-measure and MAE.

Background-independent evaluation model for infrared jamming effectiveness of false targets

Given the lack of a standardized evaluation system for the infrared jamming effectiveness of false targets, this paper first uses a co-saliency detection model to extract the main parts of the true and false targets. Then the perceptual similarity algorithm is improved by combining the operational requirements of false targets in the infrared band. Finally, a background-independent evaluation model for infrared jamming effectiveness of false targets is constructed. The experimental results show that the model can quantitatively reflect the infrared jamming effectiveness of a single false target and distinguish the infrared jamming effectiveness of different types of false targets. In addition, the model has stronger robustness than traditional evaluation models.

Re-Thinking the Relations in Co-Saliency Detection

Co-salient object detection (CoSOD) aims to detect common salient objects sharing the same attributes in an image group. The key issue of CoSOD is how to model the inter-saliency relations within an image group. The major limitation of previous methods is that they pre-define the group-to-one relations within an image group. In this paper, we propose a new concept of structural inter-saliency relations and solve the CoSOD with deep reinforcement learning framework. Firstly, we design a semantic relation graph (SRG) to model the structural inter-saliency relations. Then the feature selecting agent (FS-agent) aims to select the informative features, which can help the SRG effectively model structural inter-saliency relations. Finally, relation updating agent (RU-agent) progressively updates the SRG to focus on the co-salient relations like human decision-making process. Extensive experiments on co-saliency datasets show that because of well modeling inter-saliency relations in image group, our proposed method achieves superior performance compared to the state-of-the-art methods. We hope that this paper can motivate future research for visual co-analysis tasks.

Object-Oriented Change Detection Method Based on Spectral–Spatial–Saliency Change Information and Fuzzy Integral Decision Fusion for HR Remote Sensing Images

Spectral features in remote sensing images are extensively utilized to detect land cover changes. However, detection noise appearing in the changing maps due to the abundant spatial details in the high-resolution images makes it difficult to acquire an accurate interpretation result. In this paper, an object-oriented change detection approach is proposed which integrates spectral–spatial–saliency change information and fuzzy integral decision fusion for high-resolution remote sensing images with the purpose of eliminating the impact of detection noise. First, to reduce the influence of feature uncertainty, spectral feature change is generated by three independent methods, and spatial change information is obtained by spatial feature set construction and the optimal feature selection strategy. Secondly, the saliency change map of bi-temporal images is obtained with the co-saliency detection method to complement the insufficiency of image features. Then, the image objects are acquired by multi-scale segmentation based on the staking images. Finally, different pixel-level image change information and the segmentation result are fused using the fuzzy integral decision theory to determine the object change probability. Three high-resolution remote sensing image datasets and three comparative experiments were carried out to evaluate the performance of the proposed algorithm. Spectral–spatial–saliency change information was found to play a major role in the change detection of high-resolution remote sensing images, and the fuzzy integral decision strategy was found to effectively obtain reliable changed objects to improve the accuracy and robustness of change detection.

Co-saliency detection with intra-group two-stage group semantics propagation and inter-group contrastive learning

Co-salient object detection (CoSOD) aims at discovering the salient objects with repeatability from multiple relevant images and suppressing non-repetitively appearing salient and non-salient ones. Recently, learning-based methods have shown the advantages in the CoSOD tasks. However, modeling the intra-group semantics and suppressing the inter-group noise objects are still two challenges. In this paper, we propose a unified Two-stage grOup semantics PropagatIon and Contrastive learning NETwork (TopicNet) for CoSOD. TopicNet consists of an intra-group two-stage group semantics propagation module (TGSP) and an inter-group contrastive learning module (CLM). On one hand, in the first stage of TGSP, we put forward an image-to-group propagation module (IGP) to capture the inter-image similarity feature interactions. In the second stage of TGSP, a group-to-pixel propagation module (GPP) is designed to construct the connection between inter-image interactions and inter-pixel correspondences to distill consensus representation. GPP redistributes the interactions with different attention weights to each pixel so as to improve the robustness of group semantics. On the other hand, in CLM, using the design of positive samples enhances the semantic consistency of group semantics. Using the design of negative samples explores the interactive suppression relationships of co-salient objects and noise objects, which suppresses non-repetitively appearing salient and non-salient objects. TGSP and CLM are jointly integrated into an end-to-end unified network predicting the co-salient objects. Experimental results on three prevailing benchmarks reveal that TopicNet outperforms other competitors under various evaluation metrics.

Co-saliency-regularized correlation filter for object tracking

Correlation filtering (CF) has emerged as one of the best object-tracking frameworks, achieving a good balance between tracking accuracy and speed. Based on this framework, many successful trackers have been developed. We propose a new co-saliency regularization method for visual object tracking based on the correlation filter, called the CRCF. To the best our knowledge, this is the first application of co-saliency regularization to CF-based tracking, in the context of widely used co-saliency detection and regularization methods. In the CRCF, the co-saliency information is extracted and introduced into the regularization component in the relevant tracking framework. The model first calculates the salient regions based on the feature differences between salient objects and background information. A co-saliency map is generated for the global correspondence of implicit learning between multiple images. To avoid traversing all the pixels, we use SLIC superpixel clustering to extract the objects saliency information, thus significantly reducing the computational overhead in the image processing step while retaining the image features and structural information. Finally, during the tracking process, it is possible to dynamically learn the co-saliency map configuration, highlight relevant object areas, and mitigate the impact of the lack of discriminatory representations on the performance. Quantitative and qualitative experimental results on OTB-2015, UAV123, LaSOT, GOT-10k and VOT-2018 datasets show that our CRCF outperforms the latest trackers.

Cosaliency Detection and Region-of-Interest Extraction via Manifold Ranking and MRF in Remote Sensing Images

Saliency-based region-of-interest (ROI) extraction is significant for the interpretation of remote sensing images (RSIs). Recently, cosaliency detection has shown its superiority of better extraction of common ROIs by using both intraimage and interimage cues. However, most existing methods still suffer from the complex backgrounds of RSIs, resulting in incomplete ROI extraction, many false positives, and blurred boundaries. In this article, we propose a cosaliency detection framework via manifold ranking and the Markov random field (MRF) for RSIs to address these problems. First, we design a two-stage manifold ranking schema for converting single-image saliency maps (SISMs) to multi-image saliency maps (MISMs). This step takes full advantage of the correlation between images to improve the integrity of ROIs and reduce false positives. Second, we locally fuse saliency proposals by minimizing the energy function in an MRF. The design of the energy function comprehensively considers the global and local performance of saliency proposals to assign appropriate fusion weights. Finally, we generate the ROI masks by thresholding the cosaliency maps. Our approach is evaluated on four RSI datasets and compared to the state-of-the-art methods. Experimental results demonstrate the effectiveness of our model in both cosaliency detection and ROI extraction.

Image Co-Saliency Detection and Instance Co-Segmentation Using Attention Graph Clustering Based Graph Convolutional Network

Co-Saliency Detection (CSD) is to explore the concurrent patterns and salient objects from a group of relevant images, while Instance Co-Segmentation (ICS) aims to identify and segment out all of these co-salient instances, generating corresponding mask for each instance. To simultaneously tackle these two tasks, we present a novel adaptive graph convolutional network with attention graph clustering (GCAGC) for CSD and ICS, termed as GCAGC-CSD and GCAGC-ICS, respectively. The GCAGC-CSD contains three key model designs: first, we develop a graph convolutional network architecture to extract multi-scale representations to characterize the intra- and inter-image consistency. Second, we propose an attention graph clustering algorithm to distinguish the salient foreground objects from common areas in an unsupervised manner. Third, we present a unified framework with encoder-decoder structure to jointly train and optimize the graph convolutional network, attention graph cluster, and CSD decoder in an end-to-end fashion. Afterwards, we design a salient instance segmentation network for GCAGC-ICS, and combine the outputs of GCAGC-CSD and the instance segmentation branch to obtain instance-aware co-segmentation masks. The proposed GCAGC-CSD and GCAGC-ICS are extensively evaluated on four CSD benchmark datasets (iCoseg, Cosal2015, COCO-SEG and CoSOD3k) and five ICS benchmark datasets (CoSOD3k, COCO-NONVOC, COCO-VOC, VOC12 and SOC), and achieve superior performance over state-of-the-arts on both tasks.

Toward Stable Co-Saliency Detection and Object Co-Segmentation.

In this paper, we present a novel model for simultaneous stable co-saliency detection (CoSOD) and object co-segmentation (CoSEG). To detect co-saliency (segmentation) accurately, the core problem is to well model inter-image relations between an image group. Some methods design sophisticated modules, such as recurrent neural network (RNN), to address this problem. However, order-sensitive problem is the major drawback of RNN, which heavily affects the stability of proposed CoSOD (CoSEG) model. In this paper, inspired by RNN-based model, we first propose a multi-path stable recurrent unit (MSRU), containing dummy orders mechanisms (DOM) and recurrent unit (RU). Our proposed MSRU not only helps CoSOD (CoSEG) model captures robust inter-image relations, but also reduces order-sensitivity, resulting in a more stable inference and training process. Moreover, we design a cross-order contrastive loss (COCL) that can further address order-sensitive problem by pulling close the feature embedding generated from different input orders. We validate our model on five widely used CoSOD datasets (CoCA, CoSOD3k, Cosal2015, iCoseg and MSRC), and three widely used datasets (Internet, iCoseg and PASCAL-VOC) for object co-segmentation, the performance demonstrates the superiority of the proposed approach as compared to the state-of-the-art (SOTA) methods.

Reverse collaborative fusion model for co-saliency detection

Reverse collaborative fusion model for co-saliency detection

Multi-scale Graph Fusion for Co-saliency Detection

The key challenge of co-saliency detection is to extract discriminative features to distinguish the common salient foregrounds from backgrounds in a group of relevant images. In this paper, we propose a new co-saliency detection framework which includes two strategies to improve the discriminative ability of the features. Specifically, on one hand, we segment each image to semantic superpixel clusters as well as generate different scales/sizes of images for each input image by the VGG-16 model. Different scales capture different patterns of the images. As a result, multi-scale images can capture various patterns among all images by many kinds of perspectives. Second, we propose a new method of Graph Convolutional Network (GCN) to fine-tune the multi-scale features, aiming at capturing the common information among the features from all scales and the private or complementary information for the feature of each scale. Moreover, the proposed GCN method jointly conducts multi-scale feature fine-tune, graph learning, and feature learning in a unified framework. We evaluated our method on three benchmark data sets, compared to state-of-the-art co-saliency detection methods. Experimental results showed that our method outperformed all comparison methods in terms of different evaluation metrics.

Airport Detection-Based Cosaliency on Remote Sensing Images

This paper proposes a contour extraction model based on cosaliency detection for remote sensing image airport detection and improves the traditional line segmentation detection (LSD) algorithm to make it more suitable for the goal of this paper. Our model consists of two parts, a cosaliency detection module and a contour extraction module. In the first part, the cosaliency detection module mainly uses the network framework of Visual Geometry Group-19 (VGG-19) to obtain the result maps of the interimage comparison and the intraimage consistency, and then the two result maps are multiplied pixel by pixel to obtain the cosaliency mask. In the second part, the contour extraction module uses superpixel segmentation and parallel line segment detection (PLSD) to refine the airport contour and runway information to obtain the preprocessed result map, and then we merge the result of cosaliency detection with the preprocessed result to obtain the final airport contour. We compared the model proposed in this article with four commonly used methods. The experimental results show that the accuracy of the model is 15% higher than that of the target detection result based on the saliency model, and the accuracy of the active contour model based on the saliency analysis is improved by 1%. This shows that the model proposed in this paper can extract a contour that closely matches the actual target.