Coarse Saliency Map Research Articles

An Adaptive Multi-Content Complementary Network for Salient Object Detection

Deep learning methods for salient object detection (SOD) have been studied actively and promisingly. However, the existing methods mainly focus on the decoding process and ignore the differences in the contributions of different encoder blocks. To address this problem for SOD, we propose an adaptive multi-content complementary network (PASNet) for salient object detection which aims to exploit the valuable contextual information in the encoder fully. Unlike existing CNN-based methods, we adopt the pyramidal visual transformer (PVTv2) as the backbone network to learn global and local representations with its self-attention mechanism. Then, we follow the coarse-to-fine strategy and introduce two novel modules, including an advanced semantic fusion module (ASFM) and a self-refinement module (SRM). Among these, the ASFM takes local branches and adjacent branches as inputs and collects semantic and location information of salient objects from high-level features to generate an initial coarse saliency map. The coarse saliency map serves as the location guidance for low-level features, and the SRM is applied to capture detailed information disguised in low-level features. We expand the location information with high-level semantics from top to bottom across the salient region, which is effectively fused with detailed information through feature modulation. The model effectively suppresses noises in the features and significantly improves their expressive capabilities. To verify the effectiveness of our PASNet, we conducted extensive experiments on five challenging datasets, and the results show that the proposed model is superior to some of the current state-of-the-art methods under different evaluation metrics.

LeNo: Adversarial Robust Salient Object Detection Networks with Learnable Noise

Pixel-wise prediction with deep neural network has become an effective paradigm for salient object detection (SOD) and achieved remarkable performance. However, very few SOD models are robust against adversarial attacks which are visually imperceptible for human visual attention. The previous work robust saliency (ROSA) shuffles the pre-segmented superpixels and then refines the coarse saliency map by the densely connected conditional random field (CRF). Different from ROSA that rely on various pre- and post-processings, this paper proposes a light-weight Learnable Noise (LeNo) to defend adversarial attacks for SOD models. LeNo preserves accuracy of SOD models on both adversarial and clean images, as well as inference speed. In general, LeNo consists of a simple shallow noise and noise estimation that embedded in the encoder and decoder of arbitrary SOD networks respectively. Inspired by the center prior of human visual attention mechanism, we initialize the shallow noise with a cross-shaped gaussian distribution for better defense against adversarial attacks. Instead of adding additional network components for post-processing, the proposed noise estimation modifies only one channel of the decoder. With the deeply-supervised noise-decoupled training on state-of-the-art RGB and RGB-D SOD networks, LeNo outperforms previous works not only on adversarial images but also on clean images, which contributes stronger robustness for SOD. Our code is available at https://github.com/ssecv/LeNo.

Transformers and CNNs fusion network for salient object detection

Recently, Salient Object Detection (SOD) has achieved promising results thanks to the rapid evolution of CNN architectures. However, those CNN-based methods show limited capacity in modeling long-range interaction among pixels. In this paper, we propose to combine the merits from CNN and Transformer and design a unified model for RGB and RGB-D SOD. First, we represent the image with a sequence-to-sequence perspective and use a Transformer-based branch to express long-distance relationships of image tokens to obtain global semantic information and predict a coarse saliency map. Second, we employ a CNN-based branch to extract multi-scale local detail features to predict contour prediction for auxiliary supervision at each level. Finally, we propose the Bi-enhancement Fusion Module to fuse multi-scale cues from two branches to predict a more accurate saliency map. In addition, for RGB-D SOD, to obtain effective cross-modality features, we propose a Cross-modality Multi-Scale Transformer Module and a Depth-induced Enhancement Module to fuse RGB and depth cues in the Transformers branch and the CNNs branch, respectively. Experiments on both RGB and RGB-D SOD datasets demonstrate that our proposed model achieves satisfactory performance compared with state-of-the-art methods.

DS-Net: Dynamic spatiotemporal network for video salient object detection

As moving objects always draw more attention of human eyes, the temporal motion information is always exploited complementarily with spatial information to detect salient objects in videos. Although efficient tools such as optical flow have been proposed to extract temporal motion information, it often encounters difficulties when used for saliency detection due to the movement of camera or the partial movement of salient objects. In this paper, we investigate the complementary roles of spatial and temporal information and propose a novel dynamic spatiotemporal network (DS-Net) for more effective fusion of spatiotemporal information. We construct a symmetric two-bypass network to explicitly extract spatial and temporal features. A dynamic weight generator (DWG) is designed to automatically learn the reliability of corresponding saliency branch. And a top-down cross attentive aggregation (CAA) procedure is designed to facilitate dynamic complementary aggregation of spatiotemporal features. Finally, the features are modified by spatial attention with the guidance of coarse saliency map and then go through decoder part for final saliency map. Experimental results on five benchmarks VOS, DAVIS, FBMS, SegV2, and ViSal demonstrate that the proposed method achieves superior performance than state-of-the-art algorithms. The source code is available at https://github.com/TJUMMG/DS-Net.

Salient Region Detection via Multiple Hierarchy and Competition Mechanism

The detection of salient regions has attracted an increasing attention in machine vision. In this study, a novel and effective framework for saliency region detection is proposed to solve the problem of the low detection accuracy of traditional methods. Firstly, we divide the image into three levels. Second, each level uses three different feature methods to generate different feature saliency maps. Subsequently, a novel integration mechanism, termed competition mechanism, is introduced into the coarse saliency maps at the same level, and the two coarse saliency maps with the highest similarity are selected for fusion to ensure the effectiveness of the salient region map. Accordingly, after adjusting the scales of the saliency map after the fusion of different levels, among three coarse saliency maps of the different levels, the two feature maps with the most significant difference are selected to fuse to further obtain the final refined saliency map. Finally, using the proposed method, experiments on three benchmark datasets were conducted. As demonstrated by the experimental results, the proposed algorithm is superior to other state-of-the-art methods.

Lightweight Salient Object Detection in Optical Remote Sensing Images via Feature Correlation

Salient object detection in optical remote sensing images (ORSI-SOD) has been widely explored for understanding ORSIs. However, previous methods focus mainly on improving the detection accuracy while neglecting the cost in memory and computation, which may hinder their real-world applications. In this paper, we propose a novel lightweight ORSI-SOD solution, named CorrNet, to address these issues. In CorrNet, we first lighten the backbone (VGG-16) and build a lightweight subnet for feature extraction. Then, following the coarse-to-fine strategy, we generate an initial coarse saliency map from high-level semantic features in a Correlation Module (CorrM). The coarse saliency map serves as the location guidance for low-level features. In CorrM, we mine the object location information between high-level semantic features through the cross-layer correlation operation. Finally, based on low-level detailed features, we refine the coarse saliency map in the refinement subnet equipped with Dense Lightweight Refinement Blocks, and produce the final fine saliency map. By reducing the parameters and computations of each component, CorrNet ends up having only 4.09M parameters and running with 21.09G FLOPs. Experimental results on two public datasets demonstrate that our lightweight CorrNet achieves competitive or even better performance compared with 26 state-of-the-art methods (including 16 large CNN-based methods and 2 lightweight methods), and meanwhile enjoys the clear memory and run time efficiency. The code and results of our method are available at https://github.com/MathLee/CorrNet.

CAT-EDNet: Cross-Attention Transformer-Based Encoder–Decoder Network for Salient Defect Detection of Strip Steel Surface

The morphologies of various surface defects on strip steel suffer from oil stain, water drops, steel textures, and erratic illumination. It is still challenging to recognize defect boundary precisely from cluttered backgrounds. This article emphasizes such a fact that skip connections between encoder and decoder are not equally effective, attempts to adaptively allocate the aggregation weights that represent differentiated information entropy values in channelwise, by importing a stack of cross-attention transformer (CAT) into the encoder–decoder network (EDNet). Besides, a cross-attention refinement module (CARM) is constructed closely after the decoder to further optimize the coarse saliency maps. This newly nominated CAT-EDNet can well address the semantic gap issue among the multiscale features for its multihead attention structure. The CAT-EDNet performs best on insuring defect integrity and maintaining defect boundary details when compared with 12 state-of-the-arts, and the detection efficiency is at 28 fps even under the noise interfered scenario.

Revise-Net: Exploiting Reverse Attention Mechanism for Salient Object Detection

Recently, deep learning-based methods, especially utilizing fully convolutional neural networks, have shown extraordinary performance in salient object detection. Despite its success, the clean boundary detection of the saliency objects is still a challenging task. Most of the contemporary methods focus on exclusive edge detection modules in order to avoid noisy boundaries. In this work, we propose leveraging on the extraction of finer semantic features from multiple encoding layers and attentively re-utilize it in the generation of the final segmentation result. The proposed Revise-Net model is divided into three parts: (a) the prediction module, (b) a residual enhancement module, and (c) reverse attention modules. Firstly, we generate the coarse saliency map through the prediction modules, which are fine-tuned in the enhancement module. Finally, multiple reverse attention modules at varying scales are cascaded between the two networks to guide the prediction module by employing the intermediate segmentation maps generated at each downsampling level of the REM. Our method efficiently classifies the boundary pixels using a combination of binary cross-entropy, similarity index, and intersection over union losses at the pixel, patch, and map levels, thereby effectively segmenting the saliency objects in an image. In comparison with several state-of-the-art frameworks, our proposed Revise-Net model outperforms them with a significant margin on three publicly available datasets, DUTS-TE, ECSSD, and HKU-IS, both on regional and boundary estimation measures.

Double cross-modality progressively guided network for RGB-D salient object detection

With the advent of depth sensors, the use of RGB and depth(D) information for salient object detection (SOD) has been explored extensively in recent years. However, the depth quality from the different scenes usually varies, leading to fusing and achieving complementary between RGB and low-quality depth is still a challenging problem. In this paper, we first design a Double Dilated Merge Module (DDMM) to extract comprehensive and beneficial high-level cross-modality features and explore further global context information at multi-scales to obtain a coarse saliency map. Then, we propose a Cross-Modality Enhance Module (CMEM) to enhance cross-modality features compatibility and fuse them with the previous predicted coarse saliency map to generate a more accurate saliency map. Furthermore, we introduce a Distribution-Region Combination Loss (DRcom Loss) to optimize our proposed Double Cross-Modality Progressively Guided Network (DCPGNet) in a coarse-to-fine manner. DCPGNet achieves satisfactory performance on five public benchmarks compared with recent state-of-the-art algorithms.

Salient object detection via a boundary-guided graph structure

Graph-based salient object detection methods have gained more and more attention recently. However, existing works fail to separate effectively salient object and background in some challenging scenes. Inspired by this observation, we propose an effective salient object detection method based on a novel boundary-guided graph structure. More specifically, the input image is firstly segmented into a series of superpixels. Then we integrate two prior cues to generate the coarse saliency map, a novel weighting mechanism is proposed to balance the proportion of two prior cues according to their performance. Secondly, we propose a novel boundary-guided graph structure to explore deeply the intrinsic relevance between superpixels. Based on the proposed graph structure, an iterative propagation mechanism is constructed to refine the coarse saliency map. Experimental results on four datasets show adequately the superiority of the proposed method than other state-of-the-art methods.

EDRNet: Encoder–Decoder Residual Network for Salient Object Detection of Strip Steel Surface Defects

It is still a challenging task to detect the surface defects of strip steel due to its complex variations, including variable defect types, cluttered background, low contrast, and noise interference. The existing detection methods cannot effectively segment the defect objects from complex background and have poor real-time performance. To address these issues, we propose a novel saliency detection method based on Encoder–Decoder Residual network (EDRNet). In the encoder stage, we use a fully convolutional neural network to extract rich multilevel defect features and fuse the attention mechanism to accelerate the convergence of the model. Then in the decoder stage, we adopt the channels weighted block (CWB) and the residual decoder block (RDB) alternatively to integrate the spatial features of shallower layers and semantic features of deep layers and recover the predicted spatial saliency values step by step. Finally, we design the residual refinement structure with 1D filters (RRS_1D) to further optimize the coarse saliency map. Compared with the existing saliency detection methods, the deeply supervised EDRNet can accurately segment the complete defect objects with well-defined boundary and effectively filter out irrelevant background noise. The extensive experimental results prove that our method is consistently superior to the state-of-the-art methods with large margins and strong robustness, and the detection efficiency is at over 27 fps on a single GPU.

Salient object detection via effective background prior and novel graph

Salient object detection is getting more and more attention in computer vision field. In this paper, we propose a novel and effective framework for salient object detection. Firstly, we develop a robust background-based map by using spatial prior to remove the foreground noises of image boundary regions. The proposed background-based map and Objectness map are integrated to obtain a coarse saliency map. Then, an effective saliency propagation mechanism is utilized to further highlight salient object and suppress background region by defining a novel graph model, each node connects to its more similar neighbors and nodes with low saliency values in the proposed graph. As a result, the coarse saliency map is optimized to the refined saliency map by novel graph based saliency propagation. Finally, we construct a novel integration framework to further integrate two saliency maps for performance improvement. Experiments on three benchmark datasets are tested, experimental results show the superiority of the proposed algorithm than other state-of-the-art methods.

Saliency detection using adversarial learning networks

This paper proposes a novel model for saliency detection using the adversarial learning networks, in which the generator is used to generate the saliency map and the discriminator is deployed to guide the training process of overall network. Concretely, the training procedure of our model consists of three steps including the training of generator, the training of discriminator, and the training throughout the overall network. The key point of training process lies in the discriminator, which is designed to provide the feedback information for the acceleration of the generator and the refinement of saliency map. Therefore, during the training stage of overall network, the output of the generator, i.e. the coarse saliency map, is fed into the discriminator, yielding the corresponding feedback information. Following this way, we can obtain the final generator with a higher performance. For testing, the obtained generator is employed to perform saliency detection. Extensive experiments on four challenging saliency detection datasets show that our model not only achieves the favorable performance against the state-of-the-art saliency models, but also possesses the faster convergence speed when training the proposed model.

Residual Learning for Salient Object Detection.

Recent deep learning based salient object detection methods improve the performance by introducing multi-scale strategies into fully convolutional neural networks (FCNs). The final result is obtained by integrating all the predictions at each scale. However, the existing multi-scale based methods suffer from several problems: 1) it is difficult to directly learn discriminative features and filters to regress high-resolution saliency masks for each scale; 2) rescaling the multi-scale features could pull in many redundant and inaccurate values, and this weakens the representational ability of the network. In this paper, we propose a residual learning strategy and introduce to gradually refine the coarse prediction scale-by-scale. Concretely, instead of directly predicting the finest-resolution result at each scale, we learn to predict residuals to remedy the errors between coarse saliency map and scale-matching ground truth masks. We employ a Dilated Convolutional Pyramid Pooling (DCPP) module to generate the coarse prediction and guide the the residual learning process through several novel Attentional Residual Modules (ARMs). We name our network as Residual Refinement Network (R2Net). We demonstrate the effectiveness of the proposed method against other state-of-the-art algorithms on five released benchmark datasets. Our R2Net is a fully convolutional network which does not need any post-processing and achieves a real-time speed of 33 FPS when it is run on one GPU.

Attention-based fusion network for human eye-fixation prediction in 3D images

Human eye-fixation prediction in 3D images is important for many 3D applications, such as fine-grained 3D video object segmentation and intelligent bulletproof curtains. While the vast majority of existing 2D-based approaches cannot be applied, the main challenge lies in the inconsistency, or even conflict, between the RGB and depth saliency maps. In this paper, we propose a three-stream architecture to accurately predict human visual attention on 3D images end-to-end. First, a two-stream feature extraction network based on advanced convolutional neural networks is trained for RGB and depth, and hierarchical information is extracted from each ResNet-18. Then, these multi-level features are fed into the channel attention mechanism to suppress the feature space inconsistency and make the network focus on a significant target. The enhanced saliency map is fused step-by-step by VGG-16 to generate the final coarse saliency map. Finally, each coarse map is refined empirically through refinement blocks, and the network's own identification errors are corrected based on the acquired knowledge, thus converting the prediction saliency map from coarse to fine. The results of comparison of our model with six other state-of-the-art approaches on the NUS dataset (CC of 0.5579, KLDiv of 1.0903, AUC of 0.8339, and NSS of 2.3373) and the NCTU dataset (CC of 0.8614, KLDiv of 0.2681, AUC of 0.9143, and NSS of 2.3795) indicate that the proposed model consistently outperforms them by a considerable margin as it fully employs the channel attention mechanism.

Coarse-to-fine salient object detection with low-rank matrix recovery

Low-rank matrix recovery (LRMR) has recently been applied to saliency detection by decomposing image features into a low-rank component associated with background and a sparse component associated with visual salient regions. Despite its great potential, existing LRMR-based saliency detection methods seldom consider the inter-relationship among elements within these two components, thus are prone to generating scattered or incomplete saliency maps. In this paper, we introduce a novel and efficient LRMR-based saliency detection model under a coarse-to-fine framework to circumvent this limitation. First, we roughly measure the saliency of image regions with a baseline LRMR model that integrates a ℓ1-norm sparsity constraint and a Laplacian regularization smooth term. Given samples from the coarse saliency map, we then learn a projection that maps image features to refined saliency values, to significantly sharpen the object boundaries and to preserve the object entirety. We evaluate our framework against existing LRMR-based methods on three benchmark datasets. Experimental results validate the superiority of our method as well as the effectiveness of our suggested coarse-to-fine framework, especially for images containing multiple objects. MATLAB code for implementing our method is available at https://github.com/qizhust/HLRSaliency.

Salient object detection for RGB-D image by single stream recurrent convolution neural network

Salient object detection for RGB-D images aims to utilize color and depth information to automatically localize objects of human interest in the scene and reduce the complexity of visual analysis. Different from existing saliency detection model with double-stream network, salient object detection by Single Stream Recurrent Convolution Neural Network(SSRCNN) is proposed. First RGBD four-channels input is fed into VGG-16 net to generate multiple level features which express the most original feature for RGB-D image. The coarse saliency map from the deepest features can detect and localize salient objects, but loss the boundaries and subtle structures. So Depth Recurrent Convolution Neural Network (DRCNN) is then applied to each level feature for rendering salient object outline from deep to shallow hierarchically and progressively. With the help of deeper level feature, original depth cue and coarse saliency map, each level feature can accurately predict the salient objects in different scales. At last all the saliency maps from each level are fused together to generate final results. Extensive quantitative and qualitative experimental evaluations on four dataset demonstrate that the proposed method outperforms most state-of-the-art methods.

Salient object detection via reciprocal function filter

Salient object detection (SOD) has been attracting a lot of interest, and recently many computational models have been developed. Here, the authors formulate a SOD model, in which saliency map is computed as a combination of the colour, its distribution-based saliency and orientation saliency. Similar to traditional SODs, the proposed method is based on super-pixel segmentation and super-pixel utilises both colour and its distribution-based saliency to generate a coarse saliency map. However, distinct from traditional SODs, authors further use orientation contrast to optimise the coarse saliency map to obtain an improved saliency map. Authors’ contributions are twofold. First, the authors combine colour uniqueness and its distribution with local orientation information (LOI) used in Itti's model to effectively improve profiles of salient regions. Second, a reciprocal function is defined to substitute the Gabor function used in LOI, and the authors have proved that the substitution could detect relatively homogeneous and uniform regions at the boundary of salient object, whereas it is what the traditional models lack. Authors’ approach significantly outperforms state-of-the-art methods on four benchmark datasets, while the authors demonstrate that the proposed method runs as fast as most existing algorithms.

Language-aware weak supervision for salient object detection

Natural Language Processing has achieved remarkable performance in multitudinous computer tasks, but the potential capability of textual information has not been completely explored in visual saliency detection. In this paper, we learn to detect salient object from natural language by addressing the two essential issues: finding a semantic content matching the corresponding linguistic concept and recovering fine details without any pixel-level annotations. We first propose the Feature Matching Network (FMN) to explore the internal relation between the linguistic concept and visual image in the semantic space. The FMN simultaneously establishes the textual-visual pairwise affinities and generates a language-aware coarse saliency map. to refine the coarse map, the Recurrent Fine-tune Network (RFN) is proposed to enhance its predicted performance progressively by self-supervision. Our approach only leverages the caption to provide important cues of salient object, but generates a fine-detailed foreground map at a detecting speed of 72 FPS without any post-processing. Extensive experiments demonstrate that our method takes full advantage of textual information of natural language in saliency detection, and performs favorably against state-of-the-art approaches on the most existing datasets.

Localization of Small Neoplastic Lesions in Colonoscopy by Estimating Edge, Texture and Motion Saliency.

Early screening in Colorectal Cancer consists in finding and removing small precancerous masses or neoplastic lesions developed from the mucosa, usually lesions smaller than 10 mm. Localization of small neoplastic lesions is a very challenging task since colon exploration is highly dependent on the expert training and colon preparation. Several strategies have attempted to locate neoplasias, but usually they are huge lesions that a trained gastroenterologist could hardly miss. This work presents a saliency-based strategy to localize polypoid and non-polypoid neoplastic lesions smaller than 10 mm in colonoscopy videos by combining spatio-temporal descriptors. For doing so, a per-frame-multi-scale representation is computed and edge, texture and motion features are extracted. Each of these features is used to construct a primary saliency map which are then combined to obtain a coarse saliency map. Finally, the neoplasia is localized as the bounding box of a circular region, approximated by the Hough transform, with the largest salience. The proposed approach was evaluated in 8 short colonoscopy videos obtaining an average of Annotated Area Covered of 0.75 and a precision of 0.82.