Image Salient Object Detection Research Articles

Motion-Aware Memory Network for Fast Video Salient Object Detection.

Previous methods based on 3DCNN, convLSTM, or optical flow have achieved great success in video salient object detection (VSOD). However, these methods still suffer from high computational costs or poor quality of the generated saliency maps. To address this, we design a space-time memory (STM)-based network that employs a standard encoder-decoder architecture. During the encoding stage, we extract high-level temporal features from the current frame and its adjacent frames, which is more efficient and practical than methods reliant on optical flow. During the decoding stage, we introduce an effective fusion strategy for both spatial and temporal branches. The semantic information of the high-level features is used to improve the object details in the low-level features. Subsequently, spatiotemporal features are methodically derived step by step to reconstruct the saliency maps. Moreover, inspired by the boundary supervision prevalent in image salient object detection (ISOD), we design a motion-aware loss that predicts object boundary motion, and simultaneously perform multitask learning for VSOD and object motion prediction. This can further enhance the model's capability to accurately extract spatiotemporal features while maintaining object integrity. Extensive experiments on several datasets demonstrate the effectiveness of our method and can achieve state-of-the-art metrics on some datasets. Our proposed model does not require optical flow or additional preprocessing, and can reach an impressive inference speed of nearly 100 FPS.

Lightweight Progressive Multilevel Feature Collaborative Network for Remote Sensing Image Salient Object Detection

Lightweight Progressive Multilevel Feature Collaborative Network for Remote Sensing Image Salient Object Detection

UTDNet: A unified triplet decoder network for multimodal salient object detection

Image Salient Object Detection (SOD) is a fundamental research topic in the area of computer vision. Recently, the multimodal information in RGB, Depth (D), and Thermal (T) modalities has been proven to be beneficial to the SOD. However, existing methods are only designed for RGB-D or RGB-T SOD, which may limit the utilization in various modalities, or just finetuned on specific datasets, which may bring about extra computation overhead. These defects can hinder the practical deployment of SOD in real-world applications. In this paper, we propose an end-to-end Unified Triplet Decoder Network, dubbed UTDNet, for both RGB-T and RGB-D SOD tasks. The intractable challenges for the unified multimodal SOD are mainly two-fold, i.e., (1) accurately detecting and segmenting salient objects, and (2) preferably via a single network that fits both RGB-T and RGB-D SOD. First, to deal with the former challenge, we propose the multi-scale feature extraction unit to enrich the discriminative contextual information, and the efficient fusion module to explore cross-modality complementary information. Then, the multimodal features are fed to the triplet decoder, where the hierarchical deep supervision loss further enable the network to capture distinctive saliency cues. Second, as to the latter challenge, we propose a simple yet effective continual learning method to unify multimodal SOD. Concretely, we sequentially train multimodal SOD tasks by applying Elastic Weight Consolidation (EWC) regularization with the hierarchical loss function to avoid catastrophic forgetting without inducing more parameters. Critically, the triplet decoder separates task-specific and task-invariant information, making the network easily adaptable to multimodal SOD tasks. Extensive comparisons with 26 recently proposed RGB-T and RGB-D SOD methods demonstrate the superiority of the proposed UTDNet.

Transformer-based hierarchical dynamic decoders for salient object detection

Global context and global contrast are crucial clues for Salient Object Detection (SOD) in images. Most advanced SOD methods exploit CNN-based architectures, achieving impressive results. However, these methods have intrinsic limitations in capturing long-range global information since a CNN extracts feature in local sliding windows. In contrast, transformers exploit a self-attention mechanism to extract features, gaining a powerful capability of learning global cues. Nonetheless, a pure transformer-based network consumes a large computational overhead and easily suffers from attention collapse, as it goes deeper. To address this issue, in this paper, we propose a Transformer-based Hierarchical Dynamic Decoder (T-HDDNet) for image salient object detection. Specifically, our T-HDDNet employs the transformer to encode each image patch into multi-level and multi-resolution features based on the long-range dependencies among pixels. To obtain an accurate saliency map of high resolution, we develop a dynamic dual upsampling mechanism to enlarge feature spatial size in a data-driven manner, together with a dynamic feature fusion unit. Ultimately, the hierarchical dynamic decoders built on the basis of these two units are used to attain the final saliency progressively. Extensive experimental results show that the proposed method achieves the best performance on all benchmarks, in comparison with state-of-the-art technologies.

Lightweight Distortion-Aware Network for Salient Object Detection in Omnidirectional Images

Compared with 2D image salient object detection (SOD), SOD in omnidirectional images (or 360° images) usually suffers from geometric distortion. Although existing omnidirectional image SOD (ODI-SOD) methods have improved the detection accuracy obviously, their application may be cumbersome in real scenes due to their high computational cost. To avoid distortion and reduce the computational cost simultaneously in ODI-SOD, we propose a novel lightweight distortion-aware network, named LDNet, in this paper. First, to extract features with less distortion from ODIs, we integrate the distortion-aware convolution and depth-wise separable convolution (DSConv) into distortion-aware DSConv (DDSConv) and replace the regular convolutions in the last two blocks of the ResNet-18 with DDSConvs to obtain our lightweight backbone network (LD-ResNet-18). To enhance spatial information in each channel of the extracted features at each level comprehensively, then, we propose a lightweight distortion-aware channel-wise enhancement (DCE) module (only 0.05M parameters) including DDSConvs with various dilation rates, channel shuffle operation and attention mechanism, and employ a high-to-low dense connection structure to modulate the enhanced multi-level features. Besides, we design a distortion-aware self-correlation (DSC) module (only 0.02M parameters) for mining the contextual dependency of the features via a coarse-fine strategy, and the correlated features are refined by DCE modules and integrated by another dense connection structure. The final saliency map is predicted from the densely integrated features. Compared with 12 state-of-the-art methods on two public datasets, our lightweight LDNet achieves competitive or even better performance with only 2.9M parameters and 3.4G FLOPs, which balances the efficiency and performance.

Printing roller image salient object detection based on multi-scale feature fusion

In this study, aiming at the problem that the surface defect image of the printing roller has different sizes and a large number of defects, a salient object detection method based on stepwise multi-scale feature fusion is proposed, which consists of a stepwise multi-scale feature guidance module, a multi-resolution information fusion module, and a deeply supervised side output. Specifically, the stepwise multi-scale feature guidance module contains two paths, top-down and bottom-up, which strengthen the transmission of multi-scale features in the network, and fuse multi-level features through the multi-resolution information fusion module to suppress the interference of background noise and redundant features. In addition, deeply supervised side outputs are used to eliminate differences caused by background and foreground imbalance. The experimental results on the roller surface defect dataset show that the method can achieve Fmax, MAE of 0.7959, 0.0144, and the detection rate can reach 24.98fps, which has better detection performance than other algorithms.

Salient object detection in low-light images via functional optimization-inspired feature polishing

Salient object detection under low-light conditions remains a challenge in many practical applications. Most recent works focus on feature integration without considering filtering out clutter from the darkness, thus limiting the detection performance. To tackle this issue, we propose a low-light image salient object detection network (LLISOD), which generates highly accurate saliency maps by functional optimization-inspired feature polishing strategy. The LLISOD includes: (1) An unfolded implicit nonlinear mapping (UINM) module uniquely designed for polishing feature maps; and (2) the hierarchical feature polishing (HFP) streams proposed for fusing the outputs of the UINM module on the top-down pathway to refine the saliency predictions. Furthermore, we provide a new dataset for benchmarking the investigation of salient object detection in low-light images. Extensive experiments demonstrate that our method outperforms existing state-of-the-art approaches. Code will be available at https://github.com/yuehuihui000/LLISOD.

Bifurcation Fusion Network for RGB-D Salient Object Detection

With the rapid development of sensor technology, multi-modal data fusion methods based on deep neural networks provide a reliable guarantee for object recognition and detection in complex scenarios. Most of the existing RGB-D image salient object detection methods improve the salient object detection methods in 2D scenes, which have many problems, such as ineffective fusion of feature information, incomplete image feature extraction and so on. Aiming at these problems, we proposed a salient object detection method based on bifurcated fusion network. First, we use the high- level features of the global context to locate the salient object, then we use the low-level features of the local details to extract the edge information. Second, we model the RGB information and depth information through the gate channel transformation control mechanism, and construct the bifurcation backbone network model and generate the initial salient map. Finally, based on the initial salient map and low-level features, the final salient map is generated. Experimental results show that the proposed method can fully utilize the multi-layer feature information of the image and effectively detect the salient object.

Deeper Look at Image Salient Object Detection: Bi-Stream Network With a Small Training Dataset

Compared with the conventional hand-crafted approaches, the deep learning based ISOD (image salient object detection) models have achieved tremendous performance improvements by training exquisitely crafted fancy networks over large-scale training sets. However, do we really need large-scale training set for ISOD? In this article, we provide a deeper insight into the interrelationship between the ISOD performance and the training data. To alleviate the conventional demands for large-scale training data, we provide a feasible way to construct a novel small-scale training set, which only contains 4 K images. To take full advantage of this new set, we propose a novel bi-stream network consisting of two different feature backbones. Benefit from the proposed gate control unit, this bi-stream network is able to achieve complementary fusion status for its subbranches. To our best knowledge, this is the first attempt to use a small-scale training set to compete with other large-scale ones; nevertheless, our method can still achieve the leading SOTA performance on all tested benchmark datasets. Both the code and dataset are publicly available at <uri>https://github.com/wuzhenyubuaa/TSNet</uri>.

Two-Stage Edge Reuse Network for Salient Object Detection of Strip Steel Surface Defects

Recently, researchers have paid more attention to salient object detection of strip steel surface defects, but there are lots of difficulties, such as a mass of noise, blurry defect boundaries, complex backgrounds, and various types of defects. Existing image salient object detection methods fail to overcome the above challenging scenes. To address them, in this paper, we propose a novel Two-Stage Edge Reuse Network (TSERNet), which consists of two stages, <i>i.e</i>., prediction and refinement. In the first prediction stage, we construct a primary net based on the encoder-decoder architecture. The encoder not only extracts multi-scale features but also generates an edge map. In the decoder, we propose a novel Edge-aware Foreground-Background Integration (EFBI) module to distinguish foreground and background through edge features and reverse attention mechanism, and exploit the decoder to generate an initial saliency map. In the second refinement stage, we construct a sub-net based on the same architecture as that of the first stage. Its encoder extracts features from the initial saliency map, and its decoder deploys the Edge-aware Refinement Module (ERM), which reuses the edge map generated from the first stage, to enhance these features for purifying the initial saliency map, resultting in the final saliency map. Comprehensive experiments on the public dataset show that our proposed TSERNet is consistently superior to 22 relevant state-of-the-art methods. The code and results of our method are available at https://github.com/monxxcn/TSERNet.

Salient object detection in hyperspectral images using deep background reconstruction based anomaly detection

ABSTRACT Salient object detection is a significant task that forms the basis for many image processing and computer vision applications. In recent years, the research in this area is being extended beyond RGB images to applications involving multispectral and hyperspectral images. But most of the existing algorithms for salient object detection often yield an incomplete representation of the object and often produce saliency maps with blurred edges. In this paper, we propose an efficient hyperspectral image salient object detection method through anomaly detection by combining deep learning autoencoders with one-class support vector machines. Here, the saliency detection problem in hyperspectral images is formulated as an unsupervised deep background spectral reconstruction-based anomaly detection. Our proposed method first employs deep autoencoders to model the background of an input hyperspectral image in terms of spectral reconstruction residuals of the autoencoders and then detect the salient objects from the image through a one-class support vector machine-based anomaly detection. The proposed method was evaluated on a publicly available hyperspectral image dataset for salient object detection. The experimental results show that our proposed method is found to be more efficient and superior over other previous methods in terms of various performance measures.

Spatial context-aware network for salient object detection

Salient Object Detection (SOD) is a fundamental problem in the field of computer vision. This paper presents a novel Spatial Context-Aware Network (SCA-Net) for SOD in images. Compared with other recent deep learning based SOD algorithms, SCA-Net can more effectively aggregate multi-level deep features. A Long-Path Context Module (LPCM) is employed to grant better discrimination ability to feature maps that incorporate coarse global information. Consequently, a more accurate initial saliency map can be obtained to facilitate subsequent predictions. SCA-Net also adopts a Short-Path Context Module (SPCM) to progressively enforce the interaction between local contextual cues and global features. Extensive experiments on five large-scale benchmarks demonstrate that SCA-Net achieves favorable performance against very recent state-of-the-art algorithms.

Rethinking Image Salient Object Detection: Object-Level Semantic Saliency Reranking First, Pixelwise Saliency Refinement Later.

Human attention is an interactive activity between our visual system and our brain, using both low-level visual stimulus and high-level semantic information. Previous image salient object detection (SOD) studies conduct their saliency predictions via a multitask methodology in which pixelwise saliency regression and segmentation-like saliency refinement are conducted simultaneously. However, this multitask methodology has one critical limitation: the semantic information embedded in feature backbones might be degenerated during the training process. Our visual attention is determined mainly by semantic information, which is evidenced by our tendency to pay more attention to semantically salient regions even if these regions are not the most perceptually salient at first glance. This fact clearly contradicts the widely used multitask methodology mentioned above. To address this issue, this paper divides the SOD problem into two sequential steps. First, we devise a lightweight, weakly supervised deep network to coarsely locate the semantically salient regions. Next, as a postprocessing refinement, we selectively fuse multiple off-the-shelf deep models on the semantically salient regions identified by the previous step to formulate a pixelwise saliency map. Compared with the state-of-the-art (SOTA) models that focus on learning the pixelwise saliency in single images using only perceptual clues, our method aims at investigating the object-level semantic ranks between multiple images, of which the methodology is more consistent with the human attention mechanism. Our method is simple yet effective, and it is the first attempt to consider salient object detection as mainly an object-level semantic reranking problem.

Depth quality-aware selective saliency fusion for RGB-D image salient object detection

Previous RGB-D salient object detection (SOD) methods have widely adopted the deep learning tools to automatically strike a trade-off between RGB and depth (D). The key rationale is to take full advantage of the complementary nature between RGB and D, aiming for a much-improved SOD performance than that of using either of them solely. However, because to the D quality itself usually varies from scene to scene, such fully automatic fusion schemes may not always be helpful for the SOD task. Moreover, as an objective factor, the D quality has long been overlooked by previous work. Thus, this paper proposes a simple yet effective scheme to measure D quality in advance. The key idea is to devise a series of features in accordance with the common attributes of the high-quality D regions. To be more concrete, we advocate to conduct D quality assessments following a multi-scale methodology, which includes low-level edge consistency, mid-level regional uncertainty and high-level model variance. All these components will be computed independently and later be combined with RGB and D saliency cues to guide the selective RGBD fusion. Compared with the SOTA fusion schemes, our method can achieve better fusion result between RGB and D. Specifically, the proposed D quality measurement method is able to achieve steady performance improvements for almost 2.0% averagely.

Image Salient Object Detection Based on K-means and Level Set Superpixel Segmentation

In order to detect and segment the salient objects in digital images to solve complex machine visual problems, this paper proposes an image salient object detection algorithm based on K-means and level set superpixel segmentation. The algorithm segments a given target image into multiple superpixel regions with similar features to abstract unnecessary details in the images, and reduce the number of colors in all superpixels by Histogram acceleration to increase computational efficiency. The saliency map obtained by calculating the distance of all superpixel in the Lab color space, and optimizes the detection effect by background prior and multi-scale spatial fusion. A large number of experiments show that the saliency detection method proposed in this paper is superior to algorithms such as SR, AC, FT, MSS, LC, CA, SF, HC and RC in accuracy, recall rate, AUC value and average absolute error.

CmSalGAN: RGB-D Salient Object Detection With Cross-View Generative Adversarial Networks

Image salient object detection (SOD) is an active research topic in computer vision and multimedia area. Fusing complementary information of RGB and depth has been demonstrated to be effective for image salient object detection which is known as RGB-D salient object detection problem. The main challenge for RGB-D salient object detection is how to exploit the salient cues of both intra-modality (RGB, depth) and cross-modality simultaneously which is known as cross-modality detection problem. In this paper, we tackle this challenge by designing a novel cross-modality Saliency Generative Adversarial Network ( cm SalGAN). cm SalGAN aims to learn an optimal view-invariant and consistent pixel-level representation for RGB and depth images via a novel adversarial learning framework, which thus incorporates both information of intra-view and correlation information of cross-view images simultaneously for RGB-D saliency detection problem. To further improve the detection results, the attention mechanism and edge detection module are also incorporated into cm SalGAN. The entire cm SalGAN can be trained in an end-to-end manner by using the standard deep neural network framework. Experimental results show that cm SalGAN achieves the new state-of-the-art RGB-D saliency detection performance on several benchmark datasets.

Optimisation for image salient object detection based on semantic‐aware clustering and CRF

State‐of‐the‐art optimisation methods for salient object detection neglect that saliency maps of different images usually show different imperfections. Therefore, the saliency maps of some images cannot achieve effective optimisation. Based on the observation that the saliency maps of semantically similar images usually show similar imperfections, the authors propose an optimisation method for salient object detection based on semantic‐aware clustering and conditional random field (CRF), named CCRF. They first cluster the training images into some clusters using the image semantic features extracted by using a deep convolutional neural network model for image classification. Then for each cluster, they use a CRF to optimise the saliency maps generated by existing salient object detection methods. A grid search method is used to compute the optimal weights of the kernels of the CRF. The saliency maps of the testing images are optimised by the corresponding CRFs with the optimal weights. The experimental results with 13 typical salient object detection methods on four datasets show that the proposed CCRF algorithm can effectively improve the results of a variety of image salient object detection methods and outperforms the compared optimisation methods.

Video salient object detection via spatiotemporal attention neural networks

Recently, deep convolutional neural networks have been widely introduced into image salient object detection and achieve good performance in this community. However, as the complexity of video scenes, video salient object detection with deep learning models is still a challenge. The specific difficulties come from two aspects. First of all, the deep networks on image saliency detection cannot capture robust motion cues in video sequences. Secondly, as for the spatiotemporal fusing features, the existing methods simply exploit element-wise addition or concatenation, which not fully explores the contextual information and complementary correlation, thus they cannot produce more robust spatiotemporal features. To address these issues, we propose a two-stream based spatiotemporal attention neural network (STAN) for video salient object detection. We amply extract the motion information in terms of long short term memory (LSTM) network and 3D convolutional operation from optical flow-based prior and video sequences. Moreover, an attentive module is designed to integrate the different types of spatiotemporal feature maps by learning the corresponding weights. Meanwhile, in order to generate sufficient pixel-wise annotated video frames, we manually generate lots of coarse labels, which are well utilized to train a robust saliency prediction network. Experiments on the widely used challenging datasets (e.g., FBMS and DAVIS) prove that the proposed STAN has competitive performances among salient object detection methods.

结合深度学习的图像显著目标检测

结合深度学习的图像显著目标检测

Image salient object detection with refined deep features via convolution neural network

Recent advances in saliency detection have used deep learning to obtain high-level features to detect salient regions. These advances have demonstrated superior results over previous works that use handcrafted low-level features for saliency detection. We propose a convolutional neural network (CNN) model to learn high-level features for saliency detection. Compared to other methods, our method presents two merits. First, when performing features extraction, apart from the convolution and pooling step in our method, we add restricted Boltzmann machine into the CNN framework to obtain more accurate features in intermediate step. Second, in order to avoid manual annotation data, we add deep belief network classifier at the end of this model to classify salient and nonsalient regions. Quantitative and qualitative experiments on three benchmark datasets demonstrate that our method performs favorably against the state-of-the-art methods.