Salient Object Detection Methods Research Articles

Studies Advanced in Salient Object Detection based on Deep Learning

A very important job in the area of computer vision is salient object detection, its goal is to check salient or obvious objects in images or videos. Thanks to rapidly developing networks of convolutive neurons, especially generative adversarial networks, CGAN has become a popular framework for salient object detection, which has promoted breakthroughs in detection precision and velocity of protruding objects. Focusing on the topic of salient object detection, in this paper, we systematically introduce the research progress of salient object detection. Specifically, we first introduce methods for detecting protracted objects based on general object detection algorithms (such as Fast R-CNN, YOLO, and SSD), including their basic ideas, key steps, advantages and disadvantages, etc. We then focus on CGAN-based salient object detection methods, including method classification, typical method introduction, and application examples. Finally, we quantitatively compare the performance of different saliency detection algorithms on common datasets, and discuss the existing research problems and feasible solutions of the topic.

Self-progress aggregate learning for weakly supervised salient object detection

The task of salient object detection aims to find the most salient object from the samples. In the field of weakly supervised learning, the existing weakly supervised salient object detection (WSSOD) methods often fail to utilize the limited label information (such as self-affinity or edge features, and scale transform) for learning. Therefore, this paper proposes a self-progress aggregate learning method named SPAL. First, the feature optimization scheme of the edge information module is put forward based on analysis of the problems existing in the current convolutional neural network for detection of the edge information of the object. Obviously, the salient object has a low requirement for high-level information, In particular, in order to improve the utilization rate of the network structure without increasing its complexity, the affinity global context is design in view of the particularity of the structure of a salient object. The structure of a salient object not only depends on the deep-level semantic feature information to a certain extent, but also has a certain guiding effect on the object position and edge information. Second, high-level affinity information is used to complement the slight-level edge information globally, and the scale attention module is adopted to guide the network to adapt the multi-scale reinforcement feature learning ability for the salient object regions. Our method SPAL achieved better experimental results than the other competitive models for comparison on five benchmark data sets (i.e. for DUTS-TEST, compared with CCFNet our method achieved an improvement of 0.6% for mean absolute error (MAE), 5.1% for Fb, and 1.1% for Eℰ ), which demonstrates the effectiveness of our proposed method.

TANet: Transformer‐based asymmetric network for RGB‐D salient object detection

AbstractExisting RGB‐D salient object detection methods mainly rely on a symmetric two‐stream Convolutional Neural Network (CNN)‐based network to extract RGB and depth channel features separately. However, there are two problems with the symmetric conventional network structure: first, the ability of CNN in learning global contexts is limited; second, the symmetric two‐stream structure ignores the inherent differences between modalities. In this study, a Transformer‐based asymmetric network is proposed to tackle the issues mentioned above. The authors employ the powerful feature extraction capability of Transformer to extract global semantic information from RGB data and design a lightweight CNN backbone to extract spatial structure information from depth data without pre‐training. The asymmetric hybrid encoder effectively reduces the number of parameters in the model while increasing speed without sacrificing performance. Then, a cross‐modal feature fusion module which enhances and fuses RGB and depth features with each other is designed. Finally, the authors add edge prediction as an auxiliary task and propose an edge enhancement module to generate sharper contours. Extensive experiments demonstrate that our method achieves superior performance over 14 state‐of‐the‐art RGB‐D methods on six public datasets. The code of the authors will be released at https://github.com/lc012463/TANet.

Camouflaged Insect Segmentation Using a Progressive Refinement Network

Accurately segmenting an insect from its original ecological image is the core technology restricting the accuracy and efficiency of automatic recognition. However, the performance of existing segmentation methods is unsatisfactory in insect images shot in wild backgrounds on account of challenges: various sizes, similar colors or textures to the surroundings, transparent body parts and vague outlines. These challenges of image segmentation are accentuated when dealing with camouflaged insects. Here, we developed an insect image segmentation method based on deep learning termed the progressive refinement network (PRNet), especially for camouflaged insects. Unlike existing insect segmentation methods, PRNet captures the possible scale and location of insects by extracting the contextual information of the image, and fuses comprehensive features to suppress distractors, thereby clearly segmenting insect outlines. Experimental results based on 1900 camouflaged insect images demonstrated that PRNet could effectively segment the camouflaged insects and achieved superior detection performance, with a mean absolute error of 3.2%, pixel-matching degree of 89.7%, structural similarity of 83.6%, and precision and recall error of 72%, which achieved improvements of 8.1%, 25.9%, 19.5%, and 35.8%, respectively, when compared to the recent salient object detection methods. As a foundational technology for insect detection, PRNet provides new opportunities for understanding insect camouflage, and also has the potential to lead to a step progress in the accuracy of the intelligent identification of general insects, and even being an ultimate insect detector.

Learning How to Detect Salient Objects in Nighttime Scenes

The detection of salient objects in nighttime scene settings is an essential research issue in computer vision. None of the known approaches can accurately anticipate salient objects in the nighttime scenes. Due to the lack of visible light, spatial visual information cannot be accurately perceived by traditional and deep network models. This paper proposed a Mountain Basin Network (MBNet) to identify salient objects for distinguishing the pixel-level saliency of low-light images. To improve the objects localizations and pixel classification performances, the proposed model incorporated a High-Low Feature Aggregation Module (HLFA) to synchronize the information from a high-level branch (named Bal-Net) and a low-level branch (called Mol-Net) to fuse the global and local context, and a Hierarchical Supervision Module (HSM) was embedded to aid in obtaining accurate salient objects, particularly the small ones. In addition, a multi-supervised integration technique was explored to optimize the structure and borders of salient objects. In the meantime, to facilitate more investigation into nighttime scenes and assessment of visual saliency models, we created a new nighttime dataset consisting of thirteen categories and a total of one thousand low-light images. Our experimental results demonstrated that the suggested MBNet model outperforms seven current state-of-the-art methods for salient object detection in nighttime scenes.

A Weakly Supervised Learning Framework for Salient Object Detection via Hybrid Labels

Fully-supervised salient object detection (SOD) methods have made great progress, but such methods often rely on a large number of pixel-level annotations, which are time-consuming and labour-intensive. In this paper, we focus on a new weakly-supervised SOD task under hybrid labels, where the supervision labels include a large number of coarse labels generated by the traditional unsupervised method and a small number of real labels. To address the issues of label noise and quantity imbalance in this task, we design a new pipeline framework with three sophisticated training strategies. In terms of model framework, we decouple the task into label refinement sub-task and salient object detection sub-task, which cooperate with each other and train alternately. Specifically, the R-Net is designed as a two-stream encoder-decoder model equipped with Blender with Guidance and Aggregation Mechanisms (BGA), aiming to rectify the coarse labels for more reliable pseudo-labels, while the S-Net is a replaceable SOD network supervised by the pseudo labels generated by the current R-Net. Note that, we only need to use the trained S-Net for testing. Moreover, in order to guarantee the effectiveness and efficiency of network training, we design three training strategies, including alternate iteration mechanism, group-wise incremental mechanism, and credibility verification mechanism. Experiments on five SOD benchmarks show that our method achieves competitive performance against weakly-supervised/unsupervised methods both qualitatively and quantitatively. The code and results can be found from the link of <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://rmcong.github.io/proj_Hybrid-Label-SOD.html</uri> .

Learning Salient Feature for Salient Object Detection Without Labels.

Supervised salient object detection (SOD) methods achieve state-of-the-art performance by relying on human-annotated saliency maps, while unsupervised methods attempt to achieve SOD by not using any annotations. In unsupervised SOD, how to obtain saliency in a completely unsupervised manner is a huge challenge. Existing unsupervised methods usually gain saliency by introducing other handcrafted feature-based saliency methods. In general, the location information of salient objects is included in the feature maps. If the features belonging to salient objects are called salient features and the features that do not belong to salient objects, such as background, are called nonsalient features, by dividing the feature maps into salient features and nonsalient features in an unsupervised way, then the object at the location of the salient feature is the salient object. Based on the above motivation, a novel method called learning salient feature (LSF) is proposed, which achieves unsupervised SOD by LSF from the data itself. This method takes enhancing salient feature and suppressing nonsalient features as the objective. Furthermore, a salient object localization method is proposed to roughly locate objects where the salient feature is located, so as to obtain the salient activation map. Usually, the object in the salient activation map is incomplete and contains a lot of noise. To address this issue, a saliency map update strategy is introduced to gradually remove noise and strengthen boundaries. The visualization of images and their salient activation maps show that our method can effectively learn salient visual objects. Experiments show that we achieve superior unsupervised performance on a series of datasets.

Saliency Detection Based on Low-Level and High-Level Features via Manifold-Space Ranking

Saliency detection as an active research direction in image understanding and analysis has been studied extensively. In this paper, to improve the accuracy of saliency detection, we propose an efficient unsupervised salient object detection method. The first step of our method is that we extract local low-level features of each superpixel after segmenting the image into different scale parts, which helps to locate the approximate locations of salient objects. Then, we use convolutional neural networks to extract high-level, semantically rich features as complementary features of each superpixel, and low-level features, as well as high-level features of each superpixel, are incorporated into a new feature vector to measure the distance between different superpixels. The last step is that we use a manifold space-ranking method to calculate the saliency of each superpixel. Extensive experiments over four challenging datasets indicate that the proposed method surpasses state-of-the-art methods and is closer to the ground truth.

MENet: Lightweight multimodality enhancement network for detecting salient objects in RGB-thermal images

Most red–green–blue and thermal (RGB-T) salient object detection methods require high memory consumption and incur large computational costs, which limit their applicability. To alleviate the computational resource requirements, we propose a lightweight multimodality enhancement network (MENet) for RGB-T salient object detection with relatively few parameters. As the RGB and thermal images are from different domains, the modality gap leads to unsatisfactory results if using simple feature concatenation. Instead, we introduce a multimodality complementary enhancement module with a nested residual structure to fuse RGB and thermal features. During decoding, we design a recursive sharpening module that is inspired by atrous spatial pyramid pooling and dense connections and has fewer parameters than similar methods. By reducing the number of parameters and computations of each component, MENet ends up having only 21.26 M parameters and running with 2.501G floating-point operations. Experiments were performed on three benchmark datasets. The results show that the proposed MENet can consistently outperform 16 state-of-the-art methods.

Deep learning-based marine big data fusion for ocean environment monitoring: Towards shape optimization and salient objects detection

ObjectiveDuring the last few years, underwater object detection and marine resource utilization have gained significant attention from researchers and become active research hotspots in underwater image processing and analysis domains. This research study presents a data fusion-based method for underwater salient object detection and ocean environment monitoring by utilizing a deep model.MethodologyA hybrid model consists of an upgraded AlexNet with Inception v-4 for salient object detection and ocean environment monitoring. For the categorization of spatial data, AlexNet is utilized, whereas Inception V-4 is employed for temporal data (environment monitoring). Moreover, we used preprocessing techniques before the classification task for underwater image enhancement, segmentation, noise and fog removal, restoration, and color constancy.ConclusionThe Real-Time Underwater Image Enhancement (RUIE) dataset and the Marine Underwater Environment Database (MUED) dataset are used in this research project’s data fusion and experimental activities, respectively. Root mean square error (RMSE), computing usage, and accuracy are used to construct the model’s simulation results. The suggested model’s relevance form optimization and conspicuous item prediction issues in the seas is illustrated by the greatest accuracy of 95.7% and low RMSE value of 49 when compared to other baseline models.

Lightweight Salient Object Detection in Optical Remote-Sensing Images via Semantic Matching and Edge Alignment

Recently, relying on convolutional neural networks (CNNs), many methods for salient object detection in optical remote-sensing images (ORSI-SOD) are proposed. However, most methods ignore the number of parameters and computational cost brought by CNNs, and only a few pay attention to portability and mobility. To facilitate practical applications, in this article, we propose a novel lightweight network for ORSI-SOD based on semantic matching and edge alignment, termed SeaNet. Specifically, SeaNet includes a lightweight MobileNet-V2 for feature extraction, a dynamic semantic matching module (DSMM) for high-level features, an edge self-alignment module (ESAM) for low-level features, and a portable decoder for inference. First, the high-level features are compressed into semantic kernels. Then, semantic kernels are used to activate salient object locations in two groups of high-level features through dynamic convolution operations in DSMM. Meanwhile, in ESAM, cross-scale edge information extracted from two groups of low-level features is self-aligned through <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$L_{2}$ </tex-math></inline-formula> loss and used for detail enhancement. Finally, starting from the highest level features, the decoder infers salient objects based on the accurate locations and fine details contained in the outputs of the two modules. Extensive experiments on two public datasets demonstrate that our lightweight SeaNet not only outperforms most state-of-the-art lightweight methods, but also yields comparable accuracy with state-of-the-art conventional methods, while having only 2.76 M parameters and running with 1.7 G floating point operations (FLOPs) for <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$288 \times 288$ </tex-math></inline-formula> inputs. Our code and results are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/MathLee/SeaNet</uri> .

Salient Object Detection in Optical Remote Sensing Images Driven by Transformer.

Existing methods for Salient Object Detection in Optical Remote Sensing Images (ORSI-SOD) mainly adopt Convolutional Neural Networks (CNNs) as the backbone, such as VGG and ResNet. Since CNNs can only extract features within certain receptive fields, most ORSI-SOD methods generally follow the local-to-contextual paradigm. In this paper, we propose a novel Global Extraction Local Exploration Network (GeleNet) for ORSI-SOD following the global-to-local paradigm. Specifically, GeleNet first adopts a transformer backbone to generate four-level feature embeddings with global long-range dependencies. Then, GeleNet employs a Direction-aware Shuffle Weighted Spatial Attention Module (D-SWSAM) and its simplified version (SWSAM) to enhance local interactions, and a Knowledge Transfer Module (KTM) to further enhance cross-level contextual interactions. D-SWSAM comprehensively perceives the orientation information in the lowest-level features through directional convolutions to adapt to various orientations of salient objects in ORSIs, and effectively enhances the details of salient objects with an improved attention mechanism. SWSAM discards the direction-aware part of D-SWSAM to focus on localizing salient objects in the highest-level features. KTM models the contextual correlation knowledge of two middle-level features of different scales based on the self-attention mechanism, and transfers the knowledge to the raw features to generate more discriminative features. Finally, a saliency predictor is used to generate the saliency map based on the outputs of the above three modules. Extensive experiments on three public datasets demonstrate that the proposed GeleNet outperforms relevant state-of-the-art methods. The code and results of our method are available at https://github.com/MathLee/GeleNet.

TCRNet: A Trifurcated Cascaded Refinement Network for Salient Object Detection

Due to the rapid development of deep learning, salient object detection has achieved promising results in recent years. However, how to construct a powerful saliency detection network to generate saliency maps with completely highlighted salient objects and effectively suppressed background noise still remains an open and challenging problem. In this paper, we propose a novel trifurcated cascaded refinement network (TCRNet) to explore multi-level feature fusion and global information representation. Specifically, all the side-output features from the backbone network are divided into three branches (i.e., global information, deep information and shallow information) according to feature similarities, and then the global contexts, semantic knowledge and fined details respectively generated by the three branches are integrated in a cascaded refinement way. This refinement strategy aims to guide the feature learning of important regions and suppress background noises. Moreover, we propose a cross-level feature fusion (CFF) module to fuse multi-level similar features within a same branch by exploring their interrelationship, thus capturing the important semantic information and structure details. To this end, the complementarity of similar features is utilized to refine feature at each level by sharing an encoded spatial weight map. Besides, we design a context-aware feature learning (CFL) module to capture global contexts in a single layer by taking account of both single-feature representation and multi-feature interaction, thus providing important location information of salient objects. Extensive experiments on five benchmarks are conducted to demonstrate that the proposed network TCRNet performs favorably against 20 state-of-the-art salient object detection methods on five benchmark datasets, indicating the effectiveness and superiority of our method.

PANet: Patch-Aware Network for Light Field Salient Object Detection.

Most existing light field saliency detection methods have achieved great success by exploiting unique light field data-focus information in focal slices. However, they process light field data in a slicewise way, leading to suboptimal results because the relative contribution of different regions in focal slices is ignored. How we can comprehensively explore and integrate focused saliency regions that would positively contribute to accurate saliency detection. Answering this question inspires us to develop a new insight. In this article, we propose a patch-aware network to explore light field data in a regionwise way. First, we excavate focused salient regions with a proposed multisource learning module (MSLM), which generates a filtering strategy for integration followed by three guidances based on saliency, boundary, and position. Second, we design a sharpness recognition module (SRM) to refine and update this strategy and perform feature integration. With our proposed MSLM and SRM, we can obtain more accurate and complete saliency maps. Comprehensive experiments on three benchmark datasets prove that our proposed method achieves competitive performance over 2-D, 3-D, and 4-D salient object detection methods. The code and results of our method are available at https://github.com/OIPLab-DUT/IEEE-TCYB-PANet.

Rethinking Lightweight Salient Object Detection via Network Depth-Width Tradeoff.

Existing salient object detection methods often adopt deeper and wider networks for better performance, resulting in heavy computational burden and slow inference speed. This inspires us to rethink saliency detection to achieve a favorable balance between efficiency and accuracy. To this end, we design a lightweight framework while maintaining satisfying competitive accuracy. Specifically, we propose a novel trilateral decoder framework by decoupling the U-shape structure into three complementary branches, which are devised to confront the dilution of semantic context, loss of spatial structure and absence of boundary detail, respectively. Along with the fusion of three branches, the coarse segmentation results are gradually refined in structure details and boundary quality. Without adding additional learnable parameters, we further propose Scale-Adaptive Pooling Module to obtain multi-scale receptive field. In particular, on the premise of inheriting this framework, we rethink the relationship among accuracy, parameters and speed via network depth-width tradeoff. With these insightful considerations, we comprehensively design shallower and narrower models to explore the maximum potential of lightweight SOD. Our models are proposed for different application environments: 1) a tiny version CTD-S (1.7M, 125FPS) for resource constrained devices, 2) a fast version CTD-M (12.6M, 158FPS) for speed-demanding scenarios, 3) a standard version CTD-L (26.5M, 84FPS) for high-performance platforms. Extensive experiments validate the superiority of our method, which achieves better efficiency-accuracy balance across five benchmarks.

Self-Sufficient Feature Enhancing Networks for Video Salient Object Detection

Detecting salient objects in videos is a very challenging task. Current state-of-the-art methods are dominated by motion based deep neural networks, among which optical flow is often leveraged as motion representation. Though with robust performance, these optical flow-based video salient object detection methods face at least two problems that may hinder their generalization and application. First, computing optical flow as a pre-processing step does not support direct end-to-end learning; second, little attention has been given to the quality of visual features due to high computational cost of spatiotemporal feature encoding. In this paper we propose a novel self-sufficient feature enhancing network (SFENet) for video salient object detection, which leverages optical flow estimation as an auxiliary task while being end-to-end trainable. With a joint training scheme of both salient object detection and optical flow estimation, its multi-task architecture can be totally self-sufficient for achieving good performance without any pre-processing. Furthermore, for improving feature quality, we design four lightweight modules in spatial and temporal domains, including cross-layer fusion, multi-level warping, spatial-channel attention and boundary-aware refinement. The proposed method is evaluated through extensive experiments on five video salient object detection datasets. Experimental results show that our SFENet can be easily trained with fast convergence speed. It significantly outperforms previous methods in terms of various evaluation metrics. Moreover, with optical flow estimation and unsupervised video object segmentation as example applications, our method also yields state-of-the-art results on standard datasets.

CAVER: Cross-Modal View-Mixed Transformer for Bi-Modal Salient Object Detection.

Most of the existing bi-modal (RGB-D and RGB-T) salient object detection methods utilize the convolution operation and construct complex interweave fusion structures to achieve cross-modal information integration. The inherent local connectivity of the convolution operation constrains the performance of the convolution-based methods to a ceiling. In this work, we rethink these tasks from the perspective of global information alignment and transformation. Specifically, the proposed cross-modal view-mixed transformer (CAVER) cascades several cross-modal integration units to construct a top-down transformer-based information propagation path. CAVER treats the multi-scale and multi-modal feature integration as a sequence-to-sequence context propagation and update process built on a novel view-mixed attention mechanism. Besides, considering the quadratic complexity w.r.t. the number of input tokens, we design a parameter-free patch-wise token re-embedding strategy to simplify operations. Extensive experimental results on RGB-D and RGB-T SOD datasets demonstrate that such a simple two-stream encoder-decoder framework can surpass recent state-of-the-art methods when it is equipped with the proposed components.

Salient object detection method based on multi-scale feature-fusion guided by edge information

Salient object detection method based on multi-scale feature-fusion guided by edge information

RGB-D Salient Object Detection Method Based on Multi-Modal Fusion and Contour Guidance

RGB-D Salient Object Detection Method Based on Multi-Modal Fusion and Contour Guidance

View-Aware Salient Object Detection for $360^{\circ }$ Omnidirectional Image

Image-based salient object detection (ISOD) in <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> scenarios is significant for understanding and applying panoramic information. However, research on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> ISOD has not been widely explored due to the lack of large, complex, high-resolution, and well-labeled datasets. Towards this end, we construct a large scale <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> ISOD dataset with object-level pixel-wise annotation on equirectangular projection (ERP), which contains rich panoramic scenes with not less than 2K resolution and is the largest dataset for <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> ISOD by far to our best knowledge. By observing the data, we find current methods face three significant challenges in panoramic scenarios: diverse distortion degrees, discontinuous edge effects and changeable object scales. Inspired by humans' observing process, we propose a view-aware salient object detection method based on a Sample Adaptive View Transformer (SAVT) module with two sub-modules to mitigate these issues. Specifically, the sub-module View Transformer (VT) contains three transform branches based on different kinds of transformations to learn various features under different views and heighten the model's feature toleration of distortion, edge effects and object scales. Moreover, the sub-module Sample Adaptive Fusion (SAF) is to adjust the weights of different transform branches based on various sample features and make transformed enhanced features fuse more appropriately. The benchmark results of 20 state-of-the-art ISOD methods reveal the constructed dataset is very challenging. Moreover, exhaustive experiments verify the proposed approach is practical and outperforms the state-of-the-art methods.