Salient Object Detection Methods Research Articles

Salient object detection with bayesian inference based on radar and camera fusion used in UAV obstacle avoidance

Most existing salient object detection methods are sensitive to background noise and rely on prior information in UAV obstacle avoidance applications despite detection methods witnessing rapid progress. In this paper, we propose an efficient framework for salient object detection based on radar-camera fusion and iterative Bayesian optimization. A rough salient object (RSO) image is first built through radar and camera calibration. Next, the RSO image is used to calculate the edge response based on the receptive field mechanism of the primary visual cortex to construct the contour image. Finally, the above two images and the 2D Gaussian distribution are jointly integrated into an iterative Bayesian optimization scheme to get the final salient object image. Different from typical detection methods, this method suppresses background noise by filtering out irrelevant pixels using fusion information. The Bayesian inference framework’s detection performance is improved by precise spatial prior, consisting of optimized contour and RSO images. Experimental results indicate that the presented algorithm performs well against state-of-the-art salient object detection methods on several reference datasets in different evaluation metrics.

Multi-Prior Driven Network for RGB-D Salient Object Detection

Most existing RGB-D salient object detection (SOD) methods rely on high-quality depth images. However, their performance is limited when processing low-quality depth maps. This paper exploits more complementary image priors to guide the model to learn on variable depth maps, and a novel multi-prior driven network called MPDNet is proposed for RGB-D SOD. MPDNet utilizes four processing pipelines to process RGB images and other priors, which include an RGB image processing pipeline, a depth map processing pipeline, a fine-grained and gradient prior processing pipeline, and an edge learning pipeline. Specifically, fine-grained and gradient priors are input to the same processing pipeline. For the depth maps, fine-grained and gradient priors, a prior channel attention module utilizes the channel attention mechanism to filter noises and highlights the salient cues. The RGB image processing pipeline uses a multi-feature progressive enhancement module to fuse and enhance features from depth maps. And a multi-feature prediction decoder decodes initial salient masks. In the edge learning pipeline, edge prior serves as an edge label and is captured by an edge capture module. Finally, the clear salient masks are obtained by fusing the salient information from the four pipelines. The experimental results on six benchmarks indicate that the proposed method outperforms thirteen state-of-the-art methods in six evaluation metrics.

Saliency and edge features-guided end-to-end network for salient object detection

The rapid development of the Vision Transformer backbones has enabled the capture of feature information with global dependencies, leading to excellent performance in salient object detection tasks. However, it fails to adequately emphasize fine local features of edges, resulting in coarse and blurry edges in the final output. Therefore, this paper proposes a set prediction method for salient object detection. The approach enables the model to simultaneously consider both saliency (salient object) and edge features, achieving an end-to-end edge feature fusion. There is no need to design multiple complex branch structures and multiple training as with other methods. The model integrates random edge neighborhood sampling to enhance the recognition accuracy of local edge features in images. This approach addresses the weak perception of local features by Transformers and the issue encountered during actual training, where edge pixels typically occupy a much smaller proportion of the image compared to the background, resulting in insufficient learning of edge features by the model. The proposed end-to-end model in this paper fuses multiple features, including edge and salient objects. They are extracted within a unified framework while simultaneously outputting edge and salient object maps. Experimental results on six public datasets show that the proposed method significantly improves model performance benchmarks for detecting salient objects.

Progressive expansion for semi-supervised bi-modal salient object detection

Existing bi-modal salient object detection (SOD) methods primarily rely on fully supervised training strategies that require extensive manual annotation. Undoubtedly, extensive manual annotation is time-consuming and laborious, and the fully supervised strategy is also prone to overfitting on the training set. Therefore, we introduce a semi-supervised learning architecture (SSLA) to alleviate these problems while ensuring detection performance. Considering that the inherent training mode and concise architecture of basic SSLA will limit its ability to effectively explore the learning potential of the model, we further propose two optimization strategies, dynamic adjustment and active expansion. Specifically, we dynamically adjust the supervision scheme for unlabeled samples during training so that the model can continuously utilize the model’s gains (pseudo labels) to supervise and guide the model to further explore the unlabeled samples. Furthermore, the active expansion strategy enables the model to acquire more beneficial supervised information and focuses its attention on difficult-to-segment samples. In summary, an effective progressive expansion network (PENet) architecture for semi-supervised bi-modal SOD is proposed. Extensive experiments indicate that our PENet architecture, while effectively alleviating 90% of annotation burdens, has achieved highly competitive results in RGB-T and RGB-D tasks compared to fully supervised methods. The performance is even more pronounced during cross-dataset testing.

DASOD: Detail-aware salient object detection

Salient object detection (SOD) is a challenging task in computer vision. Current SOD approaches have made significant progress, but they fail in challenging scenarios. This paper categorizes the existing challenges in SOD into four groups: images with complex backgrounds, low contrast, transparent objects, and occluded objects. Then, the Detail-Aware Salient Object Detection (DASOD) method is proposed to address these challenging scenarios. To the best of our knowledge, DASOD is the first method that considers mentioned challenging situations together and detects salient objects in images through camouflaged object detection (COD). DASOD has two main stages: 1) pseudo-mask generation and 2) refinement. It first generates a pseudo-mask using the body label and super-resolution technique, then refines the pseudo-mask with the detail map produced by the pseudo-edge generator to detect salient objects with clear boundaries in the pseudo-mask refinement module. This module quantifies uncertainty using the conditional normalizing flows (cFlow) based conditional variational auto-encoder (cVAE) to generate reliable results. Extensive experiments are conducted on six datasets, and the performance of DASOD is compared with 18 state-of-the-art methods. The results demonstrate that DASOD outperforms its competitors and can accurately detect the salient objects when the image background is cluttered and the contrast between foreground and background is low. Also, it effectively detects the transparent and occluded objects in images. It achieves MAE rates of 0.052, 0.033, 0.027, 0.024, 0.059, and 0.088 on DUT-OMRON, DUTS-TE, ECSSD, HKU-IS, PASCAL-S, and SCAS datasets, respectively. All the implementation source codes and results are available at: https://github.com/BaharehAsheghi/DASOD.

Cross‐scale resolution consistent network for salient object detection

AbstractThe salient object detection task tries to simulate the human visual system for most eye‐catching objects or regions detection. However, due to the complexity of the visual mechanisms, current methods will suffer from severe performance degradation, leading to inconsistent prediction results for the same regions, when directly adopting a model trained on a fixed resolution to evaluate at other different resolutions. Considering that consistency in predictions is essential for salient object detection, a cross‐scale resolution consistent salient object detection method, called RCNet, is proposed. Specifically, to enhance the model's capacity for generalization across images of varying resolutions and make the model implicitly learn the scale invariance, a multi‐resolution data enhancement module is constructed to generate images with arbitrary resolutions for the same scene. Moreover, to accomplish better multi‐level feature fusion, a cross‐scale fusion module is developed to fuse high‐level semantic features and low‐level detail features. Additionally, to explicitly learn the scale invariance of the salient scores, a hybrid salient consistency loss is formulated on salient object detection with different resolutions. Comprehensive evaluations on five benchmark datasets show that RCNet achieves a highly competitive result.

CMIGNet: Cross-Modal Inverse Guidance Network for RGB-Depth salient object detection

Currently, the majority of RGB-Depth salient object detection (SOD) methods utilize the encoder–decoder architecture. However, they often fail to utilize the encoding and decoding features fully. This paper rethinks the differences and correlations between them and proposes the Cross-Modal Inverse Guidance Network (CMIGNet) for SOD. Specifically, a Multi-level Feature Guidance Enhancement (MFGE) module is integrated into every layer of the foundational network. It employs a high-level decoding feature to guide the low-level RGB and depth encoding features, facilitating the rapid identification of salient regions and noise removal. The dual-stream encoding features guided by the MFGE module are combined using the proposed Dual-Stream Interactive Fusion (DSIF) module. It could simultaneously reduce dependence on two modal features during the fusion process. Thus, the impact on the results can be reduced in complex scenes when one modality is absent or confusing. Finally, the edge information is supplemented using the proposed Edge Refinement Awareness (ERA) module to generate the final salient map. Comparisons on seven widely used and one latest challenging RGB-D datasets show that the performance of the proposed CMIGNet is highly competitive with the state-of-the-art RGB-Depth SOD models. Additionally, our model is lighter and faster.

WBNet: Weakly-supervised salient object detection via scribble and pseudo-background priors

Weakly supervised salient object detection (WSOD) methods endeavor to boost sparse labels to get more salient cues in various ways. Among them, an effective approach is using pseudo labels from multiple unsupervised self-learning methods, but inaccurate and inconsistent pseudo labels could ultimately lead to detection performance degradation. To tackle this problem, we develop a new multi-source WSOD framework, WBNet, that can effectively utilize pseudo-background (non-salient region) labels combined with scribble labels to obtain more accurate salient features. We first design a comprehensive salient pseudo-mask generator from multiple self-learning features. Then, we pioneer the exploration of generating salient pseudo-labels via point-prompted and box-prompted Segment Anything Models (SAM). Then, WBNet leverages a pixel-level Feature Aggregation Module (FAM), a mask-level Transformer-decoder (TFD), and an auxiliary Boundary Prediction Module (EPM) with a hybrid loss function to handle complex saliency detection tasks. Comprehensively evaluated with state-of-the-art methods on five widely used datasets, the proposed method significantly improves saliency detection performance. The code and results are publicly available at https://github.com/yiwangtz/WBNet.

BMFNet: Bifurcated multi-modal fusion network for RGB-D salient object detection

Although deep learning-based RGB-D salient object detection methods have achieved impressive results in the recent years, there are still some issues need to be addressed including multi-modal fusion and multi-level aggregation. In this paper, we propose a bifurcated multi-modal fusion network (BMFNet) to address these two issues cooperatively. First, we design a multi-modal feature interaction (MFI) module to fully capture the complementary information between the RGB and depth features by leveraging the channel attention and spatial attention. Second, unlike the widely used layer-by-layer progressive fusion, we adopt a bifurcated fusion strategy for all the multi-level unimodal and cross-modal features to effectively reduce the gaps between features at different levels. For the intra-group feature aggregation, a multi-modal feature fusion (MFF) module is designed to integrate the intra-group multi-modal features to produce a low-level/high-level saliency feature. For the inter-group aggregation, a multi-scale feature learning (MFL) module is introduced to exploit the contextual interactions between different scales to boost fusion performance. Experimental results on five public RGB-D datasets demonstrate the effectiveness and superiority of our proposed network. The code and prediction maps will be available at https://github.com/ZhangQing0329/BMFNet

A novel jujube tree trunk and branch salient object detection method for catch-and-shake robotic visual perception

Visual perception has become a prerequisite for automated jujube harvesting robot operations under complex orchard conditions. Catch-and-Shake harvesting, as the most efficient and common harvesting method, has widely been applied on various manually operated harvesters to complete large-area jujube fruit harvesting. However, the main factors restricting the development of existing harvesters are labor shortage, high labor cost, and low operating efficiency. To address the issues, we designed a catch-and-shake harvesting robot for jujube tree trunks and branches visual perception that can provide a barrier-free catch-and-shake operation area and guide the manipulator to reach the area to complete the harvesting operation. Meanwhile, a visual perception system including tree trunks and branches detection, skeleton extraction, catch-and-shake area confirmation was presented to guide robot intelligent operations. In the visual perception system, a novel salientobjectdetectionmodel called feature intersection and fusion Transformer (FIT-Transformer) network was proposed to split branches and background to provide reference for determining safe catch-and-shake areas. Moreover, we designed a diverse feature aggregation (DFA) and an attention feature fusion module (AFFM) to strengthen feature learning capabilities and obtain robust perception models. Comparative experimental results showed that our proposed FIT-Transformer model outperformed 12 state-of-the-art (SOTA) algorithms including C2FNet, RAS, BASNet, U2Net, SCRNet, PiCANet, EDRNet, EGNet, ICONR, VST, TransSOD and ABiU_Net. Specifically, the segmentation accuracy of jujube tree trunks and branches using our method showed the satisfactory result on five evaluation indexes under natural environment (the EM, SM, WF, FM and MAE reached 0.9713, 0.8991, 0.8854, 0.8905, and 0.0302, respectively). Field experiments also proved that our method could meet the requirements of operational accuracy and real-time operations.

CCNet: Collaborative Camouflaged Object Detection via decoder-induced information interaction and supervision refinement network

Recently, research based on the camouflaged object detection (COD) task has achieved great progress, while the collaborative camouflaged object detection (CoCOD) task is still lacking. Our research focuses on the simultaneous detection and localization of the collaborative camouflaged objects, i.e., CoCOD task. We use the cooperative information between a single image and a group of camouflage images to discover cooperative camouflaged objects effectively. In this paper, we propose a collaborative cross-scale feature learning network (CCNet). Our model is characterized by two innovative constructions: We proposed an edge augmentation module (EAM), which effectively extracts edge information of the camouflaged object and integrates it with the collaborative information employed in auxiliary supervision. In addition, we design a group decoder module (GDM) to extract and merge co-camouflage information. Extensive experiments on CoCOD8K datasets demonstrate that our CCNet significantly outperforms the existing 13 state-of-the-art COD and 6 state-of-the-art collaborative salient object detection (CoSOD) methods under six widely used evaluation metrics. The code will be available at: https://github.com/zc199823/CCNet--CoCOD.

Scribble-based complementary graph reasoning network for weakly supervised salient object detection

Current salient object detection (SOD) methods rely heavily on accurate pixel-level annotations. To reduce the annotation workload, some scribble-based methods have emerged. Recent works address the sparse scribble annotations by introducing auxiliary information and enhancing local features. However, the impact of long-range dependence between pixels on energy propagation and model performance has not been explored in this field. In this paper, we propose a novel complementary graph reasoning network (CGRNet), which globally infers relationships between salient regions by building graph representations. Specifically, we introduce a dual-stream cross-interactive graph reasoning pipeline to model high-level representations and incorporate efficient graph cooperation unit (GCU) to adaptively select complementary components from the representations. Additionally, considering the lack of structural information in scribble data, we design an edge-oriented module (EOM) to explicitly mine boundary semantics. Finally, we propose a dense fusion strategy (DFS) to aggregate multi-source semantics in a multi-guidance manner for obtaining complete global information. Experimental and visual results on five benchmarks demonstrate the superiority of our proposed CGRNet.

Superpixel-wise contrast exploration for salient object detection

Salient object detection (SOD) methods typically consider SOD as a pixel-wise binary classification problem and utilize the binary cross-entropy (BCE) loss for optimization. However, the BCE loss ignores the global dependencies between pixels of the entire image, which is important for ensuring the accuracy and integrity of objects. To address this limitation explicitly, contrastive learning is introduced to enhance both the intra-pixel compactness of the foreground and inter-pixel separability between the foreground and background. Unfortunately, the random pixel sampling strategies of existing contrast algorithms cannot ensure that the sampled pixels cover all semantic contents across the entire image because of the lack of fine-grained semantic categories for pixels in SOD. Hence, in this study, a SuperPixel-wise Contrastive (SPCont) algorithm is proposed, which samples all superpixel centers to improve the diversity and representativeness of the sampled pixels. In addition, a SOD-specific hard pixel sampling strategy is designed to refine the edge details of the objects. An intra-image multi-level negative sampling approach is presented to increase the number of negative samples. Extensive experiments are conducted to demonstrate that incorporating the SPCont algorithm into the state-of-the-art network results in significant performance improvements. We expect that our work will open new paths for the application of contrastive learning to SOD.

Semantic-Consistency-guided Learning on Deep Features for Unsupervised Salient Object Detection

Unsupervised salient object detection is an important task in many real-world scenarios where pixel-wise label information is of scarce availability. Despite its significance, this problem remains rarely explored, with a few works that consider unsupervised salient object detection methods based on the fused graph from the sum fusion of multiple deep feature similarity matrices. However, these methods ignore the interrelation of the low-level feature similarity matrices and the high-level semantic similarity matrice, which degrades the quality of the fused graph. In this article, we propose a semantic-consistency-guided multi-graph fusion learning algorithm for unsupervised saliency detection, where the consistency and inconsistency between multiple low-level feature similarity matrices and the high-level semantic similarity matrice are explored to promote the robustness and quality of the fused graph. In the first stage, a semantic-consistency-guided multi-graph fusion learning method is proposed to exploit consistency and inconsistency of multiple low-level deep features and the high-level semantic feature. The semantic-consistency-guided similarity matrices are computed for preliminary saliency ranking. In the following saliency refinement stage, the semantic-enhanced similarity matrices are built by the cross diffusion to fuse the multiple low-level deep features and the high semantic deep feature. Based on the semantic-enhanced similarity matrices, the refinement saliency maps are calculated in a semantic-enhanced cellular automata manner. Furthermore, the final ensemble stage of the large margin semi-supervised classification views the preliminary ranking results and refinement results as features, adopts the large margin graphs for saliency ensemble. Extensive evaluations over four benchmark datasets show that the proposed unsupervised method performs favorably against the state-of-the-art approaches and is competitive with some supervised deep learning-based methods.

Segmentation of tumor regions using 3D-UNet in magnetic resonance imaging

<p>Brain tumor has been a severe problem for a few decades ago. With the advancement in medical technologies, a brain tumor can be treated if observed earlier. This paper aims to segment and classify the tumor regions from Magnetic Resonance Imaging (MRI). The work consists of two steps. In step1, the 3D MRI images are pre-processed by the Salient Object Detection method to improve efficiency. In step2, the improved 3D-Res2UNet segments the tumor regions. The segmented tumors are partitioned into two classes using a Support Vector Machine (SVM) classifier. The method is tested using BRATS 2017 and 2018 datasets and obtained 87.1% and 99.2% dice score for BRATS 2017 and 2018, respectively. The performance of the proposed method is better compared to most recent methods.</p>

Feature interaction and two-stage cross-modal fusion for RGB-D salient object detection

Most existing RGB-D salient object detection (SOD) methods extract features of both modalities in parallel or adopt depth features as supplementary information for unidirectional interaction from depth modality to RGB modality in the encoder stage. These methods ignore the influence of low-quality depth maps, and there is still room for improvement in effectively fusing RGB features and depth features. To address the above problems, this paper proposes a Feature Interaction Network (FINet), which performs bi-directional interaction through feature interaction module (FIM) in the encoder stage. The feature interaction module is divided into two parts: depth enhancement module (DEM) filters the noise in the depth features through the attention mechanism; and cross enhancement module (CEM) effectively interacts RGB features and depth features. In addition, this paper proposes a two-stage cross-modal fusion strategy: high-level fusion adopts the semantic information of high level for coarse localization of salient regions, and low-level fusion makes full use of the detailed information of low level through boundary fusion, and then we progressively refine high-level and low-level cross-modal features to obtain the final saliency prediction map. Extensive experiments show that the proposed model achieves better performance than eight state-of-the-art models on five standard datasets.

Robust Perception and Precise Segmentation for Scribble-Supervised RGB-D Saliency Detection.

This paper proposes a scribble-based weakly supervised RGB-D salient object detection (SOD) method to relieve the annotation burden from pixel-wise annotations. In view of the ensuing performance drop, we summarize two natural deficiencies of the scribbles and try to alleviate them, which are the weak richness of the pixel training samples (WRPS) and the poor structural integrity of the salient objects (PSIO). WRPS hinders robust saliency perception learning, which can be alleviated via model design for robust feature learning and pseudo labels generation for training sample enrichment. Specifically, we first design a dynamic searching process module as a meta operation to conduct multi-scale and multi-modal feature fusion for the robust RGB-D SOD model construction. Then, a dual-branch consistency learning mechanism is proposed to generate enough pixel training samples for robust saliency perception learning. PSIO makes direct structural learning infeasible since scribbles can not provide integral structural supervision. Thus, we propose an edge-region structure-refinement loss to recover the structural information and make precise segmentation. We deploy all components and conduct ablation studies on two baselines to validate their effectiveness and generalizability. Experimental results on eight datasets show that our method outperforms other scribble-based SOD models and achieves comparable performance with fully supervised state-of-the-art methods.

A Visual Representation-Guided Framework With Global Affinity for Weakly Supervised Salient Object Detection

Fully supervised salient object detection (SOD) methods have made considerable progress in performance, yet these models rely heavily on expensive pixel-wise labels. Recently, to achieve a trade-off between labeling burden and performance, scribble-based SOD methods have attracted increasing attention. Previous scribble-based models directly implement the SOD task only based on SOD training data with limited information, it is extremely difficult for them to understand the image and further achieve a superior SOD task. In this paper, we propose a simple yet effective framework guided by general visual representations with rich contextual semantic knowledge for scribble-based SOD. These general visual representations are generated by self-supervised learning based on large-scale unlabeled datasets. Our framework consists of a task-related encoder, a general visual module, and an information integration module to efficiently combine the general visual representations with task-related features to perform the SOD task based on understanding the contextual connections of images. Meanwhile, we propose a novel global semantic affinity loss to guide the model to perceive the global structure of the salient objects. Experimental results on five public benchmark datasets demonstrate that our method, which only utilizes scribble annotations without introducing any extra label, outperforms the state-of-the-art weakly supervised SOD methods. Specifically, it outperforms the previous best scribble-based method on all datasets with an average gain of 5.5% for max f-measure, 5.8% for mean f-measure, 24% for MAE, and 3.1% for E-measure. Moreover, our method achieves comparable or even superior performance to the state-of-the-art fully supervised models.

Transformer Fusion and Pixel-Level Contrastive Learning for RGB-D Salient Object Detection

Current RGB-D salient object detection (RGB-D SOD) methods mainly develop a generalizable model trained by binary cross-entropy (BCE) loss based on convolutional or Transformer backbones. However, they usually exploit convolutional modules to fuse multi-modality features, with little attention paid to capturing the long-range multi-modality interactions for feature fusion. Furthermore, BCE loss does not explicitly explore intra- and inter-pixel relationships in a joint embedding space. To address these issues, we propose a cross-modality interaction parallel-transformer (CIPT) module, which better captures the long-range multi-modality interactions, generating more comprehensive fusion features. Besides, we propose a pixel-level contrastive learning (PCL) method that improves inter-pixel discrimination and intra-pixel compactness, resulting in a well-structured embedding space and a better saliency detector. Specifically, we propose an asymmetric network (TPCL) for RGB-D SOD, which consists of a Swin V2 Transformer-based backbone and a designed lightweight backbone (LDNet). Moreover, an edge-guided module and a feature enhancement (FE) module are proposed to refine the learned fusion features. Extensive experiments demonstrate that our method achieves excellent performance against 15 state-of-the-art methods on seven public datasets. We expect our work to facilitate the exploration of applying Transformer and contrastive learning for RGB-D SOD tasks. Our codes and predicted saliency maps will be released on GitHub <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/TomorrowJW/TPCL_RGBDSOD</uri> .

Rethinking Object Saliency Ranking: A Novel Whole-Flow Processing Paradigm.

Existing salient object detection methods are capable of predicting binary maps that highlight visually salient regions. However, these methods are limited in their ability to differentiate the relative importance of multiple objects and the relationships among them, which can lead to errors and reduced accuracy in downstream tasks that depend on the relative importance of multiple objects. To conquer, this paper proposes a new paradigm for saliency ranking, which aims to completely focus on ranking salient objects by their "importance order". While previous works have shown promising performance, they still face ill-posed problems. First, the saliency ranking ground truth (GT) orders generation methods are unreasonable since determining the correct ranking order is not well-defined, resulting in false alarms. Second, training a ranking model remains challenging because most saliency ranking methods follow the multi-task paradigm, leading to conflicts and trade-offs among different tasks. Third, existing regression-based saliency ranking methods are complex for saliency ranking models due to their reliance on instance mask-based saliency ranking orders. These methods require a significant amount of data to perform accurately and can be challenging to implement effectively. To solve these problems, this paper conducts an in-depth analysis of the causes and proposes a whole-flow processing paradigm of saliency ranking task from the perspective of "GT data generation", "network structure design" and "training protocol". The proposed approach outperforms existing state-of-the-art methods on the widely-used SALICON set, as demonstrated by extensive experiments with fair and reasonable comparisons. The saliency ranking task is still in its infancy, and our proposed unified framework can serve as a fundamental strategy to guide future work. The code and data will be available at https://github.com/MengkeSong/Saliency-Ranking-Paradigm.