Salient Object Detection Methods Research Articles

BMFNet: Bifurcated multi-modal fusion network for RGB-D salient object detection

Although deep learning-based RGB-D salient object detection methods have achieved impressive results in the recent years, there are still some issues need to be addressed including multi-modal fusion and multi-level aggregation. In this paper, we propose a bifurcated multi-modal fusion network (BMFNet) to address these two issues cooperatively. First, we design a multi-modal feature interaction (MFI) module to fully capture the complementary information between the RGB and depth features by leveraging the channel attention and spatial attention. Second, unlike the widely used layer-by-layer progressive fusion, we adopt a bifurcated fusion strategy for all the multi-level unimodal and cross-modal features to effectively reduce the gaps between features at different levels. For the intra-group feature aggregation, a multi-modal feature fusion (MFF) module is designed to integrate the intra-group multi-modal features to produce a low-level/high-level saliency feature. For the inter-group aggregation, a multi-scale feature learning (MFL) module is introduced to exploit the contextual interactions between different scales to boost fusion performance. Experimental results on five public RGB-D datasets demonstrate the effectiveness and superiority of our proposed network. The code and prediction maps will be available at https://github.com/ZhangQing0329/BMFNet

WBNet: Weakly-supervised salient object detection via scribble and pseudo-background priors

Weakly supervised salient object detection (WSOD) methods endeavor to boost sparse labels to get more salient cues in various ways. Among them, an effective approach is using pseudo labels from multiple unsupervised self-learning methods, but inaccurate and inconsistent pseudo labels could ultimately lead to detection performance degradation. To tackle this problem, we develop a new multi-source WSOD framework, WBNet, that can effectively utilize pseudo-background (non-salient region) labels combined with scribble labels to obtain more accurate salient features. We first design a comprehensive salient pseudo-mask generator from multiple self-learning features. Then, we pioneer the exploration of generating salient pseudo-labels via point-prompted and box-prompted Segment-Anything Models (SAM). Then, WBNet leverages a pixel-level Feature Aggregation Module (FAM), a mask-level Transformer-decoder (TFD), and an auxiliary Boundary Prediction Module (EPM) with a hybrid loss function to handle complex saliency detection tasks. Comprehensively evaluated with state-of-the-art methods on five widely used datasets, the proposed method significantly improves saliency detection performance. The code and results are publicly available at https://github.com/yiwangtz/WBNet.

A novel jujube tree trunk and branch salient object detection method for catch-and-shake robotic visual perception

Visual perception has become a prerequisite for automated jujube harvesting robot operations under complex orchard conditions. Catch-and-Shake harvesting, as the most efficient and common harvesting method, has widely been applied on various manually operated harvesters to complete large-area jujube fruit harvesting. However, the main factors restricting the development of existing harvesters are labor shortage, high labor cost, and low operating efficiency. To address the issues, we designed a catch-and-shake harvesting robot for jujube tree trunks and branches visual perception that can provide a barrier-free catch-and-shake operation area and guide the manipulator to reach the area to complete the harvesting operation. Meanwhile, a visual perception system including tree trunks and branches detection, skeleton extraction, catch-and-shake area confirmation was presented to guide robot intelligent operations. In the visual perception system, a novel salientobjectdetectionmodel called feature intersection and fusion Transformer (FIT-Transformer) network was proposed to split branches and background to provide reference for determining safe catch-and-shake areas. Moreover, we designed a diverse feature aggregation (DFA) and an attention feature fusion module (AFFM) to strengthen feature learning capabilities and obtain robust perception models. Comparative experimental results showed that our proposed FIT-Transformer model outperformed 12 state-of-the-art (SOTA) algorithms including C2FNet, RAS, BASNet, U2Net, SCRNet, PiCANet, EDRNet, EGNet, ICONR, VST, TransSOD and ABiU_Net. Specifically, the segmentation accuracy of jujube tree trunks and branches using our method showed the satisfactory result on five evaluation indexes under natural environment (the EM, SM, WF, FM and MAE reached 0.9713, 0.8991, 0.8854, 0.8905, and 0.0302, respectively). Field experiments also proved that our method could meet the requirements of operational accuracy and real-time operations.

CCNet: Collaborative Camouflaged Object Detection via decoder-induced information interaction and supervision refinement network

Recently, research based on the camouflaged object detection (COD) task has achieved great progress, while the collaborative camouflaged object detection (CoCOD) task is still lacking. Our research focuses on the simultaneous detection and localization of the collaborative camouflaged objects, i.e., CoCOD task. We use the cooperative information between a single image and a group of camouflage images to discover cooperative camouflaged objects effectively. In this paper, we propose a collaborative cross-scale feature learning network (CCNet). Our model is characterized by two innovative constructions: We proposed an edge augmentation module (EAM), which effectively extracts edge information of the camouflaged object and integrates it with the collaborative information employed in auxiliary supervision. In addition, we design a group decoder module (GDM) to extract and merge co-camouflage information. Extensive experiments on CoCOD8K datasets demonstrate that our CCNet significantly outperforms the existing 13 state-of-the-art COD and 6 state-of-the-art collaborative salient object detection (CoSOD) methods under six widely used evaluation metrics. The code will be available at: https://github.com/zc199823/CCNet--CoCOD.

Scribble-based complementary graph reasoning network for weakly supervised salient object detection

Current salient object detection (SOD) methods rely heavily on accurate pixel-level annotations. To reduce the annotation workload, some scribble-based methods have emerged. Recent works address the sparse scribble annotations by introducing auxiliary information and enhancing local features. However, the impact of long-range dependence between pixels on energy propagation and model performance has not been explored in this field. In this paper, we propose a novel complementary graph reasoning network (CGRNet), which globally infers relationships between salient regions by building graph representations. Specifically, we introduce a dual-stream cross-interactive graph reasoning pipeline to model high-level representations and incorporate efficient graph cooperation unit (GCU) to adaptively select complementary components from the representations. Additionally, considering the lack of structural information in scribble data, we design an edge-oriented module (EOM) to explicitly mine boundary semantics. Finally, we propose a dense fusion strategy (DFS) to aggregate multi-source semantics in a multi-guidance manner for obtaining complete global information. Experimental and visual results on five benchmarks demonstrate the superiority of our proposed CGRNet.

Superpixel-wise contrast exploration for salient object detection

Salient object detection (SOD) methods typically consider SOD as a pixel-wise binary classification problem and utilize the binary cross-entropy (BCE) loss for optimization. However, the BCE loss ignores the global dependencies between pixels of the entire image, which is important for ensuring the accuracy and integrity of objects. To address this limitation explicitly, contrastive learning is introduced to enhance both the intra-pixel compactness of the foreground and inter-pixel separability between the foreground and background. Unfortunately, the random pixel sampling strategies of existing contrast algorithms cannot ensure that the sampled pixels cover all semantic contents across the entire image because of the lack of fine-grained semantic categories for pixels in SOD. Hence, in this study, a SuperPixel-wise Contrastive (SPCont) algorithm is proposed, which samples all superpixel centers to improve the diversity and representativeness of the sampled pixels. In addition, a SOD-specific hard pixel sampling strategy is designed to refine the edge details of the objects. An intra-image multi-level negative sampling approach is presented to increase the number of negative samples. Extensive experiments are conducted to demonstrate that incorporating the SPCont algorithm into the state-of-the-art network results in significant performance improvements. We expect that our work will open new paths for the application of contrastive learning to SOD.

Semantic-Consistency-guided Learning on Deep Features for Unsupervised Salient Object Detection

Unsupervised salient object detection is an important task in many real-world scenarios where pixel-wise label information is of scarce availability. Despite its significance, this problem remains rarely explored, with a few works that consider unsupervised salient object detection methods based on the fused graph from the sum fusion of multiple deep feature similarity matrices. However, these methods ignore the interrelation of the low-level feature similarity matrices and the high-level semantic similarity matrice, which degrades the quality of the fused graph. In this article, we propose a semantic-consistency-guided multi-graph fusion learning algorithm for unsupervised saliency detection, where the consistency and inconsistency between multiple low-level feature similarity matrices and the high-level semantic similarity matrice are explored to promote the robustness and quality of the fused graph. In the first stage, a semantic-consistency-guided multi-graph fusion learning method is proposed to exploit consistency and inconsistency of multiple low-level deep features and the high-level semantic feature. The semantic-consistency-guided similarity matrices are computed for preliminary saliency ranking. In the following saliency refinement stage, the semantic-enhanced similarity matrices are built by the cross diffusion to fuse the multiple low-level deep features and the high semantic deep feature. Based on the semantic-enhanced similarity matrices, the refinement saliency maps are calculated in a semantic-enhanced cellular automata manner. Furthermore, the final ensemble stage of the large margin semi-supervised classification views the preliminary ranking results and refinement results as features, adopts the large margin graphs for saliency ensemble. Extensive evaluations over four benchmark datasets show that the proposed unsupervised method performs favorably against the state-of-the-art approaches and is competitive with some supervised deep learning-based methods.

Segmentation of tumor regions using 3D-UNet in magnetic resonance imaging

<p>Brain tumor has been a severe problem for a few decades ago. With the advancement in medical technologies, a brain tumor can be treated if observed earlier. This paper aims to segment and classify the tumor regions from Magnetic Resonance Imaging (MRI). The work consists of two steps. In step1, the 3D MRI images are pre-processed by the Salient Object Detection method to improve efficiency. In step2, the improved 3D-Res2UNet segments the tumor regions. The segmented tumors are partitioned into two classes using a Support Vector Machine (SVM) classifier. The method is tested using BRATS 2017 and 2018 datasets and obtained 87.1% and 99.2% dice score for BRATS 2017 and 2018, respectively. The performance of the proposed method is better compared to most recent methods.</p>

Feature interaction and two-stage cross-modal fusion for RGB-D salient object detection

Most existing RGB-D salient object detection (SOD) methods extract features of both modalities in parallel or adopt depth features as supplementary information for unidirectional interaction from depth modality to RGB modality in the encoder stage. These methods ignore the influence of low-quality depth maps, and there is still room for improvement in effectively fusing RGB features and depth features. To address the above problems, this paper proposes a Feature Interaction Network (FINet), which performs bi-directional interaction through feature interaction module (FIM) in the encoder stage. The feature interaction module is divided into two parts: depth enhancement module (DEM) filters the noise in the depth features through the attention mechanism; and cross enhancement module (CEM) effectively interacts RGB features and depth features. In addition, this paper proposes a two-stage cross-modal fusion strategy: high-level fusion adopts the semantic information of high level for coarse localization of salient regions, and low-level fusion makes full use of the detailed information of low level through boundary fusion, and then we progressively refine high-level and low-level cross-modal features to obtain the final saliency prediction map. Extensive experiments show that the proposed model achieves better performance than eight state-of-the-art models on five standard datasets.

Multi-prior Driven Network for RGB-D Salient Object Detection

Most existing RGB-D salient object detection (SOD) methods rely on high-quality depth images. However, their performance is limited when processing low-quality depth maps. This paper exploits more complementary image priors to guide the model to learn on variable depth maps, and a novel multi-prior driven network called MPDNet is proposed for RGB-D SOD. MPDNet utilizes four processing pipelines to process RGB images and other priors, which include an RGB image processing pipeline, a depth map processing pipeline, a fine-grained and gradient prior processing pipeline, and an edge learning pipeline. Specifically, fine-grained and gradient priors are input to the same processing pipeline. For the depth maps, fine-grained and gradient priors, a prior channel attention module utilizes the channel attention mechanism to filter noises and highlights the salient cues. The RGB image processing pipeline uses a multi-feature progressive enhancement module to fuse and enhance features from depth maps. And a multi-feature prediction decoder decodes initial salient masks. In the edge learning pipeline, edge prior serves as an edge label and is captured by an edge capture module. Finally, the clear salient masks are obtained by fusing the salient information from the four pipelines. The experimental results on six benchmarks indicate that the proposed method outperforms thirteen state-of-the-art methods in six evaluation metrics.

Robust Perception and Precise Segmentation for Scribble-Supervised RGB-D Saliency Detection.

This paper proposes a scribble-based weakly supervised RGB-D salient object detection (SOD) method to relieve the annotation burden from pixel-wise annotations. In view of the ensuing performance drop, we summarize two natural deficiencies of the scribbles and try to alleviate them, which are the weak richness of the pixel training samples (WRPS) and the poor structural integrity of the salient objects (PSIO). WRPS hinders robust saliency perception learning, which can be alleviated via model design for robust feature learning and pseudo labels generation for training sample enrichment. Specifically, we first design a dynamic searching process module as a meta operation to conduct multi-scale and multi-modal feature fusion for the robust RGB-D SOD model construction. Then, a dual-branch consistency learning mechanism is proposed to generate enough pixel training samples for robust saliency perception learning. PSIO makes direct structural learning infeasible since scribbles can not provide integral structural supervision. Thus, we propose an edge-region structure-refinement loss to recover the structural information and make precise segmentation. We deploy all components and conduct ablation studies on two baselines to validate their effectiveness and generalizability. Experimental results on eight datasets show that our method outperforms other scribble-based SOD models and achieves comparable performance with fully supervised state-of-the-art methods.

A Visual Representation-Guided Framework With Global Affinity for Weakly Supervised Salient Object Detection

Fully supervised salient object detection (SOD) methods have made considerable progress in performance, yet these models rely heavily on expensive pixel-wise labels. Recently, to achieve a trade-off between labeling burden and performance, scribble-based SOD methods have attracted increasing attention. Previous scribble-based models directly implement the SOD task only based on SOD training data with limited information, it is extremely difficult for them to understand the image and further achieve a superior SOD task. In this paper, we propose a simple yet effective framework guided by general visual representations with rich contextual semantic knowledge for scribble-based SOD. These general visual representations are generated by self-supervised learning based on large-scale unlabeled datasets. Our framework consists of a task-related encoder, a general visual module, and an information integration module to efficiently combine the general visual representations with task-related features to perform the SOD task based on understanding the contextual connections of images. Meanwhile, we propose a novel global semantic affinity loss to guide the model to perceive the global structure of the salient objects. Experimental results on five public benchmark datasets demonstrate that our method, which only utilizes scribble annotations without introducing any extra label, outperforms the state-of-the-art weakly supervised SOD methods. Specifically, it outperforms the previous best scribble-based method on all datasets with an average gain of 5.5% for max f-measure, 5.8% for mean f-measure, 24% for MAE, and 3.1% for E-measure. Moreover, our method achieves comparable or even superior performance to the state-of-the-art fully supervised models.

Transformer Fusion and Pixel-Level Contrastive Learning for RGB-D Salient Object Detection

Current RGB-D salient object detection (RGB-D SOD) methods mainly develop a generalizable model trained by binary cross-entropy (BCE) loss based on convolutional or Transformer backbones. However, they usually exploit convolutional modules to fuse multi-modality features, with little attention paid to capturing the long-range multi-modality interactions for feature fusion. Furthermore, BCE loss does not explicitly explore intra- and inter-pixel relationships in a joint embedding space. To address these issues, we propose a cross-modality interaction parallel-transformer (CIPT) module, which better captures the long-range multi-modality interactions, generating more comprehensive fusion features. Besides, we propose a pixel-level contrastive learning (PCL) method that improves inter-pixel discrimination and intra-pixel compactness, resulting in a well-structured embedding space and a better saliency detector. Specifically, we propose an asymmetric network (TPCL) for RGB-D SOD, which consists of a Swin V2 Transformer-based backbone and a designed lightweight backbone (LDNet). Moreover, an edge-guided module and a feature enhancement (FE) module are proposed to refine the learned fusion features. Extensive experiments demonstrate that our method achieves excellent performance against 15 state-of-the-art methods on seven public datasets. We expect our work to facilitate the exploration of applying Transformer and contrastive learning for RGB-D SOD tasks. Our codes and predicted saliency maps will be released on GitHub <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/TomorrowJW/TPCL_RGBDSOD</uri> .

Rethinking Object Saliency Ranking: A Novel Whole-Flow Processing Paradigm.

Existing salient object detection methods are capable of predicting binary maps that highlight visually salient regions. However, these methods are limited in their ability to differentiate the relative importance of multiple objects and the relationships among them, which can lead to errors and reduced accuracy in downstream tasks that depend on the relative importance of multiple objects. To conquer, this paper proposes a new paradigm for saliency ranking, which aims to completely focus on ranking salient objects by their "importance order". While previous works have shown promising performance, they still face ill-posed problems. First, the saliency ranking ground truth (GT) orders generation methods are unreasonable since determining the correct ranking order is not well-defined, resulting in false alarms. Second, training a ranking model remains challenging because most saliency ranking methods follow the multi-task paradigm, leading to conflicts and trade-offs among different tasks. Third, existing regression-based saliency ranking methods are complex for saliency ranking models due to their reliance on instance mask-based saliency ranking orders. These methods require a significant amount of data to perform accurately and can be challenging to implement effectively. To solve these problems, this paper conducts an in-depth analysis of the causes and proposes a whole-flow processing paradigm of saliency ranking task from the perspective of "GT data generation", "network structure design" and "training protocol". The proposed approach outperforms existing state-of-the-art methods on the widely-used SALICON set, as demonstrated by extensive experiments with fair and reasonable comparisons. The saliency ranking task is still in its infancy, and our proposed unified framework can serve as a fundamental strategy to guide future work. The code and data will be available at https://github.com/MengkeSong/Saliency-Ranking-Paradigm.

SODRet: Instance retrieval using salient object detection for self-service shopping

Self-service shopping technologies have become commonplace in modern society. Although various innovative solutions have been adopted, there is still a gap in providing efficient services to consumers. Recent developments in mobile application technologies and internet-of-things devices promote information and knowledge dissemination by integrating innovative services to meet users’ needs. We argue that object retrieval applications can be used to provide effective online or self-service shopping. Therefore, to fill this technological void, this study aims to propose an object retrieval system using a fusion-based salient object detection (SOD) method. The SOD has attracted significant attention, and recently many heuristic computational models have been developed for object detection. It has been widely used in object detection and retrieval applications. This work proposes an instance retrieval system based on the SOD to find the objects from the commodity datasets. A prediction about the object’s position is made using the saliency detection system through a saliency model, and the proposed SOD-based retrieval (SODRet) framework uses saliency maps for retrieving the searched items. The method proposed in this work is evaluated on INSTRE and Flickr32 datasets. Our proposed work outperforms state-of-the-art object retrieval methods and can further be employed for large-scale self-service shopping-based points of sales.

An Adaptive Multi-Content Complementary Network for Salient Object Detection

Deep learning methods for salient object detection (SOD) have been studied actively and promisingly. However, the existing methods mainly focus on the decoding process and ignore the differences in the contributions of different encoder blocks. To address this problem for SOD, we propose an adaptive multi-content complementary network (PASNet) for salient object detection which aims to exploit the valuable contextual information in the encoder fully. Unlike existing CNN-based methods, we adopt the pyramidal visual transformer (PVTv2) as the backbone network to learn global and local representations with its self-attention mechanism. Then, we follow the coarse-to-fine strategy and introduce two novel modules, including an advanced semantic fusion module (ASFM) and a self-refinement module (SRM). Among these, the ASFM takes local branches and adjacent branches as inputs and collects semantic and location information of salient objects from high-level features to generate an initial coarse saliency map. The coarse saliency map serves as the location guidance for low-level features, and the SRM is applied to capture detailed information disguised in low-level features. We expand the location information with high-level semantics from top to bottom across the salient region, which is effectively fused with detailed information through feature modulation. The model effectively suppresses noises in the features and significantly improves their expressive capabilities. To verify the effectiveness of our PASNet, we conducted extensive experiments on five challenging datasets, and the results show that the proposed model is superior to some of the current state-of-the-art methods under different evaluation metrics.

ELWNet: An Extremely Lightweight Approach for Real-Time Salient Object Detection

Existing lightweight salient object detection (SOD) methods aim to solve the problem of high computational costs that is prevalent with heavyweight methods. However, compared with heavyweight methods, the detection accuracy of lightweight methods is greatly reduced while real-time performance is not significantly improved. Therefore, we aim to establish a trade off between computational cost and detection performance by improving the network efficiency. We propose a fast and extremely lightweight end-to-end wavelet neural network (ELWNet) for real-time salient object detection. ELWNet can achieve salient object detection and segmentation at approximately 70FPS (GPU), 19FPS (CPU) with 76K parameters and 0.38G FLOPs. We introduce wavelet transform theory into a neural network, proposing a wavelet transform module (WTM), a wavelet transform fusion module (WTFM), a novel feature residual mechanism, and construct an efficient architecture. The wavelet transform theory is integrated into the neural network to realize the interaction between the features in the frequency and the time domain. Meanwhile, ELWNet does not rely on a pre-trained model, which significantly reduces redundant features. We validate the performance of ELWNet using five well-known datasets, and demonstrate state-of-the-art performance compared with 24 other SOD models in terms of being lightweight, detection accuracy and real-time capabilities. Our method maintains high detection performance while reducing the number of model parameters by approximately 99% compared with heavyweight methods.

TMNet: Triple-modal interaction encoder and multi-scale fusion decoder network for V-D-T salient object detection

Salient object detection methods based on two-modal images have achieved remarkable success with the aid of image acquisition equipment. However, environmental factors often interfere with the Depth and Thermal maps, rendering them ineffective in providing object information. To address this weakness, we utilize the VDT dataset, which includes Visible, Depth, and Thermal images, and propose a triple-modal interaction encoder and multi-scale fusion decoder network (TMNet) to highlight the salient regions. The triple-modal interaction encoder comprises the separation context-aware feature module, channel-wise fusion module, and triple-modal refinement and fusion module, enabling us to fully explore and utilize the complementarity between Visible, Depth, and Thermal information. The multi-scale fusion decoder involves the semantic-aware localizing module and contour-aware refinement module to extract and fuse the location and boundary information, yielding a high-quality saliency map. Extensive experiments on the public VDT-2048 dataset demonstrate that our TMNet outperforms existing state-of-the-art methods in terms of all evaluation metrics.

Salient Object Detection Based on Optimization of Feature Computation by Neutrosophic Set Theory.

In recent saliency detection research, too many or too few image features are used in the algorithm, and the processing of saliency map details is not satisfactory, resulting in significant degradation of the salient object detection result. To overcome the above deficiencies and achieve better object detection results, we propose a salient object detection method based on feature optimization by neutrosophic set (NS) theory in this paper. First, prior object knowledge is built using foreground and background models, which include pixel-wise and super-pixel cues. Simultaneously, the feature maps are selected and extracted for feature computation, allowing the object and background features of the image to be separated as much as possible. Second, the salient object is obtained by fusing the features decomposed by the low-rank matrix recovery model with the object prior knowledge. Finally, for salient object detection, we present a novel mathematical description of neutrosophic set theory. To reduce the uncertainty of the obtained saliency map and then obtain good saliency detection results, the new NS theory is proposed. Extensive experiments on five public datasets demonstrate that the results are competitive and superior to previous state-of-the-art methods.

A Fusion Underwater Salient Object Detection Based on Multi-Scale Saliency and Spatial Optimization

Underwater images contain abundant information, but many challenges remain for underwater object detection tasks. Various salient object detection methods may encounter low detection precision, and the segmented map has an incomplete region of the target object. To deal with blurry underwater scenes and vague detection problems, a novel fusion underwater salient object detection algorithm (FUSOD) is proposed based on multi-scale saliency and spatial optimization. Firstly, an improved underwater color restoration was utilized to restore the color information for afterward color contrast saliency calculation. Secondly, a more accurate multi-scale fusion saliency map was obtained by fully considering both the global and local feature contrast information. Finally, the fusion saliency was optimized by the proposed spatial optimization method to enhance the spatial coherence. The proposed FUSOD algorithm may process turbid and complex underwater scenes and preserve a complete structure of the target object. Experimental results on the USOD dataset show that the proposed FUSOD algorithm can segment the salient object with a comparatively higher detection precision than the other traditional state-of-the-art algorithms. An ablation experiment showed that the proposed spatial optimization method increases the detection precision by 0.0325 scores in the F-Measure.