Salient Object Detection Research Articles

Computer vision-based intelligent detection method for the residual capability of energy dissipators in flexible protection systems

A residual capability intelligent detection method based on computer vision is proposed to address the issues of low efficiency, poor accuracy, and high danger in manual measurement of energy dissipators in flexible protection systems. The proposed method first establishes a binary semantic segmentation dataset for energy dissipators and trains a salient object detection deep neural network to segment the energy dissipator binary map; Then, it uses morphological image processing and contour detection to calculate the residual capability automatically. U2-Net, U2-Netp, and BASENet were trained and compared by a dataset with 500 ring-type energy dissipator images. The proposed method was validated through a quasi-static tensile test and a full-scale impact test. Compared with the most accurate integration calculation method, the error of the proposed method does not exceed 3 %, and the efficiency is improved by about 25 times compared to the most commonly used manual detection method.

Salient Object Detection From Arbitrary Modalities.

Toward desirable saliency prediction, the types and numbers of inputs for a salient object detection (SOD) algorithm may dynamically change in many real-life applications. However, existing SOD algorithms are mainly designed or trained for one particular type of inputs, failing to be generalized to other types of inputs. Consequentially, more types of SOD algorithms need to be prepared in advance for handling different types of inputs, raising huge hardware and research costs. Differently, in this paper, we propose a new type of SOD task, termed Arbitrary Modality SOD (AM SOD). The most prominent characteristics of AM SOD are that the modality types and modality numbers will be arbitrary or dynamically changed. The former means that the inputs to the AM SOD algorithm may be arbitrary modalities such as RGB, depths, or even any combination of them. While, the latter indicates that the inputs may have arbitrary modality numbers as the input type is changed, e.g. single-modality RGB image, dual-modality RGB-Depth (RGB-D) images or triple-modality RGB-Depth-Thermal (RGB-D-T) images. Accordingly, a preliminary solution to the above challenges, i.e. a modality switch network (MSN), is proposed in this paper. In particular, a modality switch feature extractor (MSFE) is first designed to extract discriminative features from each modality effectively by introducing some modality indicators, which will generate some weights for modality switching. Subsequently, a dynamic fusion module (DFM) is proposed to adaptively fuse features from a variable number of modalities based on a novel Transformer structure. Finally, a new dataset, named AM-XD, is constructed to facilitate research on AM SOD. Extensive experiments demonstrate that our AM SOD method can effectively cope with changes in the type and number of input modalities for robust salient object detection. Our code and AM-XD dataset will be released on https://github.com/nexiakele/AMSODFirst.

A Strip Steel Surface Defect Salient Object Detection Based on Channel, Spatial and Self-Attention Mechanisms

Strip steel is extensively utilized in industries such as automotive manufacturing and aerospace due to its superior machinability, economic benefits, and adaptability. However, defects on the surface of steel strips, such as inclusions, patches, and scratches, significantly affect the performance and service life of the product. Therefore, the salient object detection of surface defects on strip steel is crucial to ensure the quality of the final product. Many factors, such as the low contrast of surface defects on strip steel, the diversity of defect types, complex texture structures, and irregular defect distribution, hinder existing detection technologies from accurately identifying and segmenting defect areas against complex backgrounds. To address the above problems, we propose a novel detector called S3D-SOD for the salient object detection of strip steel surface defects. For the encoding stage, a residual self-attention block is proposed to explore semantic information cues of high-level features to locate and guide low-level feature information. In addition, we apply a general residual channel and spatial attention to low-level features, enabling the model to adaptively focus on the key channels and spatial areas of feature maps with high resolutions, thereby enhancing the encoder features and accelerating the convergence of the model. For the decoding stage, a simple residual decoder block with an upsampling operation is proposed to realize the integration and interaction of feature information between different layers. Here, the simple residual decoder block is used for feature integration due to the following observation: backbone networks like ResNet and the Swin Transformer, after being pretrained on the large dataset ImageNet and then fine-tuned on a smaller dataset for strip steel surface defects, are capable of extracting feature maps that contain both general image features and the specific characteristics required for the salient object detection of strip steel surface defects. The experimental results on the SD-saliency-900 dataset show that S3D-SOD is better than advanced methods, and it has strong generalization ability and robustness.

SOD‐diffusion: Salient Object Detection via Diffusion‐Based Image Generators

AbstractSalient Object Detection (SOD) is a challenging task that aims to precisely identify and segment the salient objects. However, existing SOD methods still face challenges in making explicit predictions near the edges and often lack end‐to‐end training capabilities. To alleviate these problems, we propose SOD‐diffusion, a novel framework that formulates salient object detection as a denoising diffusion process from noisy masks to object masks. Specifically, object masks diffuse from ground‐truth masks to random distribution in latent space, and the model learns to reverse this noising process to reconstruct object masks. To enhance the denoising learning process, we design an attention feature interaction module (AFIM) and a specific fine‐tuning protocol to integrate conditional semantic features from the input image with diffusion noise embedding. Extensive experiments on five widely used SOD benchmark datasets demonstrate that our proposed SOD‐diffusion achieves favorable performance compared to previous well‐established methods. Furthermore, leveraging the outstanding generalization capability of SOD‐diffusion, we applied it to publicly available images, generating high‐quality masks that serve as an additional SOD benchmark testset.

Attenuative Unified U-Net with Conditional PatchMatch Algorithm for Edge Preserving Salient Object Detection

The rise of Deep Neural Networks (DNNs) has significantly boosted salient object detection in computer vision tasks. However, downsampling methods like striding and pooling often introduce bluish artifacts near edges, hindering detection accuracy. To address this, the "Attenuative Unified U-Net with Conditional PatchMatch Algorithm" is proposed. The method employs Gaussian smoothing for noise reduction and depth refinement in preprocessing. In existing methods, the Edge-preserving salient object detection struggles with cluttered backgrounds and connected objects, like a bird on a rock, making it difficult to accurately distinguish edges and salient regions due to shared boundaries. Hence, an Attenuative Unified Backpropagated Entropy UNet is proposed, which integrates an Attention Mechanism that enhance feature map spatial weight assignment, emphasizing features or regions that are judged most relevant for edge and salient region detection. Then, the Cascaded Laplace pyramid is incorporated into the Unified U-Net’s design for multi-scale information processing, capturing contextual details effectively. The Backpropagated Entropy Loss Function is then created to ensure accurate and confident fused image output. The depth map, edge map, and salient area map are combined in the U-Net’s final output layer to create an overall fused image. Next, to addresses depth discontinuity problem a noteworthy novel idea the Conditional PatchMatch Random Field Algorithm (CPMRF), which combines PatchMatch efficiency and CRFs’ contextual modeling. PatchMatch iteratively propagates matches from neighboring patches to find similar patches in the remaining portion of the image and Conditional Random Field (CRF) is the next step, which improves the PatchMatch results by taking into account the connections between the matched patches and their neighboring areas. Hence, the experimental outcomes of the proposed model effectively show the improved detection of edge in salient object detection with better accuracy, precision, recall, MaxF, and F1 score with minimized MAE.

Salient object detection with bayesian inference based on radar and camera fusion used in UAV obstacle avoidance

Most existing salient object detection methods are sensitive to background noise and rely on prior information in UAV obstacle avoidance applications despite detection methods witnessing rapid progress. In this paper, we propose an efficient framework for salient object detection based on radar-camera fusion and iterative Bayesian optimization. A rough salient object (RSO) image is first built through radar and camera calibration. Next, the RSO image is used to calculate the edge response based on the receptive field mechanism of the primary visual cortex to construct the contour image. Finally, the above two images and the 2D Gaussian distribution are jointly integrated into an iterative Bayesian optimization scheme to get the final salient object image. Different from typical detection methods, this method suppresses background noise by filtering out irrelevant pixels using fusion information. The Bayesian inference framework’s detection performance is improved by precise spatial prior, consisting of optimized contour and RSO images. Experimental results indicate that the presented algorithm performs well against state-of-the-art salient object detection methods on several reference datasets in different evaluation metrics.

R-Net: Recursive decoder with edge refinement network for salient object detection

Fully convolutional neural (FCN) networks based on encoder-decoder structures are adopted for salient object detection (SOD) and achieve state-of-the-art performance. However, most of the previous works aggregate multilevel features layer by layer and ignore long-range dependencies of spatial context, leading to imbalanced feature processes and context separation. In addition, the blurry edges obtained from previous SOD methods inevitably influence the detection accuracy. To handle these problems, this study proposes an effective recursion-based SOD network R-Net adopting a partial cascade structure, which contains a recursive decoder module (RDM), a long-range dependency module (LRDM), and an edge refinement module (ERM). The RDM is composed of three subdecoders, which can scale different level features to the same size through recursive pooling and recursive upsampling for full-scale feature fusion. The LRDM bridges the encoder and decoder by weighting multilevel features to establish the long-range feature dependency of the spatial context. And the ERM attached to the decoder introduces the shallow reference feature to refine the blurry edges for obtaining delicate detection results. The SOD experimental results on DUTS-TE, HKU-IS, PASCAL-S, ECSSD, and DUT-OMRON datasets demonstrate that the proposed R-Net is more robust under different complex scenes compared with some state-of-the-art methods. The source codes are available at https://github.com/ZEROICEWANG/R-Net.

Feature differences reduction and specific features preserving network for RGB-T salient object detection

In RGB-T salient object detection, effective utilization of the different characteristics of RGB and thermal modalities is essential to achieve accurate detection. Most of the previous methods usually only focus on reducing the differences between modalities, which may ignore the specific features that are crucial for salient object detection, leading to suboptimal results. To address the above issue, an RGB-T SOD network that simultaneously considers the reduction of modality differences and the preservation of specific features is proposed. Specifically, we construct a modality differences reduction and specific features preserving module (MDRSFPM) which aims to bridge the gap between modalities and enhance the specific features of each modality. In MDRSFPM, the dynamic vector generated by the interaction of RGB and thermal features is used to reduce modality differences, and then a dual branch is constructed to deal with the RGB and thermal modalities separately, employing a combination of channel-level and spatial-level operations to preserve their respective specific features. In addition, a multi-scale global feature enhancement module (MGFEM) is proposed to enhance global contextual information to provide guidance information for the subsequent decoding stage, so that the model can more easily localize the salient objects. Furthermore, our approach includes a fully fusion and gate module (FFGM) that utilises dynamically generated importance maps to selectively filter and fuse features during the decoding process. Extensive experiments demonstrate that our proposed model surpasses other state-of-the-art models on three publicly available RGB-T datasets remarkably. Our code will be released at https://github.com/JOOOOKII/FRPNet.

Multi-Prior Driven Network for RGB-D Salient Object Detection

Most existing RGB-D salient object detection (SOD) methods rely on high-quality depth images. However, their performance is limited when processing low-quality depth maps. This paper exploits more complementary image priors to guide the model to learn on variable depth maps, and a novel multi-prior driven network called MPDNet is proposed for RGB-D SOD. MPDNet utilizes four processing pipelines to process RGB images and other priors, which include an RGB image processing pipeline, a depth map processing pipeline, a fine-grained and gradient prior processing pipeline, and an edge learning pipeline. Specifically, fine-grained and gradient priors are input to the same processing pipeline. For the depth maps, fine-grained and gradient priors, a prior channel attention module utilizes the channel attention mechanism to filter noises and highlights the salient cues. The RGB image processing pipeline uses a multi-feature progressive enhancement module to fuse and enhance features from depth maps. And a multi-feature prediction decoder decodes initial salient masks. In the edge learning pipeline, edge prior serves as an edge label and is captured by an edge capture module. Finally, the clear salient masks are obtained by fusing the salient information from the four pipelines. The experimental results on six benchmarks indicate that the proposed method outperforms thirteen state-of-the-art methods in six evaluation metrics.

Lightweight cross-modal transformer for RGB-D salient object detection

Recently, Transformer-based RGB-D salient object detection (SOD) models have pushed the performance to a new level. However, they come at the cost of consuming abundant resources, including memory and power, thus hindering their real-life applications. To remedy this situation, a novel lightweight cross-modal Transformer (LCT) for RGB-D SOD will be presented in this paper. Specifically, LCT will first reduce its parameters and computational costs by employing a middle-level feature fusion structure and taking a lightweight Transformer as the backbone. Then, with the aid of Transformers, it will compensate for performance degradation by effectively capturing the cross-modal and cross-level complementary information from the multi-modal input images. To this end, a cross-modal enhancement and fusion module (CEFM) with a lightweight channel-wise cross attention block (LCCAB) will be designed to capture the cross-modal complementary information effectively but with fewer costs. A bi-directional multi-level feature interaction module (Bi-MFIM) with a lightweight spatial-wise cross attention block (LSCAB) will be designed to capture the cross-level complementary context information. By virtue of CEFM and Bi-MFIM, the performance degradation caused by parameter reduction can be well compensated, thus boosting the performances. By doing so, our proposed model has only 2.8M parameters with 7.6G FLOPs and runs at 66 FPS. Furthermore, experimental results on several benchmark datasets show that our proposed model can achieve competitive or even better results than other models. Our code will be released on https://github.com/nexiakele/lightweight-cross-modal-Transformer-LCT-for-RGB-D-SOD.

Bridging spatiotemporal feature gap for video salient object detection

The mutual transfer of spatiotemporal features is the main challenge for the two-stream video salient object detection. Current methods adopt the spatiotemporal feature interaction to achieve it. However, these methods still have two issues: modal feature gap and layer feature gap. To address these, we propose a Bridging Spatiotemporal feature Gap Network (BSGNet) with a global correspondence interaction and gate filtering (GCGF) module, a global-local distribution consistency (GLDC) module, and a modality-layer feature fusion framework (MLFF). Compared with previous works, BSGNet not only explores more effective interaction by GCGF, but also bridges modality and layer feature gaps by GLDC and MLFF. Firstly, GCGF achieves the spatiotemporal feature interaction by modeling intra-modal and inter-modal global correspondences. Besides, GCGF employs a gate mechanism to control the proportion of message transfer between appearance and motion information, which characterizes the contribution provided by spatiotemporal features. Secondly, at both global and local levels, GLDC pushes the spatiotemporal feature distribution between same scenes, and pulls the spatiotemporal feature distribution between different scenes. This can enhance the distribution consistency to align spatiotemporal features and bridge modal feature gap. Finally, MLFF designs an inter-modal and inter-layer feature fusion framework to bridge the layer feature gap brought by the different modalities and different receptive fields. Extensive experiments on five benchmarks reveal that our BSGNet outperforms state-of-the-arts.

An improved algorithm for salient object detection of microscope based on U2-Net.

With the rapid advancement of modern medical technology, microscopy imaging systems have become one of the key technologies in medical image analysis. However, manual use of microscopes presents issues such as operator dependency, inefficiency, and time consumption. To enhance the efficiency and accuracy of medical image capture and reduce the burden of subsequent quantitative analysis, this paper proposes an improved microscope salient object detection algorithm based on U2-Net, incorporating deep learning technology. The improved algorithm first enhances the network's key information extraction capability by incorporating the Convolutional Block Attention Module (CBAM) into U2-Net. It then optimizes network complexity by constructing a Simple Pyramid Pooling Module (SPPM) and uses Ghost convolution to achieve model lightweighting. Additionally, data augmentation is applied to the slides to improve the algorithm's robustness and generalization. The experimental results show that the size of the improved algorithm model is 72.5MB, which represents a 56.85% reduction compared to the original U2-Net model size of 168.0MB. Additionally, the model's prediction accuracy has increased from 92.24 to 97.13%, providing an efficient means for subsequent image processing and analysis tasks in microscopy imaging systems.

DMGNet: Depth mask guiding network for RGB-D salient object detection

Though depth images can provide supplementary spatial structural cues for salient object detection (SOD) task, inappropriate utilization of depth features may introduce noisy or misleading features, which may greatly destroy SOD performance. To address this issue, we propose a depth mask guiding network (DMGNet) for RGB-D SOD. In this network, a depth mask guidance module (DMGM) is designed to pre-segment the salient objects from depth images and then create masks using pre-segmented objects to guide the RGB subnetwork to extract more discriminative features. Furthermore, a feature fusion pyramid module (FFPM) is employed to acquire more informative fused features using multi-branch convolutional channels with varying receptive fields, further enhancing the fusion of cross-modal features. Extensive experiments on nine benchmark datasets demonstrate the effectiveness of the proposed network.

Saliency detection for underwater moving object with sonar based on motion estimation and multi-trajectory analysis

Due to the limited detection range of optical sensors in the ocean, sonar stands out as the predominant method for detecting dynamic objects in the deep sea. Since there is stronger reflection underwater, the objects are more salient in sonar images compared to the background. In this paper, we propose a saliency detection framework for underwater moving object, which consists of three stages. In the first stage, we use optical flow to get rough global motion cues under background interference in unstable sonar platforms. In the second stage, we propose a trajectory analysis paradigm, which converts the motion cues into isolated connected domains for tracking, and then evaluates the trajectories to get the path of real target. In the third stage, we propose a salient object detection (SOD) model that predicts local saliency maps through feature fusion and multi-level supervision. The global saliency map is then obtained by remapping. Finally, extensive experiments conducted on eight sonar videos demonstrate that our proposed methodology outperforms fifteen other saliency object detection approaches.

High-precision integral imaging 3D salient object detection and reconstruction with texture features based on E2E-TransGAN

Under the framework of integral imaging, a high-precision 3D salient object detection and high-quality texture features reconstruction method is proposed by using the element to element-transformer and generative adversarial network (E2E-TransGAN). The proposed method fully exploits the global salient clues of the element image array to establish cross-regional element image dependency, thereby the accuracy of the salient prediction improved. Meanwhile, our proposed E2E-TransGAN algorithm stratifies the binary salient object area and the grayscale salient object edge, effectively addressing salient region prediction errors, obvious background noise, and color loss. The experimental results show that the detection accuracy of the proposed method improves to an average of 90% across various evaluation metrics. Compared with other methods, the proposed method can reconstruct 3D images with texture features, which have high values of PSNR and SSIM, at 30 and 0.95 on average, respectively. We have experimentally verified that the proposed method achieves precise 3D object salient detection and restores the detailed textures and depth information of 3D salient objects. The proposed method provides strong support for efficient and accurate medical image analysis and simplifies subsequent computer vision tasks.

Dual-Stream Feature Collaboration Perception Network for Salient Object Detection in Remote Sensing Images

As the core technology of artificial intelligence, salient object detection (SOD) is an important approach to improve the analysis efficiency of remote sensing images by intelligently identifying key areas in images. However, existing methods that rely on a single strategy, convolution or Transformer, exhibit certain limitations in complex remote sensing scenarios. Therefore, we developed a Dual-Stream Feature Collaboration Perception Network (DCPNet) to enable the collaborative work and feature complementation of Transformer and CNN. First, we adopted a dual-branch feature extractor with strong local bias and long-range dependence characteristics to perform multi-scale feature extraction from remote sensing images. Then, we presented a Multi-path Complementary-aware Interaction Module (MCIM) to refine and fuse the feature representations of salient targets from the global and local branches, achieving fine-grained fusion and interactive alignment of dual-branch features. Finally, we proposed a Feature Weighting Balance Module (FWBM) to balance global and local features, preventing the model from overemphasizing global information at the expense of local details or from inadequately mining global cues due to excessive focus on local information. Extensive experiments on the EORSSD and ORSSD datasets demonstrated that DCPNet outperformed the current 19 state-of-the-art methods.

A novel multiscale cGAN approach for enhanced salient object detection in single haze images

In computer vision, image dehazing is a low-level task that employs algorithms to analyze and remove haze from images, resulting in haze-free visuals. The aim of Salient Object Detection (SOD) is to locate the most visually prominent areas in images. However, most SOD techniques applied to visible images struggle in complex scenarios characterized by similarities between the foreground and background, cluttered backgrounds, adverse weather conditions, and low lighting. Identifying objects in hazy images is challenging due to the degradation of visibility caused by atmospheric conditions, leading to diminished visibility and reduced contrast. This paper introduces an innovative approach called Dehaze-SOD, a unique integrated model that addresses two vital tasks: dehazing and salient object detection. The key novelty of Dehaze-SOD lies in its dual functionality, seamlessly integrating dehazing and salient object identification into a unified framework. This is achieved using a conditional Generative Adversarial Network (cGAN) comprising two distinct subnetworks: one for image dehazing and another for salient object detection. The first module, designed with residual blocks, Dark Channel Prior (DCP), total variation, and the multiscale Retinex algorithm, processes the input hazy images. The second module employs an enhanced EfficientNet architecture with added attention mechanisms and pixel-wise refinement to further improve the dehazing process. The outputs from these subnetworks are combined to produce dehazed images, which are then fed into our proposed encoder–decoder framework for salient object detection. The cGAN is trained with two modules working together: the generator aims to produce haze-free images, whereas the discriminator distinguishes between the generated haze-free images and real haze-free images. Dehaze-SOD demonstrates superior performance compared to state-of-the-art dehazing methods in terms of color fidelity, visibility enhancement, and haze removal. The proposed method effectively produces high-quality, haze-free images from various hazy inputs and accurately detects salient objects within them. This makes Dehaze-SOD a promising tool for improving salient object detection in challenging hazy conditions. The effectiveness of our approach has been validated using benchmark evaluation metrics such as mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and structural similarity index measure (SSIM).

Advancing in RGB-D Salient Object Detection: A Survey

The human visual system can rapidly focus on prominent objects in complex scenes, significantly enhancing information processing efficiency. Salient object detection (SOD) mimics this biological ability, aiming to identify and segment the most prominent regions or objects in images or videos. This reduces the amount of data needed to process while enhancing the accuracy and efficiency of information extraction. In recent years, SOD has made significant progress in many areas such as deep learning, multi-modal fusion, and attention mechanisms. Additionally, it has expanded in real-time detection, weakly supervised learning, and cross-domain applications. Depth images can provide three-dimensional structural information of a scene, aiding in a more accurate understanding of object shapes and distances. In SOD tasks, depth images enhance detection accuracy and robustness by providing additional geometric information. This additional information is particularly crucial in complex scenes and occlusion situations. This survey reviews the substantial advancements in the field of RGB-Depth SOD, with a focus on the critical roles played by attention mechanisms and cross-modal fusion methods. It summarizes the existing literature, provides a brief overview of mainstream datasets and evaluation metrics, and quantitatively compares the discussed models.

DBTSF-VSOD: a decision-based two-stage framework for video salient object detection

DBTSF-VSOD: a decision-based two-stage framework for video salient object detection

Light field salient object detection network based on feature enhancement and mutual attention

Light field salient object detection network based on feature enhancement and mutual attention