Salient Object Detection Models Research Articles

Lightweight cross-modal transformer for RGB-D salient object detection

Recently, Transformer-based RGB-D salient object detection (SOD) models have pushed the performance to a new level. However, they come at the cost of consuming abundant resources, including memory and power, thus hindering their real-life applications. To remedy this situation, a novel lightweight cross-modal Transformer (LCT) for RGB-D SOD will be presented in this paper. Specifically, LCT will first reduce its parameters and computational costs by employing a middle-level feature fusion structure and taking a lightweight Transformer as the backbone. Then, with the aid of Transformers, it will compensate for performance degradation by effectively capturing the cross-modal and cross-level complementary information from the multi-modal input images. To this end, a cross-modal enhancement and fusion module (CEFM) with a lightweight channel-wise cross attention block (LCCAB) will be designed to capture the cross-modal complementary information effectively but with fewer costs. A bi-directional multi-level feature interaction module (Bi-MFIM) with a lightweight spatial-wise cross attention block (LSCAB) will be designed to capture the cross-level complementary context information. By virtue of CEFM and Bi-MFIM, the performance degradation caused by parameter reduction can be well compensated, thus boosting the performances. By doing so, our proposed model has only 2.8M parameters with 7.6G FLOPs and runs at 66 FPS. Furthermore, experimental results on several benchmark datasets show that our proposed model can achieve competitive or even better results than other models. Our code will be released on https://github.com/nexiakele/lightweight-cross-modal-Transformer-LCT-for-RGB-D-SOD.

Multi-source information fusion attention network for weakly supervised salient object detection in optical remote sensing images

Due to the complex backgrounds and diverse attributes of salient objects in optical remote sensing images (RSIs), existing methods heavily rely on pixel-level annotations, with sparse annotations receiving limited attention. Furthermore, the restricted information in sparse annotations results in a significant performance gap between weakly supervised salient object detection models and fully supervised models. To address these limitations, this paper proposes a novel Multi-source Information Fusion Attention Network (MIFA-Net). MIFA-Net employs an encoder–decoder architecture comprising Boundary Detection Block (BDB), Region Activation Block (RAB), and Multi-source Information Fusion Block (MIFB). The encoder leverages a pre-trained VGG-16 model to extract basic features from input images. The BDB, supervised by pseudo-boundary masks, detects the boundaries of salient objects from the basic features, while the RAB, under the supervision of image-level class labels, activates the region information of salient objects. Acting as the decoder, the MIFB gradually integrates various information types, restoring high-quality optical remote sensing saliency maps through scribble annotations. Additionally, we introduce a deep supervision strategy and define a comprehensive loss function to constrain the training process. Experimental results on two benchmark datasets demonstrate that MIFA-Net significantly outperforms weakly supervised models, achieving an Sα of 0.905 and reducing M to 0.008 on EORSSD. This performance is comparable to, or even surpasses, fully supervised models.

Saliency detection for underwater moving object with sonar based on motion estimation and multi-trajectory analysis

Due to the limited detection range of optical sensors in the ocean, sonar stands out as the predominant method for detecting dynamic objects in the deep sea. Since there is stronger reflection underwater, the objects are more salient in sonar images compared to the background. In this paper, we propose a saliency detection framework for underwater moving object, which consists of three stages. In the first stage, we use optical flow to get rough global motion cues under background interference in unstable sonar platforms. In the second stage, we propose a trajectory analysis paradigm, which converts the motion cues into isolated connected domains for tracking, and then evaluates the trajectories to get the path of real target. In the third stage, we propose a salient object detection (SOD) model that predicts local saliency maps through feature fusion and multi-level supervision. The global saliency map is then obtained by remapping. Finally, extensive experiments conducted on eight sonar videos demonstrate that our proposed methodology outperforms fifteen other saliency object detection approaches.

Cascaded Aggregation Convolution Network for Salient Grain Pests Detection.

Pest infestation poses significant threats to grain storage due to pests' behaviors of feeding, respiration, excretion, and reproduction. Efficient pest detection and control are essential to mitigate these risks. However, accurate detection of small grain pests remains challenging due to their small size, high variability, low contrast, and cluttered background. Salient pest detection focuses on the visual features that stand out, improving the accuracy of pest identification in complex environments. Drawing inspiration from the rapid pest recognition abilities of humans and birds, we propose a novel Cascaded Aggregation Convolution Network (CACNet) for pest detection and control in stored grain. Our approach aims to improve detection accuracy by employing a reverse cascade feature aggregation network that imitates the visual attention mechanism in humans when observing and focusing on objects of interest. The CACNet uses VGG16 as the backbone network and incorporates two key operations, namely feature enhancement and feature aggregation. These operations merge the high-level semantic information and low-level positional information of salient objects, enabling accurate segmentation of small-scale grain pests. We have curated the GrainPest dataset, comprising 500 images showcasing zero to five or more pests in grains. Leveraging this dataset and the MSRA-B dataset, we validated our method's efficacy, achieving a structure S-measure of 91.9%, and 90.9%, and a weighted F-measure of 76.4%, and 91.0%, respectively. Our approach significantly surpasses the traditional saliency detection methods and other state-of-the-art salient object detection models based on deep learning. This technology shows great potential for pest detection and assessing the severity of pest infestation based on pest density in grain storage facilities. It also holds promise for the prevention and control of pests in agriculture and forestry.

A foreground-context dual-guided network for light-field salient object detection

Light-field salient object detection (SOD) has become an emerging trend as it records comprehensive information about natural scenes that can benefit salient object detection in various ways. However, salient object detection models with light-field data as input have not been thoroughly explored. The existing methods cannot effectively suppress the noise, and it is difficult to distinguish the foreground and background under challenging conditions including self-similarity, complex backgrounds, large depth of field, and non-Lambertian scenarios. In order to extract the feature of light-field images effectively and suppress the noise in light-field, in this paper, we propose a foreground and context dual guided network. Specifically, we design a global context extraction module (GCEM) and a local foreground extraction module (LFEM). GCEM is used to suppress global noise and roughly predict saliency maps. GCEM also can extract global context information from deep-level features to guide decoding process. By extracting local information from shallow-level, LFEM refines the prediction obtained by GCEM. In addition, we use RGB images to enhance the light-field images before the input GCEM. Experimental results show that our proposed method is effective in suppressing global noise and achieves better results when dealing with transparent objects and complex backgrounds. The experimental results show that the proposed method outperforms several other state-of-the-art methods on three light-field datasets.

MFUR-Net: Multimodal feature fusion and unimodal feature refinement for RGB-D salient object detection

RGB-D salient object detection aims to integrate multimodal feature information for accurate salient region localization. Despite the development of several RGB-D salient object detection models, existing methods face challenges in effectively fusing RGB with Depth features to exploit their complementary aspects. To address this challenge, this study introduces MFUR-Net, a network based on multimodal feature fusion and unimodal feature refinement. The contributions of this study are primarily threefold: First, a multimodal multilevel feature fusion module is proposed at the encoder stage to integrate multimodal and multilevel features, generating enhanced RGB-D features; Second, a multi-input feature aggregation module at the decoder stage is introduced, which incorporates the RGB and Depth feature streams into the RGB-D feature streams so that they collaborate with the RGB-D features to learn more discriminative information related to the salient object; Third, a unimodal saliency feature refinement module is introduced to refine saliency feature information across modalities and eliminate redundancy before the integration of feature streams into the decoder; With the gradual refinement of saliency features, MFUR-Net achieves accurate saliency map prediction at the decoder stage. This method has been validated through extensive experiments on seven recognized datasets, demonstrating significant advantages over existing state-of-the-art techniques in key performance metrics. The source code can be found in https://github.com/wangwei678/MFUR-Net.

Improving RGB-D salient object detection by addressing inconsistent saliency problems

RGB-D salient object detection (SOD) models based on a two-stream structure have achieved good performance in single-object scenes. In multi-object scenes, there is an inconsistent saliency problem between RGB modality and depth modality, which deteriorates the accuracy of subsequent fusion results. Inconsistent saliency is caused by the following issues: firstly, artifacts, missing depth values, and confusion in depth maps render depth modality unreliable, leading to increased reliance on RGB modality for results. Secondly, RGB modality and depth modality lack guidance in salient object detection. Thirdly, there is a lack of interaction between modalities. To address these issues, we first propose a depth recovery (DR) block to mitigate the negative effects of both the original and estimated depth maps. Next, we design the saliency detection (SD) block, which effectively guides each modality to focus on salient objects using semantic information. Meanwhile, SD combines multi-scale information to enhance the ability to detect multi-scale objects in each modality. Finally, a specific fusion block (SFB) is designed to fuse salient object information obtained from RGB and depth modalities. Quantitative and qualitative experiments demonstrate that our method achieves state-of-the-art (SOTA) performance among 10 methods.

Mutual Information Regularization for Weakly-Supervised RGB-D Salient Object Detection

In this paper, we present a weakly-supervised RGB-D salient object detection model via scribble supervision. Specifically, as a multimodal learning task, we focus on effective multimodal representation learning via inter-modal mutual information regularization. In particular, following the principle of disentangled representation learning, we introduce a mutual information upper bound with a mutual information minimization regularizer to encourage the disentangled representation of each modality for salient object detection. Based on our multimodal representation learning framework, we introduce an asymmetric feature extractor for our multimodal data, which is proven more effective than the conventional symmetric backbone setting. We also introduce multimodal variational auto-encoder as stochastic prediction refinement techniques, which takes pseudo labels from the first training stage as supervision and generates refined prediction. Experimental results on benchmark RGB-D salient object detection datasets verify both effectiveness of our explicit multimodal disentangled representation learning method and the stochastic prediction refinement strategy, achieving comparable performance with the state-of-the-art fully <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">supervised</i> models. Our code and data are available at: https://npucvr.github.io/MIRV/.

KD-SCFNet: Towards more accurate and lightweight salient object detection via knowledge distillation

Most existing salient object detection (SOD) models are difficult to apply due to the complex and huge model structures. Although some lightweight models are proposed, the accuracy is barely satisfactory. In this paper, we design a novel semantics-guided contextual fusion network (SCFNet) that focuses on the interactive fusion of multi-level features for accurate and efficient salient object detection. Furthermore, we apply knowledge distillation to SOD task and provide a sizeable dataset KD-SOD80K. In detail, we transfer the rich knowledge from a seasoned teacher to the untrained SCFNet through unlabeled images, enabling SCFNet to learn a strong generalization ability to detect salient objects more accurately. The knowledge distillation based SCFNet (KD-SCFNet) achieves comparable accuracy to the state-of-the-art heavyweight methods with less than 1M parameters and 174 FPS real-time detection speed. Extensive experiments demonstrate the robustness and effectiveness of the proposed distillation method and SOD framework.

TSVT: Token Sparsification Vision Transformer for robust RGB-D salient object detection

Visual transformer-based salient object detection (SOD) models have attracted increasing research attention. However, the existing transformer-based RGB-D SOD models usually operate on the full token sequences of RGB-D images and use an equal tokenization process to treat appearance and depth modalities, which leads to limited feature richness and inefficiency. To address these limitations, we present a novel token sparsification vision transformer architecture for RGB-D SOD, named TSVT, that explicitly extracts global-local multi-modality features with sparse tokens. The TSVT is an asymmetric encoder–decoder architecture with a dynamic sparse token encoder that adaptively selects and operates on sparse tokens, along with an multiple cascade aggregation decoder (MCAD) that predicts saliency results. Furthermore, we deeply investigate the differences and similarities between the appearance and depth modalities and develop an interactive diversity fusion module (IDFM) to integrate each pair of multi-modality tokens in different stages. Finally, to comprehensively evaluate the performance of the proposed model, we conduct extensive experiments on seven standard RGB-D SOD benchmarks in terms of five evaluation metrics. The experimental results reveal that the proposed model is more robust and effective than fifteen existing RGB-D SOD models. Moreover, the complexity of our model with the sparsification module is more than two times lower than that of the variant model without the dynamic sparse token module (DSTM).

Personalized Text-to-Image Model Enhancement Strategies: SOD Preprocessing and CNN Local Feature Integration

Recent advancements in text-to-image models have been substantial, generating new images based on personalized datasets. However, even within a single category, such as furniture, where the structures vary and the patterns are not uniform, the ability of the generated images to preserve the detailed information of the input images remains unsatisfactory. This study introduces a novel method to enhance the quality of the results produced by text-image models. The method utilizes mask preprocessing with an image pyramid-based salient object detection model, incorporates visual information into input prompts using concept image embeddings and a CNN local feature extractor, and includes a filtering process based on similarity measures. When using this approach, we observed both visual and quantitative improvements in CLIP text alignment and DINO metrics, suggesting that the generated images more closely follow the text prompts and more accurately reflect the input image’s details. The significance of this research lies in addressing one of the prevailing challenges in the field of personalized image generation: enhancing the capability to consistently and accurately represent the detailed characteristics of input images in the output. This method enables more realistic visualizations through textual prompts enhanced with visual information, additional local features, and unnecessary area removal using a SOD mask; it can also be beneficial in fields that prioritize the accuracy of visual data.

Snake species classification using deep learning techniques

Incorrect snake identification from the observable visual traits is a major reason of death resulting from snake bites. The classification of snake species has a significant role in determining the appropriate treatment without any delay, the delay may cause dangerous complications or lead to the death of the victim. The difficulty of classifying snakes by human lies in the variations of snake pattern based on geographic variation and age, the intraclass variance is high for specific classes and the interclass variance is low among others, and there may be two remarkably similar types in shape, with one being toxic and the other not. The limitation of the experts’ number in the herpetology and their geographical distribution leads us to the importance of using deep learning in the snake species classification. A model to classify snake species accately is proposed in this study. It is divided into two main processes, detecting the salient object by applying Salient Object Detection (SOD) model based on VGG16 architecture is the first process, the presence of snakes in places with a complex background led to the necessity of separating the salient object, then the classification model is applied with use of image augmentations parameters which improved the results. Four CNN models were used in the classification process including VGG16, ResNet50, MobileNetV2, and DenseNet121. Different experiments on 5,10,16,20, 22, and 45 number of classes and different models were conducted, and the model achieved unprecedented results. The results indicated that the VGG16, DenseNet121, and MobileNetV2 have achieved superior results in the same order from highest to lowest accuracy. The best accuracy is achieved using VGG16 architecture with accuracy 97.09% when using 45 number of classes.

DMNet: Dynamic Memory Network for RGB-D Salient Object Detection

RGB-D salient object detection aims to identify the most salient object using the color and depth information of images. Currently, most of the existing salient object detection models use the classical U-Net architecture, which uses up-sampling and short link decoding to exploit saliency cues after generating multi-level features by successive convolution and pooling operations. However, the multi-level encoding and progressive up-sampling decoding used by these models may lose some of the high-level semantic information during feature extraction and information fusion and some of the detailed information during image recovery, which in turn affects the quality of generated saliency maps. To solve these problems, we propose a dynamic memory network (DMNet) consisting of an interactive enhancement encoder and a dynamic memory decoder, which achieves high-precision detection of salient objects with a more complete and superior encoding and decoding strategy. Specifically, in the interactive enhancement encoder, we propose a feature interactive fusion module (FIFM), which can enhance intermediate-scale RGB features by fusing RGB features of three adjacent scales. Moreover, we design a depth-guided dense fusion module (DDFM) to fuse the features from RGB and depth modalities. In the dynamic memory decoder, we design a full-dimensional dynamic convolutional expansion module (FDCEM) to enhance the information representation capability of features at different scales. Furthermore, we also design a gated decoding module (GDM) to decode features of different scales and eliminate non-salient information. Extensive experimental results on STERE, SIP, NLPR, NJU2K, SSD, and DES datasets demonstrate that our model outperforms most of the state-of-the-art RGB-D SOD methods. In addition, the experimental results on three RGB-T datasets, VT821, VT1000, and VT5000 also show that our model can be effectively used for RGB-T SOD.

CoSOV1Net: A Cone- and Spatial-Opponent Primary Visual Cortex-Inspired Neural Network for Lightweight Salient Object Detection.

Salient object-detection models attempt to mimic the human visual system's ability to select relevant objects in images. To this end, the development of deep neural networks on high-end computers has recently achieved high performance. However, developing deep neural network models with the same performance for resource-limited vision sensors or mobile devices remains a challenge. In this work, we propose CoSOV1net, a novel lightweight salient object-detection neural network model, inspired by the cone- and spatial-opponent processes of the primary visual cortex (V1), which inextricably link color and shape in human color perception. Our proposed model is trained from scratch, without using backbones from image classification or other tasks. Experiments on the most widely used and challenging datasets for salient object detection show that CoSOV1Net achieves competitive performance (i.e., Fβ=0.931 on the ECSSD dataset) with state-of-the-art salient object-detection models while having a low number of parameters (1.14 M), low FLOPS (1.4 G) and high FPS (211.2) on GPU (Nvidia GeForce RTX 3090 Ti) compared to the state of the art in lightweight or nonlightweight salient object-detection tasks. Thus, CoSOV1net has turned out to be a lightweight salient object-detection model that can be adapted to mobile environments and resource-constrained devices.

LDWS-net: A learnable deep wavelet scattering network for RGB salient object detection

In the salient object detection task, convolutional neural network (CNN) based models have been extensively used. However, preserving a variety of boundary features of objects is equally important while detecting the salient objects. Detecting salient objects with poor boundaries have a significant detrimental effect on the salient object detection (SOD) models' robustness and accuracy. The proposed method leverages a unique and novel edge-directed salient object detection network, which combines wavelet scattering network features with CNN-based features. This integration enables the network to capture both textural and high-level semantic information, leading to improved SOD performance. Additionally, a learnable wavelet scattering network allows for the efficient collection and preservation of textural aspects of objects. This network is seamlessly embedded into the encoder section of the proposed architecture, enhancing the discriminative power of the model. Furthermore, a weighted feature integration module (WFIM) is proposed in the decoder section to adaptively integrate linked nearby features by evaluating their relevance, resulting in improved representation and discrimination capabilities. Extensive testing on well-known benchmark SOD datasets demonstrates that the proposed LDWS-Net outperforms state-of-the-art techniques, exhibiting accurate identification of salient objects and efficient detection of their edges.

Motion Context guided Edge-preserving network for video salient object detection

Video salient object detection targets at extracting the most conspicuous objects in a video sequence, which facilitate various video processing tasks, e.g., video compression, video recognition, etc. Although remarkable progress has been made for video salient object detection, most existing methods still suffer from coarse edge boundaries which may hinder their usage in real-world applications. To alleviate this problem, in this paper, we propose a Motion Context guided Edge-preserving network (MCE-Net) model for video salient object detection. MCE-Net can generate temporally consistent salient edges, which are then leveraged to refine the salient object regions completely and uniformly. The core innovation in MCE-Net is an Asymmetric Cross-Reference Module (ACRM), which is designed to exploit the cross-modal complementarity between spatial structure and motion flow, facilitating robust salient object edge extraction. To leverage the extracted edge features for salient object refinement, we fuse them with multi-level spatial–temporal embeddings in a paralleled guidance manner, generating the final saliency results. The proposed method is end-to-end trainable and the edge annotations are generated automatically from ground truth saliency maps. Experimental evaluations on five widely-used benchmarks demonstrate that our proposed method can achieve superior performance to other outstanding methods. Moreover, the experimental results indicate that our method can preserve salient objects with clear boundary structures in video sequences.

TBINet: fabric defect detection based on a top-down and bottom-up inference network

A key aspect of textile production is the use of automatic defect detection methods for quality control, which is a critical part of the current fabric defect detection system. Benefiting from the explosive development of convolutional neural networks, it has been proved to be feasible to apply it to salient object detection model in fabric defect detection, but how to learn more robust features and better feature fusion for saliency are still complicated tasks. This article presents a novel saliency detection method for fabric defect detection based on both top-down and bottom-up saliency inference combined with two-way information. The top-down process is to infer high-level saliency by gradually using higher-level and richer semantic features on the basis of the backbone. Then a self-fusion enhanced representation module is proposed, which simulates the parallel processing mechanism with residual connection in a top-down manner to generate robust features and effectively characterize the texture features of complex fabrics. In addition, in the down-top process, the interactive feature fusion module is designed, which is used for coarse-to-fine saliency estimation, where the high-level saliency is gradually integrated with finer lower-level features to obtain a fine-grained result. Finally, fabric defect detection is localized by segmenting the generated saliency map. Extensive experimental results on two kinds of fabric image datasets have demonstrated that the proposed TBINet can localize the defect region with high accuracy, and perform favorably against seven state-of-the-art methods on eight widely tested metrics.

Salient Object Detection From Unlabeled Images

Salient object detection via deep neural networks usually needs a large amount of images with human annotation. To avoid laborious and consuming annotation, we propose a robust unsupervised salient object detection method with three stages in this work. Our method first uses unlabeled data to generate an activation map which indicates the coarse location of object. Then one scribble generation method based on the activation map is proposed, which provides foreground scribble and background scribble with high confidence. Finally, a salient object detection model is trained with the supervision of the generated scribble. Performance comparison is carried on four public datasets, showing that our method significantly outperforms state-of-the-art unsupervised methods, and also achieves better performance than some weakly supervised methods.

Absolute and Relative Depth-Induced Network for RGB-D Salient Object Detection

Detecting salient objects in complicated scenarios is a challenging problem. Except for semantic features from the RGB image, spatial information from the depth image also provides sufficient cues about the object. Therefore, it is crucial to rationally integrate RGB and depth features for the RGB-D salient object detection task. Most existing RGB-D saliency detectors modulate RGB semantic features with absolution depth values. However, they ignore the appearance contrast and structure knowledge indicated by relative depth values between pixels. In this work, we propose a depth-induced network (DIN) for RGB-D salient object detection, to take full advantage of both absolute and relative depth information, and further, enforce the in-depth fusion of the RGB-D cross-modalities. Specifically, an absolute depth-induced module (ADIM) is proposed, to hierarchically integrate absolute depth values and RGB features, to allow the interaction between the appearance and structural information in the encoding stage. A relative depth-induced module (RDIM) is designed, to capture detailed saliency cues, by exploring contrastive and structural information from relative depth values in the decoding stage. By combining the ADIM and RDIM, we can accurately locate salient objects with clear boundaries, even from complex scenes. The proposed DIN is a lightweight network, and the model size is much smaller than that of state-of-the-art algorithms. Extensive experiments on six challenging benchmarks, show that our method outperforms most existing RGB-D salient object detection models.

FESNet: Frequency-Enhanced Saliency Detection Network for Grain Pest Segmentation

As insect infestation is the leading factor accounting for nutritive and economic losses in stored grains, it is important to detect the presence and number of insects for the sake of taking proper control measures. Inspired by the human visual attention mechanism, we propose a U-net-like frequency-enhanced saliency (FESNet) detection model, resulting in the pixelwise segmentation of grain pests. The frequency clues, as well as the spatial information, are leveraged to enhance the detection performance of small insects from the cluttered grain background. Firstly, we collect a dedicated dataset, GrainPest, with pixel-level annotation after analyzing the image attributes of the existing salient object detection datasets. Secondly, we design a FESNet with the discrete wavelet transformation (DWT) and the discrete cosine transformation (DCT), both involved in the traditional convolution layers. As current salient object detection models will reduce the spatial information with pooling operations in the sequence of encoding stages, a special branch of the discrete wavelet transformation (DWT) is connected to the higher stages to capture accurate spatial information for saliency detection. Then, we introduce the discrete cosine transform (DCT) into the backbone bottlenecks to enhance the channel attention with low-frequency information. Moreover, we also propose a new receptive field block (NRFB) to enlarge the receptive fields by aggregating three atrous convolution features. Finally, in the phase of decoding, we use the high-frequency information and aggregated features together to restore the saliency map. Extensive experiments and ablation studies on our dataset, GrainPest, and open dataset, Salient Objects in Clutter (SOC), demonstrate that the proposed model performs favorably against the state-of-the-art model.