Saliency Detection Performance Research Articles

WBNet: Weakly-supervised salient object detection via scribble and pseudo-background priors

Weakly supervised salient object detection (WSOD) methods endeavor to boost sparse labels to get more salient cues in various ways. Among them, an effective approach is using pseudo labels from multiple unsupervised self-learning methods, but inaccurate and inconsistent pseudo labels could ultimately lead to detection performance degradation. To tackle this problem, we develop a new multi-source WSOD framework, WBNet, that can effectively utilize pseudo-background (non-salient region) labels combined with scribble labels to obtain more accurate salient features. We first design a comprehensive salient pseudo-mask generator from multiple self-learning features. Then, we pioneer the exploration of generating salient pseudo-labels via point-prompted and box-prompted Segment Anything Models (SAM). Then, WBNet leverages a pixel-level Feature Aggregation Module (FAM), a mask-level Transformer-decoder (TFD), and an auxiliary Boundary Prediction Module (EPM) with a hybrid loss function to handle complex saliency detection tasks. Comprehensively evaluated with state-of-the-art methods on five widely used datasets, the proposed method significantly improves saliency detection performance. The code and results are publicly available at https://github.com/yiwangtz/WBNet.

Application of cultural elements of dunhuang murals in landscape design based on mean shift algorithm extraction

Although picture extraction is challenging, the murals at Dunhuang are historically significant and offer rich content. The work suggests an image segmentation model based on the Mean Shift algorithm and an area salience prioritisation model to extract the cultural aspects in the Dunhuang murals for landscape design. First, an image segmentation model based on the Mean Shift algorithm is established, and then a region salience value calculation method and a region prioritisation method are designed to establish a region salience prioritisation model. The outcomes showed that a segmentation model built using the Mean Shift algorithm in the study processed a 405175 image with a processing time of 3.18 seconds, an edge integrity rate of 88.9%, an accuracy rate of 87.4%, an F-value of 88.7%, and a total of 302 regions. The segmented Dunhuang image featured few noise points and a distinct shape. Salient region transfer path is more regular and more in line with the human visual transfer mechanism thanks to the research design of the region saliency value calculation method, which also improves saliency detection performance. The highest correct rate when dividing the image is 0.97, the highest check rate is 0.8, and the highest F1 value is 1. In conclusion, the study’s methodology has some favourable implications for landscape design and may be effectively used to extract cultural components from photographs.

Underwater image enhancement method based on golden jackal optimization

The propagation of light underwater is affected by water absorption and scattering, resulting in poor visibility of obtained underwater images, and exhibiting degradation phenomena such as color deviation and low contrast. To improve the quality and visibility of underwater images, a method of underwater image enhancement based on Golden Jackal optimization is proposed. The algorithm consists of two processing stages. The first stage is mainly processed in the RGB color space. Based on the attenuation characteristics of underwater color channels, a constructed color compensation and correction strategy is used to eliminate color deviation caused by water absorption. In second stage, it is mainly processed in the Lab color space, using the Golden Jackal optimization algorithm and fitness function to search the optimal parameters of transformation function to improve the image contrast by optimizing the grey intensity of L component and its histogram distribution. The feasibility and effectiveness of the proposed method have been verified through testing and analysis of much underwater degraded image data. In subjective and objective analysis with multiple state-of-the-art underwater image clarity methods, the proposed method performs excellent in color restoration and detail texture enhancement. Further experiments have shown that the proposed method has strong robustness and universality. It is not only suitable for enhancing various underwater scene images, but also for enhancing environmental images with fog, low lighting, and sandstorms. Meanwhile, the proposed method can effectively improve the performance of underwater image segmentation, keypoint detection, and saliency detection.

Cross-modal hierarchical interaction network for RGB-D salient object detection

How to effectively exchange and aggregate the information of multiple modalities (e.g. RGB image and depth map) is a big challenge in the RGB-D salient object detection community. To address this problem, in this paper, we propose a cross-modal Hierarchical Interaction Network (HINet), which boosts the salient object detection by excavating the cross-modal feature interaction and progressively multi-level feature fusion. To achieve it, we design two modules: cross-modal information exchange (CIE) module and multi-level information progressively guided fusion (PGF) module. Specifically, the CIE module is proposed to exchange the cross-modal features for learning the shared representations, as well as the beneficial feedback to facilitate the discriminative feature learning of different modalities. Besides, the PGF module is designed to aggregate the hierarchical features progressively with the reverse guidance mechanism, which employs the high-level feature fusion to guide the low-level feature fusion and thus improve the saliency detection performance. Extensive experiments show that our proposed model significantly outperforms the existing nine state-of-the-art models on five challenging benchmark datasets. Codes and results are available at: https://github.com/RanwanWu/HINet.

Unidirectional RGB-T salient object detection with intertwined driving of encoding and fusion

The U-shaped encoder–decoder architecture based on CNNs has been rooted in salient object detection (SOD) tasks, and it have revealed two drawbacks while driving the rapid development of saliency detection. (1) The inherent characteristics of CNNs dictate that it is difficult to learn long-range dependencies and model global correlations. (2) For the common purpose of improving the performance of saliency detection, the encoder and decoder should complement each other and work together. However, the existing encoder–decoder architecture treats encoder and decoder independently of each other. Specifically, the encoder is responsible for extracting features and the decoder fuses multi-level or multi-modal features to produce prediction maps. That is, the encoder alone needs to be responsible for the decoder, while the valuable information after the decoder fusion will not facilitate feature extraction. Therefore, we propose a unidirectional RGB-T salient object detection network with intertwined driving of encoding and fusion to solve the above problems. Firstly, we introduce transformer (SegFormer) as the backbone of the network to deal with the problem that CNNs are difficult to establish long-range dependence. Secondly, we constructed a unidirectional architecture where encoding and fusion are intertwined and mutually driving, which discards the drawbacks of encoder–decoder architecture to make the network more powerful and concise. Based on the unidirectional architecture, the proposed Local Detail-driven Fusion Module (LDFM) uses the fused features of the previous level to drive the cross-modal fusion at the current level. Meanwhile, the proposed Local Detail-driven Weighting Module (LDWM) uses the fused features to drive the cross-modal weighting. They will drive more effective features to be fed into the next level of the encoding block. Comprehensive experiments have verified the superior performance of our method on the RGB-T saliency detection task.

Deep RGB-D Saliency Detection Without Depth

The existing saliency detection models based on RGB colors only leverage appearance cues to detect salient objects. Depth information also plays a very important role in visual saliency detection and can supply complementary cues for saliency detection. Although many RGB-D saliency models have been proposed, they require to acquire depth data, which is expensive and not easy to get. In this paper, we propose to estimate depth information from monocular RGB images and leverage the intermediate depth features to enhance the saliency detection performance in a deep neural network framework. Specically, we rst use an encoder network to extract common features from each RGB image and then build two decoder networks for depth estimation and saliency detection, respectively. The depth decoder features can be fused with the RGB saliency features to enhance their capability. Furthermore, we also propose a novel dense multiscale fusion model to densely fuse multiscale depth and RGB features based on the dense ASPP model. A new global context branch is also added to boost the multiscale features. Experimental results demonstrate that the added depth cues and the proposed fusion model can both improve the saliency detection performance. Finally, our model not only outperforms state-of-the-art RGB saliency models, but also achieves comparable results compared with state-of-the-art RGB-D saliency models.

Exploring Spatial Correlation for Light Field Saliency Detection: Expansion From a Single View.

Previous 2D saliency detection methods extract salient cues from a single view and directly predict the expected results. Both traditional and deep-learning-based 2D methods do not consider geometric information of 3D scenes. Therefore the relationship between scene understanding and salient objects cannot be effectively established. This limits the performance of 2D saliency detection in challenging scenes. In this paper, we show for the first time that saliency detection problem can be reformulated as two sub-problems: light field synthesis from a single view and light-field-driven saliency detection. This paper first introduces a high-quality light field synthesis network to produce reliable 4D light field information. Then a novel light-field-driven saliency detection network is proposed, in which a Direction-specific Screening Unit (DSU) is tailored to exploit the spatial correlation among multiple viewpoints. The whole pipeline can be trained in an end-to-end fashion. Experimental results demonstrate that the proposed method outperforms the state-of-the-art 2D, 3D and 4D saliency detection methods. Our code is publicly available at https://github.com/OIPLab-DUT/ESCNet.

Underwater Image Enhancement via Minimal Color Loss and Locally Adaptive Contrast Enhancement.

Underwater images typically suffer from color deviations and low visibility due to the wavelength-dependent light absorption and scattering. To deal with these degradation issues, we propose an efficient and robust underwater image enhancement method, called MLLE. Specifically, we first locally adjust the color and details of an input image according to a minimum color loss principle and a maximum attenuation map-guided fusion strategy. Afterward, we employ the integral and squared integral maps to compute the mean and variance of local image blocks, which are used to adaptively adjust the contrast of the input image. Meanwhile, a color balance strategy is introduced to balance the color differences between channel a and channel b in the CIELAB color space. Our enhanced results are characterized by vivid color, improved contrast, and enhanced details. Extensive experiments on three underwater image enhancement datasets demonstrate that our method outperforms the state-of-the-art methods. Our method is also appealing in its fast processing speed within 1s for processing an image of size 1024×1024×3 on a single CPU. Experiments further suggest that our method can effectively improve the performance of underwater image segmentation, keypoint detection, and saliency detection. The project page is available at https://li-chongyi.github.io/proj_MMLE.html.

SOFT: salient object detection based on feature combination using teaching-learning-based optimization

Salient object detection is a challenging task in the computer vision due to complexity of images and insufficiency of a single feature for saliency map generation. The performance of saliency detection depends upon the manner in which these visual features are combined. In this paper, we propose an efficient features combination model in which different features are combined in an appropriate manner. To meet this objective, a metaheuristic optimization algorithm called teacher-learning-based optimization (TLBO) is employed to find an optimal weight vector for feature combination. TLBO is a parameterless, efficient, and robust algorithm as it does not require parameter tuning during its implementation. Furthermore, the optimization criteria of TLBO is modified using a novel fitness function for improved performance. To check the performance of the proposed model, we have conducted extensive experiments over six benchmark datasets and compared the results against seven state-of-the-art methods.

Video salient object detection using dual-stream spatiotemporal attention

Video salient object detection plays an important role in many exciting applications in different areas. However, the existing deep learning-based video salient object detection methods still struggle in scenes of large salient object variabilities and great background scene diversity between and within frames. In this paper, we propose a dual-stream spatiotemporal attention network (DSSANet) for saliency detection in videos. It creatively introduces a multiplex attention mechanism to effectively extract and fuse spatiotemporal features of video salient object over frames in the video, thereby improving saliency detection performance. The DSSANet consists of: (1) A context feature path leverages a novel attention-augmented convolutional LSTM to effectively model the long-range dependency of the great temporal variation in the salient object over frames. (2) A content feature path creatively leverages an attention-based 1D dilated convolution to effectively model the local pixel correlation structure of each pixel in the salient object and the surrounding objects. (3) A refinement fusion module fuses these two features from their paths and further refines the fused feature by an attention-based feature selection. By integrating these three parts, DSSANet accurately detects the salient object from the video. The extensive experiments are performed on four public datasets and demonstrate the effectiveness of DSSANet and the superiority to five state-of-the-art video salient object detection methods.

Infrared salient object detection based on global guided lightweight non-local deep features

In this paper, we studied infrared (IR) maritime salient object detection based on convolutional neural networks (CNNs). There are mainly two contributions. Firstly, we constructed a large extended IR ship image dataset (ExtIRShip) for salient maritime target detection, including 9,123 labelled IR images. Secondly, we proposed a global guided lightweight non-local depth feature (DG-Light-NLDF) model. We introduced Dilated Linear Bottleneck (DLB) to replace the standard convolution and adding a simplified global module to provide the location information of the potential salient object, the proposed method can significantly improve the efficiency of Light-NLDF. Experimental results demonstrate that the proposed DG-Light-NLDF model could detect IR maritime salient objects more accurately with less parameters. In addition, comparison experiments between two datasets validated that the larger dataset is also much more beneficial in improving saliency detection performance.

Salient object detection via cross diffusion-based compactness on multiple graphs

The diffusion-based graph has been widely used in saliency detection. Most of the existing methods treat the image boundary patches as background seeds, which may result in the imprecise saliency map if the salient region touches the image boundaries. In this paper, we propose a salient object detection approach via cross diffusion-based compactness on multiple graphs. Firstly, we extract multi-view image features including low-level image features, mid-level image cues (low-level saliency priors), and background-based saliency map. Then, we compute the respective similarity matrix to construct the corresponding graph, and a cross-diffusion algorithm is presented that diffuses each similarity matrix on other’s graphs rather than at itself graph, helping to the compactness-based saliency maps. Additionally, for well propagating the saliency values, we model a propagation mechanism based on cellular automata, by linearly incorporating low-level image features and mid-level image cues together to generate a superior impact factor matrix. Extensive experiment results demonstrate that the proposed method achieves better saliency detection performance against the unsupervised state-of-the-art methods on three public datasets.

EF-Net: A novel enhancement and fusion network for RGB-D saliency detection

Salient object detection (SOD) has gained tremendous attention in the field of computer vision. Multi-modal SOD based on the complementary information from RGB images and depth maps has shown remarkable success, making RGB-D saliency detection an active research topic. In this paper, we propose a novel multi-modal enhancement and fusion network (EF-Net) for effective RGB-D saliency detection. Specifically, we first utilize a color hint map module with RGB images to predict a hint map, which encodes the coarse information of salient objects. The resulting hint map is then utilized to enhance the depth map with our depth enhancement module, which suppresses the noise and sharpens the object boundary. Finally, we propose an effective layer-wise aggregation module to fuse the features extracted from the enhanced depth maps and RGB images for the accurate detection of salient objects. Our EF-Net utilizes an enhancement-and-fusion framework for saliency detection, which makes full use of the information from RGB images and depth maps. In addition, our depth enhancement module effectively resolves the low-quality issue of depth maps, which boosts the saliency detection performance remarkably. Extensive experiments on five widely-used benchmark datasets demonstrate that our method outperforms 12 state-of-the-art RGB-D saliency detection approaches in terms of five key evaluation metrics.

Salient object detection via reliability‐based depth compactness and depth contrast

It can be intuitively inferred that a high-quality depth map can be used to quickly detect the salient region in stereo vision, implying that depth information plays an essential role in stereoscopic visual attention. However, existing methods generally use the depth map as an auxiliary cue to improve the saliency detection performance. In this study, the authors present an algorithm to directly detect the salient object from a high-quality depth image. The proposed algorithm utilises a depth reliability indicator to assess the confidence of a depth image. Depth compactness, a novel feature that incorporates the depth reliability of the super-pixels, is computed as a primary salient feature. Moreover, in order to enhance another salient feature (i.e. depth contrast), they develop a coarse background filtering method to suppress background interference. Experimental results demonstrate that the proposed method performs favourably against the popular depth-aware saliency detection approaches at a lower computational cost.

Robust Visual Saliency Optimization Based on Bidirectional Markov Chains

Saliency detection aims to automatically highlight the most important area in an image. Traditional saliency detection methods based on absorbing Markov chain only take into account boundary nodes and often lead to incorrect saliency detection when the boundaries have salient objects. In order to address this limitation and enhance saliency detection performance, this paper proposes a novel task-independent saliency detection method based on the bidirectional absorbing Markov chains that jointly exploits not only the boundary information but also the foreground prior and background prior cues. More specifically, the input image is first segmented into number of superpixels, and the four boundary nodes (duplicated as virtual nodes) are selected. Subsequently, the absorption time upon transition node’s random walk to the absorbing state is calculated to obtain the foreground possibility. Simultaneously, foreground prior (as the virtual absorbing nodes) is used to calculate the absorption time and get the background possibility. In addition, the two aforementioned results are fused to form a combined saliency map which is further optimized by using a cost function. Finally, the superpixel-level saliency results are optimized by a regularized random walks ranking model at multi-scale. The comparative experimental results on four benchmark datasets reveal superior performance of our proposed method over state-of-the-art methods reported in the literature. The experiments show that the proposed method is efficient and can be applicable to the bottom-up image saliency detection and other visual processing tasks.

CmSalGAN: RGB-D Salient Object Detection With Cross-View Generative Adversarial Networks

Image salient object detection (SOD) is an active research topic in computer vision and multimedia area. Fusing complementary information of RGB and depth has been demonstrated to be effective for image salient object detection which is known as RGB-D salient object detection problem. The main challenge for RGB-D salient object detection is how to exploit the salient cues of both intra-modality (RGB, depth) and cross-modality simultaneously which is known as cross-modality detection problem. In this paper, we tackle this challenge by designing a novel cross-modality Saliency Generative Adversarial Network ( cm SalGAN). cm SalGAN aims to learn an optimal view-invariant and consistent pixel-level representation for RGB and depth images via a novel adversarial learning framework, which thus incorporates both information of intra-view and correlation information of cross-view images simultaneously for RGB-D saliency detection problem. To further improve the detection results, the attention mechanism and edge detection module are also incorporated into cm SalGAN. The entire cm SalGAN can be trained in an end-to-end manner by using the standard deep neural network framework. Experimental results show that cm SalGAN achieves the new state-of-the-art RGB-D saliency detection performance on several benchmark datasets.

Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features.

This work presents a novel method of exploring human brain-visual representations, with a view towards replicating these processes in machines. The core idea is to learn plausible computational and biological representations by correlating human neural activity and natural images. Thus, we first propose a model, EEG-ChannelNet, to learn a brain manifold for EEG classification. After verifying that visual information can be extracted from EEG data, we introduce a multimodal approach that uses deep image and EEG encoders, trained in a siamese configuration, for learning a joint manifold that maximizes a compatibility measure between visual features and brain representations. We then carry out image classification and saliency detection on the learned manifold. Performance analyses show that our approach satisfactorily decodes visual information from neural signals. This, in turn, can be used to effectively supervise the training of deep learning models, as demonstrated by the high performance of image classification and saliency detection on out-of-training classes. The obtained results show that the learned brain-visual features lead to improved performance and simultaneously bring deep models more in line with cognitive neuroscience work related to visual perception and attention.

Rule-based aggregation driven by similar images for visual saliency detection

The visual saliency detection consists in determining the relevant visual information in a scene to segment it from the background. This paper proposes visual saliency detection in a rule-based approach using the image similarity cue to improve saliency detection performance. Our system induces rules for saliency detection, and for a given input image, determines the subset of rules to be used from a set of candidate rules. The proposed approach consists of two main stages: training and testing. Firstly, during the training stage, our system learns an ensemble of rough-set-based rules by combining knowledge extracted from outputs of four state-of-the-art saliency models. Secondly, our system determines the most suitable subset of induced rules for binary detection of pixels of a salient object in an image. The decision of the best subset of rules is based on the image similarity cue. The binary determination of saliency in the output image, exempts us from performing a post-processing stage as is needed in most saliency approaches. The proposed method is evaluated quantitatively on three challenging databases designed for the saliency detection task. The results obtained from the performed experiments indicate that the proposed method outperforms the state-of-the-art approaches used for comparison.

RGB-T Saliency Detection via Low-Rank Tensor Learning and Unified Collaborative Ranking

Saliency detection is a significant research topic in the field of image processing and computer vision. Currently, most saliency detection methods are applied to RGB images, so that they may encounter adverse scenarios characterized by complex background, inclement weather, and low illumination. Fusing complementary advantages of RGB and thermal infrared (RGB-T) images can effectively boost saliency detection performance. Therefore, we propose a novel RGB-T saliency detection method in this letter. To this end, we first regard superpixels as graph nodes and calculate the affinity matrix for each feature. Then, we propose a low-rank tensor learning model for the graph affinity, which can suppress redundant information and improve the relevance of similar image regions. Finally, a novel ranking algorithm is proposed to jointly obtain the optimal affinity matrix and saliency values under a unified structure. Test results on two RGB-T datasets illustrate the proposed method performs well when against the state-of-the-art algorithms.

Improved Saliency Detection in RGB-D Images Using Two-phase Depth Estimation and Selective Deep Fusion.

To solve the saliency detection problem in RGB-D images, the depth information plays a critical role in distinguishing salient objects or foregrounds from cluttered backgrounds. As the complementary component to color information, the depth quality directly dictates the subsequent saliency detection performance. However, due to artifacts and the limitation of depth acquisition devices, the quality of the obtained depth varies tremendously across different scenarios. Consequently, conventional selective fusion-based RGB-D saliency detection methods may result in a degraded detection performance in cases containing salient objects with low color contrast coupled with a low depth quality. To solve this problem, we make our initial attempt to estimate additional high-quality depth information, which is denoted by Depth+. Serving as a complement to the original depth, Depth+ will be fed into our newly designed selective fusion network to boost the detection performance. To achieve this aim, we first retrieve a small group of images that are similar to the given input, and then the inter-image, nonlocal correspondences are built accordingly. Thus, by using these inter-image correspondences, the overall depth can be coarsely estimated by utilizing our newly designed depth-transferring strategy. Next, we build fine-grained, object-level correspondences coupled with a saliency prior to further improve the depth quality of the previous estimation. Compared to the original depth, our newly estimated Depth+ is potentially more informative for detection improvement. Finally, we feed both the original depth and the newly estimated Depth+ into our selective deep fusion network, whose key novelty is to achieve an optimal complementary balance to make better decisions toward improving saliency boundaries.