Salient Object Detection Approaches Research Articles

ELWNet: An Extremely Lightweight Approach for Real-Time Salient Object Detection

Existing lightweight salient object detection (SOD) methods aim to solve the problem of high computational costs that is prevalent with heavyweight methods. However, compared with heavyweight methods, the detection accuracy of lightweight methods is greatly reduced while real-time performance is not significantly improved. Therefore, we aim to establish a trade off between computational cost and detection performance by improving the network efficiency. We propose a fast and extremely lightweight end-to-end wavelet neural network (ELWNet) for real-time salient object detection. ELWNet can achieve salient object detection and segmentation at approximately 70FPS (GPU), 19FPS (CPU) with 76K parameters and 0.38G FLOPs. We introduce wavelet transform theory into a neural network, proposing a wavelet transform module (WTM), a wavelet transform fusion module (WTFM), a novel feature residual mechanism, and construct an efficient architecture. The wavelet transform theory is integrated into the neural network to realize the interaction between the features in the frequency and the time domain. Meanwhile, ELWNet does not rely on a pre-trained model, which significantly reduces redundant features. We validate the performance of ELWNet using five well-known datasets, and demonstrate state-of-the-art performance compared with 24 other SOD models in terms of being lightweight, detection accuracy and real-time capabilities. Our method maintains high detection performance while reducing the number of model parameters by approximately 99% compared with heavyweight methods.

Depth cue enhancement and guidance network for RGB-D salient object detection

Depth maps have been proven profitable to provide supplements for salient object detection in recent years. However, most RGB-D salient object detection approaches ignore that there are usually low-quality depth maps, which will inevitably result in unsatisfactory results. In this paper, we propose a depth cue enhancement and guidance network (DEGNet) for RGB-D salient object detection by exploring the depth quality enhancement and utilizing the depth cue guidance to generate predictions with highlighted objects and suppressed backgrounds. Specifically, a depth cue enhancement module is designed to generate high-quality depth maps by enhancing the contrast between the foreground and the background. Then considering the different characteristics of unimodal RGB and depth features, we use different feature enhancement strategies to strengthen the representation capability of side-output unimodal features. Moreover, we propose a depth-guided feature fusion module to excavate depth cues provided by the depth stream to guide the fusion of multi-modal features by fully making use of different modal properties, thus generating discriminative cross-modal features. Besides, we aggregate cross-modal features at different levels to obtain the final prediction by adopting a pyramid feature shrinking structure. Experimental results on six benchmark datasets demonstrate that the proposed network DEGNet outperforms 17 state-of-the-art methods.

Selective feature fusion network for salient object detection

AbstractFully convolutional neural networks have achieved great success in salient object detection, in which the effective use of multi‐layer features plays a critical role. Based on this advantage, many saliency detectors have emerged in recent years, and most of them designed a series of network structures to integrate the multi‐level features generated by the backbone network. However, information in different layer play different roles in saliency object detection, how to integrate them effectively is still a great challenge. In this article, a selective feature fusion network which consists of a selective feature fusion module (SFM) and an attention‐guide hierarchical feature emphasis module (AEM) is proposed. Most of the previous works mainly integrate multi‐level feature by addition and concatenation, as a difference, SFM adaptively selects the important information from the input features in the fusion, which effectively avoids introducing too much redundant information. Besides, AEM combines spatial attention and channel attention to enhance features simply and effectively by hierarchical iteration, and further improve the accuracy of salient object detection. Experiments on five datasets show that the proposed selective feature fusion method achieve satisfactory results when comparing to other state‐of‐the‐art salient object detection approaches.

Deep Hypersphere Feature Regularization for Weakly Supervised RGB-D Salient Object Detection.

We propose a weakly supervised approach for salient object detection from multi-modal RGB-D data. Our approach only relies on labels from scribbles, which are much easier to annotate, compared with dense labels used in conventional fully supervised setting. In contrast to existing methods that employ supervision signals on the output space, our design regularizes the intermediate latent space to enhance discrimination between salient and non-salient objects. We further introduce a contour detection branch to implicitly constrain the semantic boundaries and achieve precise edges of detected salient objects. To enhance the long-range dependencies among local features, we introduce a Cross-Padding Attention Block (CPAB). Extensive experiments on seven benchmark datasets demonstrate that our method not only outperforms existing weakly supervised methods, but is also on par with several fully-supervised state-of-the-art models. Code is available at https://github.com/leolyj/DHFR-SOD.

Residual attentive feature learning network for salient object detection

Salient object detection is one of the basic challenging problems in the area of computer vision. Traditional saliency models usually utilize handcrafted features and various prior cues to locate and segment salient objects from complicated surroundings. Recently, with the development of convolutional neural networks (CNNs), most of existing approaches explore the combination of multi-level feature maps for improving detection performance. However, how to learn powerful feature is still a challenge, and the multi-level and multi-scale feature maps are usually aggregated without distinction. In this paper, we propose a novel salient object detection approach named residual attentive feature learning (RAFL) to uniformly highlight salient objects and effectively suppress background noises. In our proposed RAFL, a global perception (GP) module is designed to obtain rich global features of input image. A pyramid feature extraction (PFE) module is utilized to capture the rich context of side-output information. Moreover, an attentive feature selection (AFS) module is applied to refine these side-output features and generate attentive features, so as to find out the important feature channels having high responses to the salient objects. Finally, a residual feature learning (RFL) module is proposed to make the shallow layers generate more accurate saliency maps with the help of the deep layers which contain more global semantic information. Experimental results show that our proposed approach performs favorably against 18 state-of-the-art representative methods on five benchmark datasets.

BiconNet: An edge-preserved connectivity-based approach for salient object detection

Salient object detection (SOD) is viewed as a pixel-wise saliency modeling task by traditional deep learning-based methods. A limitation of current SOD models is insufficient utilization of inter-pixel information, which usually results in imperfect segmentation near edge regions and low spatial coherence. As we demonstrate, using a saliency mask as the only label is suboptimal. To address this limitation, we propose a connectivity-based approach called bilateral connectivity network (BiconNet), which uses connectivity masks together with saliency masks as labels for effective modeling of inter-pixel relationships and object saliency. Moreover, we propose a bilateral voting module to enhance the output connectivity map, and a novel edge feature enhancement method that efficiently utilizes edge-specific features. Through comprehensive experiments on five benchmark datasets, we demonstrate that our proposed method can be plugged into any existing state-of-the-art saliency-based SOD framework to improve its performance with negligible parameter increase.

Spatiotemporal Saliency Representation Learning for Video Action Recognition

Deep convolutional neural networks (CNNs) have achieved great success in human action recognition, however they are still limited in understanding complex and noisy videos owing to the difficulties of exploiting appearance and motion information. Most existing works have been devoted to designing CNN architectures, which overlook the quality of network inputs that is of great importance. This paper provides an alternative solution of action recognition improvement by focusing on the quality of network inputs. A multi-task video salient object detection approach with object-of-interest segmentation scheme, which takes into account both human and action-relevant cues, is proposed to immunize the input video from background clutter. Further, a simple spatiotemporal residual network architecture is presented, which operates on multiple high-quality inputs for long-term action representation learning. Empirical evaluations on various challenging datasets demonstrate that the proposed framework can perform competitively against state-of-the-art. Besides better performance, learning representations of saliency can help prevent the action recognition model from overfitting and speed up the convergence of training.

Salient object detection via cross diffusion-based compactness on multiple graphs

The diffusion-based graph has been widely used in saliency detection. Most of the existing methods treat the image boundary patches as background seeds, which may result in the imprecise saliency map if the salient region touches the image boundaries. In this paper, we propose a salient object detection approach via cross diffusion-based compactness on multiple graphs. Firstly, we extract multi-view image features including low-level image features, mid-level image cues (low-level saliency priors), and background-based saliency map. Then, we compute the respective similarity matrix to construct the corresponding graph, and a cross-diffusion algorithm is presented that diffuses each similarity matrix on other’s graphs rather than at itself graph, helping to the compactness-based saliency maps. Additionally, for well propagating the saliency values, we model a propagation mechanism based on cellular automata, by linearly incorporating low-level image features and mid-level image cues together to generate a superior impact factor matrix. Extensive experiment results demonstrate that the proposed method achieves better saliency detection performance against the unsupervised state-of-the-art methods on three public datasets.

Geometry Auxiliary Salient Object Detection for Light Fields via Graph Neural Networks.

Light field imaging, originated from the availability of light field capture technology, offers a wide range of applications in the field of computational vision. The capability of predicting salient objects of light fields remains technologically challenging due to its complicated geometry structure. In this paper, we propose a light field salient object detection approach that formulates the geometric coherence among multiple views of light fields as graphs, where the angular/central views represent the nodes and their relations compose the edges. The spatial and disparity correlations between multiple views are effectively explored through multi-scale graph neural networks, enabling the more comprehensive understanding of light field content and more representative and discriminative saliency features generation. Moreover, a multi-scale saliency feature consistency learning module is embedded to enhance the saliency features. Finally, an accurate salient object map is produced for the light field based upon the extracted features. In addition, we establish a new light field salient object detection dataset (CITYU-Lytro) that contains 817 light fields with diverse contents and their corresponding annotations, aiming to further promote the research on light field salient object detection. Quantitative and qualitative experiments demonstrate that the proposed method performs favorably compared with the state-of-the-art methods on the benchmark datasets.

Pixel-wise ordinal classification for salient object grading

Driven by business intelligence applications for rating attraction of products in shops, a new problem — salient object grading is studied in this paper. In computer vision, plenty of salient object detection approaches have been proposed, while most existing studies detect objects in a binary manner: salient or not. This paper focuses on a new problem setting that requires detecting all salient objects and categorizing them into different salient levels. Based on that, a pixel-wise ordinal classification method is proposed. It consists of a multi-resolution saliency detector which detects and segments objects, an ordinal classifier which grades pixels into different salient levels, and a binary saliency enhancer which sharpens the difference between non-saliency and all other salient levels. Two new image datasets with salient level labels are constructed. Experimental results demonstrate that, on the one hand, the proposed method provides effective salient level predictions and on the other hand, offers very comparable performance with state-of-the-art salient object detection methods in the traditional problem setting.

A deep learning approach to building an intelligent video surveillance system

Recent advances in the field of object detection and face recognition have made it possible to develop practical video surveillance systems with embedded object detection and face recognition functionalities that are accurate and fast enough for commercial uses. In this paper, we compare some of the latest approaches to object detection and face recognition and provide reasons why they may or may not be amongst the best to be used in video surveillance applications in terms of both accuracy and speed. It is discovered that Faster R-CNN with Inception ResNet V2 is able to achieve some of the best accuracies while maintaining real-time rates. Single Shot Detector (SSD) with MobileNet, on the other hand, is incredibly fast and still accurate enough for most applications. As for face recognition, FaceNet with Multi-task Cascaded Convolutional Networks (MTCNN) achieves higher accuracy than advances such as DeepFace and DeepID2+ while being faster. An end-to-end video surveillance system is also proposed which could be used as a starting point for more complex systems. Various experiments have also been attempted on trained models with observations explained in detail. We finish by discussing video object detection and video salient object detection approaches which could potentially be used as future improvements to the proposed system.

A parallel down-up fusion network for salient object detection in optical remote sensing images

The diverse spatial resolutions, various object types, scales and orientations, and cluttered backgrounds in optical remote sensing images (RSIs) challenge the current salient object detection (SOD) approaches. It is commonly unsatisfactory to directly employ the SOD approaches designed for nature scene images (NSIs) to RSIs. In this paper, we propose a novel Parallel Down-up Fusion network (PDF-Net) for SOD in optical RSIs, which takes full advantage of the in-path low- and high-level features and cross-path multi-resolution features to distinguish diversely scaled salient objects and suppress the cluttered backgrounds. To be specific, keeping a key observation that the salient objects still are salient no matter the resolutions of images are in mind, the PDF-Net takes successive down-sampling to form five parallel paths and perceive scaled salient objects that are commonly existed in optical RSIs. Meanwhile, we adopt the dense connections to take advantage of both low- and high-level information in the same path and build up the relations of cross paths, which explicitly yield strong feature representations. At last, we fuse the multiple-resolution features in parallel paths to combine the benefits of the features with different resolutions, i.e., the high-resolution feature consisting of complete structure and clear details while the low-resolution features highlighting the scaled salient objects. Extensive experiments on the ORSSD dataset demonstrate that the proposed network is superior to the state-of-the-art approaches both qualitatively and quantitatively.

Salient Object Detection by Spatiotemporal and Semantic Features in Real-Time Video Processing Systems

Object detection is significant for event analysis in various intelligent multimedia processing systems. Although there have been many studies conducting research in this area, effective and efficient object detection methods for video sequences are still much desired. In this article, we investigate salient object detection in real-time multimedia processing systems. Considering the intrinsic relationship between top-down and bottom-up saliency features, we present a new effective method for video salient object detection based on deep semantic and spatiotemporal cues. After extracting top-down semantic features for object perception by a 2-D convolutional network, we concatenate them with bottom-up spatiotemporal cues for motion perception extracted by a 3-D convolutional network. In order to combine these features effectively, we feed them into a 3-D deconvolutional network for feature-sharing learning between semantic features and spatiotemporal cues for the final saliency prediction. Additionally, we propose a novel Gaussian-like loss function with an L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> -norm regularization term for parameter learning. Experimental results show that the proposed salient object detection approach performs better in terms of both effectiveness and efficiency for video sequences compared with the state-of-the-art models.

Finding autofocus region in low contrast surveillance images using CNN-based saliency algorithm

How to automatically locate the focus region in low contrast image is a key issue for camera-equipped surveillance devices. Due to low signal to noise ratio, the performance of autofocus will seriously decline in low contrast image, making it quite difficult to recognize the focus region. To tackle this problem, we perform autofocus by conducting a salient object detection approach. A covariance based deep learning framework is proposed to evaluate the saliency of low contrast surveillance image. Based on the mechanism of human visual system, the autofocus region can be identified by the visual salient object. In this paper, low-level features of the low contrast images are first studied and extracted. Then the mutual covariances of the segmented blocks are trained via a 7-layers convolutional neural network (CNN). Next, the initial saliency map of the testing image can be obtained by estimating the saliency score of each block via the pre-trained CNN model. Finally, the resulting saliency map is refined by introducing the local-global difference and internal similarity approaches. Experimental results demonstrate that the proposed method outperforms existing ten state-of-the-art saliency models on three public datasets and a nighttime image dataset.

Salient Object Detection via Two-Stage Graphs

Despite recent advances made in salient object detection using graph theory, the approach still suffers from accuracy problems when the image is characterized by a complex structure, either in the foreground or background, causing erroneous saliency segmentation. This fundamental challenge is mainly attributed to the fact that most existing graph-based methods take only the adjacently spatial consistency among graph nodes into consideration. In this paper, we tackle this issue from a coarse-to-fine perspective and propose a two-stage-graphs approach for salient object detection, in which two graphs having the same nodes but different edges are employed. Specifically, a weighted joint robust sparse representation model, rather than the commonly used manifold ranking model, helps to compute the saliency value of each node in the first-stage graph, thereby providing a saliency map at the coarse level. In the second-stage graph, along with the adjacently spatial consistency , a new regionally spatial consistency among graph nodes is considered in order to refine the coarse saliency map, assuring uniform saliency assignment even in complex scenes. Particularly, the second stage is generic enough to be integrated in existing salient object detectors, enabling improved performance. Experimental results on benchmark data sets validate the effectiveness and superiority of the proposed scheme over related state-of-the-art methods.

Salient object detection of dairy goats in farm image based on background and foreground priors

In order to achieve object detection of dairy goats in farm image efficiently and accurately, this paper proposes a novel salient object detection approach based on background and foreground priors. First, we improve the FastMBD algorithm to generate background-prior-based saliency map, which eliminate background interference initially. Second, the approach of seed selection is optimized to determine the foreground seeds more accurately, and then saliency map is generated based on manifold ranking to further eliminate the background interference. Finally, two saliency maps are fused, and then combined with post-processing to achieve saliency object detection of farm image. Subsequently, we also use the threshold segmentation based on K-Means algorithm to separate the objects and background. Moreover, the horizontal scanning method and contour extraction are employed to realize the extraction and counting of dairy goats. Experimental results show that the proposed approach based on background and foreground priors has promising performance, and the accuracy of the proposed segmentation method was up to 89.503%.

Saliency detection via background and foreground null space learning

In this paper, we present a novel bottom-up salient object detection approach by exploiting the relationship between the saliency detection and the null space learning. A key observation is that saliency of an image segment can be estimated by measuring the distance to the single point, which represents the background or foreground salient samples in the null spaces. We apply the null Foley–Sammon transformation to model the null spaces of the background samples or foreground salient samples, where the potentially large and complex intra-class variations of the samples are totally removed and the specific features of the respective classes are represented by a single point. Afterward, we formulate the separation of the saliency regions from the background as a distance measurement to this single point in the null space. An optimization algorithm is devised to fuse the background samples based saliency map and foreground samples based saliency map. Results on five benchmark datasets show that the proposed method achieves superior performance compared with the newest state-of-the-art methods in terms of different evaluation metrics, especially for complex natural images.

A novel approach for salient object detection using double-density dual-tree complex wavelet transform in conjunction with superpixel segmentation

Salient object detection in wavelet domain has recently begun to attract researchers’ effort due to its desired ability to provide multi-scale analysis of an image simultaneously in both frequency and spatial domains. The proposed algorithm exploits the inherent multi-scale structure of the double-density dual-tree complex-oriented wavelet transform (DDDTCWT) to decompose each input image into four approximate sub-band images and 32 high-pass detailed sub-band images at each scale. These 32 detailed high-pass sub-bands at each scale are adequate to represent singularities of any geometric object with high precision and to mimic zooming-in and zooming-out process of human vision system. In the proposed model, we first compute a rough segmented saliency map (RSSM) by fusing multi-scale edge-to-texture features generated from DDDTCWT with segmentation results obtained from bipartite graph partitioning-based segmentation approach. Then, each pixel in RSSM is categorized into either background region or salient region based on a threshold. Finally, the pixels of the two regions are considered as samples to be drawn from a multivariate kernel function whose parameters are estimated using expectation maximization algorithm, to generate a saliency map. The performance of the proposed model is evaluated in terms of precision, recall, F-measure, area under the ROC curve and computation time using six publicly available image datasets. Extensive experimental results on six benchmark datasets demonstrate that the proposed model outperformed the existing 29 state-of-the-art methods in terms of F-measure on all five datasets, recall on four datasets and area under ROC curve on two datasets. In terms of mean recall value, mean F-measure value and mean AUC value on all six datasets, the proposed method outperforms all state-of-the-art methods. The proposed method also takes comparatively less computation time in comparison with many existing spatial domain methods.

Unsupervised Salient Object Detection via Inferring From Imperfect Saliency Models

Visual saliency detection has become an active research direction in recent years. A large number of saliency models, which can automatically locate objects of interest in images, have been developed. As these models take advantage of different kinds of prior assumptions, image features, and computational methodologies, they have their own strengths and weaknesses and may cope with only one or a few types of images well. Inspired by these facts, this paper proposes a novel salient object detection approach with the idea of inferring a superior model from a variety of previous imperfect saliency models via optimally leveraging the complementary information among them. The proposed approach mainly consists of three steps. First, a number of existing unsupervised saliency models are adopted to provide weak/imperfect saliency predictions for each region in the image. Then, a fusion strategy is used to fuse each image region's weak saliency predictions into a strong one by simultaneously considering the performance differences among various weak predictions and various characteristics of different image regions. Finally, a local spatial consistency constraint that ensures high similarity of the saliency labels for neighboring image regions with similar features is proposed to refine the results. Comprehensive experiments on five public benchmark datasets and comparisons with a number of state-of-the-art approaches can demonstrate the effectiveness of the proposed work.

Saliency refinement: Towards a uniformly highlighted salient object

Humans have a natural tendency to view a visually attractive (i.e., salient) object in its entirety. However, previous methods for salient object detection only highlight some parts of the salient object. This problem severely limits the adoption of such technologies to various computer vision and pattern recognition applications. To address the problem, in this paper, we present a novel framework to improve a saliency map obtained from recent state-of-the-art salient object detection approaches. Based on the fact that the L0 optimization can efficiently minimize variation between values, we integrate a background saliency and an initial saliency based on the nonlocal L0 optimization. In our work, we first extract background samples to estimate the background saliency building upon the initial saliency and color information. We then integrate the background saliency into the initial saliency by solving an optimization problem. We formulate the optimization problem based on the nonlocal L0 gradient to efficiently minimize the saliency variation in the salient object. To confirm the effectiveness of our proposed method, we apply the proposed framework to the saliency maps generated from state-of-the-art methods. Experimental results on benchmark datasets demonstrate that the proposed framework significantly improves the saliency maps. Furthermore, we compare the performance of two refinement frameworks and ours to prove the superiority of our work.