Saliency Detection Model Research Articles

A novel deep network and aggregation model for saliency detection

Recent deep learning-based methods for saliency detection have proved the effectiveness of integrating features with different scales. They usually design various complex architectures of network, e.g., multiple networks, to explore the multi-scale information of images, which is expensive in computation and memory. Feature maps produced with different subsampling convolutional layers have different spatial resolutions; therefore, they can be used as the multi-scale features to reduce the costs. In this paper, by exploiting the in-network feature hierarchy of convolutional networks, we propose a novel multi-scale network for saliency detection (MSNSD) consisting of three modules, i.e., bottom-up feature extraction, top-down feature connection and multi-scale saliency prediction. Moreover, to further boost the performance of MSNSD, an input image-aware saliency aggregation method is proposed based on the ridge regression, which combines MSNSD with some well-performed handcrafted shallow models. Extensive experiments on several benchmarks show that the proposed MSNSD outperforms the state-of-the-art saliency methods with less computational and memory complexity. Meanwhile, our aggregation method for saliency detection is effective and efficient to combine deep and shallow models and make them complementary to each other.

Coarse-to-fine salient object detection with low-rank matrix recovery

Low-rank matrix recovery (LRMR) has recently been applied to saliency detection by decomposing image features into a low-rank component associated with background and a sparse component associated with visual salient regions. Despite its great potential, existing LRMR-based saliency detection methods seldom consider the inter-relationship among elements within these two components, thus are prone to generating scattered or incomplete saliency maps. In this paper, we introduce a novel and efficient LRMR-based saliency detection model under a coarse-to-fine framework to circumvent this limitation. First, we roughly measure the saliency of image regions with a baseline LRMR model that integrates a ℓ1-norm sparsity constraint and a Laplacian regularization smooth term. Given samples from the coarse saliency map, we then learn a projection that maps image features to refined saliency values, to significantly sharpen the object boundaries and to preserve the object entirety. We evaluate our framework against existing LRMR-based methods on three benchmark datasets. Experimental results validate the superiority of our method as well as the effectiveness of our suggested coarse-to-fine framework, especially for images containing multiple objects. MATLAB code for implementing our method is available at https://github.com/qizhust/HLRSaliency.

Saliency bagging: a novel framework for robust salient object detection

Salient object detection is a challenging task, and several methods have been proposed for the same in the literature. The problem lies in that most of the methods perform good on a particular set of images but fail when exposed to a variety of different set of images. Here, we address this problem by proposing a novel framework called saliency bagging for detecting salient object(s) in digital images across a variety of images in a robust manner. The proposed framework generates the saliency map of an image in three phases: (i) Selection of existing saliency detection models and generation of initial saliency maps (ii) Generation of integrated binary map from the initial saliency maps by applying adaptive thresholding and majority voting (iii) Computation of final saliency map using integrated binary map and initial saliency maps by applying proposed integration logic. Extensive experiments on six publicly available datasets viz. MSRA10K, DUT-OMRON, ECSSD, PASCAL-S, SED2, and THUR15K have been performed to determine the effectiveness of the proposed method. The performance of the proposed method is measured in terms of Precision, Recall, F-Measure, Mean Absolute Error (MAE) and Receiver Operating Characteristic (ROC) curve and compared with 25 state-of-the-art methods including 17 classic best-performing methods of the last decade, five existing selected, and three aggregation saliency methods. The proposed method outperforms all the compared classic and existing selected methods in terms of Precision, F-Measure, and MAE, while it is comparable to the best-performing methods in terms of Recall and ROC curve across all the six datasets. The proposed framework is computationally very fast than all compared aggregation methods, while performance is almost same on all datasets that support its superiority.

Salient object detection method for breast tumor in ultrasound images based on absorbing Markov chain.

Automatic detection of tumor in breast ultrasound (BUS) images is important for the subsequent image processing and has been researched for decades. However, there still lacks a robust method due to poor quality of BUS images. To propose and test a salient object detection method for BUS images. BUS image is preprocessed by an adaptively selective replacement and speckle reducing anisotropic diffusion (SRAD) algorithm. Then, the preprocessed image is segmented into super pixels by a simple linear iterative clustering (SLIC) algorithm to form a graph model, and the saliency of the nodes in the graph is calculated by using the absorbed time of absorbing Markov chain (AMC). Finally, the initial saliency map is optimized by the recurrent time of ergodic Markov chain (EMC) and a distance weighting formula. Results of the proposed method were compared both qualitatively and quantitatively with two saliency detection models. It was observed that the proposed method outperformed the comparison models and yielded the highest Accuracy value (97.49% vs. 86.63% and 90.33%) using a dataset of 1000 BUS images. After the adaptively selective replacement, AMC can effectively distinguish tumors from background by random walks.

Grey Wolf Optimization on Modeling Saliency Detection in Stereoscopic 3D Images

Till now, several saliency detection models have been introduced for numerous applications in multimedia developments. Anyhow, specific appliances of stereoscopic imaging require improvements in saliency detection schemes for extracting the salient regions in a precise manner. The saliency detection (SD) model faces numerous shortcomings like intricacy in natural images and minor-scale patterns on salient objects. Hence, this paper endeavors to attain the SD model in two levels; Feature extraction (FE), for which Gaussian kernel model is utilized to extort the features and depth SD, for which Gabor Filter (GF) is exploited to attain the depth of saliency map. Accordingly, the adopted scheme optimizes ‘2’ coefficients such as, feature difference betwixt image patches H in feature evaluation and also fine scale c from which the accurate detection is attained. For optimization purpose, a well-known optimization termed Grey Wolf Optimization (GWO) is exploited and evaluated for varying values of a , and the results are attained with respect to ROC (i.e. Receiver Operating Characteristic), and statistical analysis.

Going From RGB to RGBD Saliency: A Depth-Guided Transformation Model.

Depth information has been demonstrated to be useful for saliency detection. However, the existing methods for RGBD saliency detection mainly focus on designing straightforward and comprehensive models, while ignoring the transferable ability of the existing RGB saliency detection models. In this article, we propose a novel depth-guided transformation model (DTM) going from RGB saliency to RGBD saliency. The proposed model includes three components, that is: 1) multilevel RGBD saliency initialization; 2) depth-guided saliency refinement; and 3) saliency optimization with depth constraints. The explicit depth feature is first utilized in the multilevel RGBD saliency model to initialize the RGBD saliency by combining the global compactness saliency cue and local geodesic saliency cue. The depth-guided saliency refinement is used to further highlight the salient objects and suppress the background regions by introducing the prior depth domain knowledge and prior refined depth shape. Benefiting from the consistency of the entire object in the depth map, we formulate an optimization model to attain more consistent and accurate saliency results via an energy function, which integrates the unary data term, color smooth term, and depth consistency term. Experiments on three public RGBD saliency detection benchmarks demonstrate the effectiveness and performance improvement of the proposed DTM from RGB to RGBD saliency.

Improving unsupervised saliency detection by migrating from RGB to multispectral images

AbstractSaliency detection has been an important topic during the last decade. The main goal of saliency detection models is to detect the most relevant objects in a given scene. Most of these models use RGB (Red, Green, Blue) images as an input because they mainly focus on applications where features (eg, faces, textures, colors, or human silhouettes) are extracted from color images, and there are many labeled databases available for RGB‐based saliency data. Nevertheless, the use of RGB inputs clearly limits the amount of information from where to extract the salient regions as spectral information is lost during the color image recording. On the contrary, multispectral systems are able to capture more than three bands in a single capture and can retrieve information from the full spectrum at a pixel. The main aim of this study is to investigate the advantages of using multispectral images instead of RGB images for saliency detection within the framework of unsupervised models. We compare the performance of several unsupervised saliency models with both RGB and multispectral images using a specific dataset of multispectral images with ground‐truth data extracted from observers' fixation patterns. Our results show a general improvement when multispectral information is taken into account. The saliency maps estimated by using the multispectral features are closer to the ground‐truth data, with the simplest Graph‐based visual saliency and Boolean Map‐based models showing good relative gain compared with other approaches.

Video saliency detection by gestalt theory

Image saliency detection has been widely explored in recent decades, but computational modeling of visual attention for video sequences is limited due to complicated temporal saliency extraction and fusion of spatial and temporal saliency. Inspired by Gestalt theory, we introduce a novel spatiotemporal saliency detection model in this study. First, we compute spatial and temporal saliency maps by low-level visual features. And then we merge these two saliency maps for spatiotemporal saliency prediction of video sequences. The spatial saliency map is calculated by extracting three kinds of features including color, luminance, and texture, while the temporal saliency map is computed by extracting motion features estimated from video sequences. A novel adaptive entropy-based uncertainty weighting method is designed to fuse spatial and temporal saliency maps to predict the final spatiotemporal saliency map by Gestalt theory. The Gestalt principle of similarity is used to estimate spatial uncertainty from spatial saliency, while temporal uncertainty is computed from temporal saliency by the Gestalt principle of common fate. Experimental results on three large-scale databases show that our method can predict visual saliency more accurately than the state-of-art spatiotemporal saliency detection algorithms.

Salient object detection for RGB-D image by single stream recurrent convolution neural network

Salient object detection for RGB-D images aims to utilize color and depth information to automatically localize objects of human interest in the scene and reduce the complexity of visual analysis. Different from existing saliency detection model with double-stream network, salient object detection by Single Stream Recurrent Convolution Neural Network(SSRCNN) is proposed. First RGBD four-channels input is fed into VGG-16 net to generate multiple level features which express the most original feature for RGB-D image. The coarse saliency map from the deepest features can detect and localize salient objects, but loss the boundaries and subtle structures. So Depth Recurrent Convolution Neural Network (DRCNN) is then applied to each level feature for rendering salient object outline from deep to shallow hierarchically and progressively. With the help of deeper level feature, original depth cue and coarse saliency map, each level feature can accurately predict the salient objects in different scales. At last all the saliency maps from each level are fused together to generate final results. Extensive quantitative and qualitative experimental evaluations on four dataset demonstrate that the proposed method outperforms most state-of-the-art methods.

Infrared and visible image fusion algorithm based on saliency detection and adaptive double-channel spiking cortical model

Based on the saliency detection which effectively extracts the saliency information of the image and the adaptive double-channel spiking cortical model (ADSCM) which has the advantages of global coupling, pulse synchronization, less parameters and high computational efficiency, which can well process information in dark areas, we proposed a new infrared and visible image fusion algorithm. Firstly, the infrared and visible images are decomposed into the base layer and the detail layers through the rolling guidance filter (RGF). Secondly, the base layer is fused with an improved saliency detection method, and the detail layers are fused by a directional gradient sum and an improved ADSCM. The experimental results show that the proposed method can also fully extract the target information of the infrared image while preserving the visible background information, in which the fusion effect is better than the traditional fusion algorithms.

The visual association test – Extended: detecting non-credible memory performance

Saliency detection is the task of locating informative regions in an image, which is a challenging task in computer vision. In contrast to the existing saliency detection models that focus on either local or global image property, an effective salient object detection method is introduced based on joint modeling global shape and local consistency. To this end, Restricted Boltzmann Machine (RBM) is utilized to model salient object shape as global image property and Conditional Random Field (CRF), on the other hand, is adopted to achieve its local consistency. In order to obtain the final saliency map, a universal framework is introduced to combine the results of RBM and CRF. Experimental results on five benchmark datasets demonstrate that the proposed saliency detection method performs favorably against the existing state-of-the-art algorithms.

Saliency detection based on self-adaptive multiple feature fusion for remote sensing images

ABSTRACTSaliency detection has been revealed an effective and reliable approach to extract the region of interest (ROI) in remote sensing images. However, most existing saliency detection methods employing multiple saliency cues ignore the intrinsic relationship between different cues and do not distinguish the diverse contributions of different cues to the final saliency map. In this paper, we propose a novel self-adaptively multiple feature fusion model for saliency detection in remote sensing images to take advantage of this relationship to improve the accuracy of ROI extraction. First, we take multiple feature channels, namely colour, intensity, texture and global contrast into consideration to produce primary feature maps. Particularly, we design a novel method based on dual-tree complex wavelet transform for remote sensing images to generate texture feature pyramids. Then, we introduce a novel self-adaptive multiple feature fusion method based on low-rank matrix recovery, in which the significances of feature maps are ranked by the low rank constraint recovery, and subsequently multiple features’ contributions are allocated adaptively to produce the final saliency map. Experimental results demonstrate that our proposal outperforms the state-of-the-art methods.

Saliency detection via integrating deep learning architecture and low-level features

Deep learning methods, with their good performance in semantic representation of different images, have been widely used for saliency detection. Recent saliency detection methods have applied deep learning to obtain high-level features and combined them with hand-crafted low-level features to estimate saliency in the images. However, it is difficult to find the relationship between high-level and low-level features, resulting in incomplete integration framework for saliency detection. In this paper, we novely propose a saliency detection model by integrating high-level and low-level features with joint probability estimation. Firstly, the high-level features from FCN-8S network are used to estimate the probability of each superpixel as foreground or background region. Secondly, low-level features are extracted from each superpixels and clustered via affinity propagation (AP) clustering. The distributions of vectors from different clusters are consequently utilized to calculate the conditional probability of each superpixel as salient object under different assumptions. Thirdly, the joint probability of each superpixel as salient object in foreground or background is computed to compose the saliency map of the whole image. To further improve the uniformity of saliency in the same object region, the structured random forest (SRF) method is used to detect the contour of the image and the saliency of superpixels in homogeneous regions are uniformly merged. The advantage of high-level features in representing semantic regions and that of low-level features in differentiating local details in the image are unified and restrained by the joint probability estimation in the proposed model. Experimental results demonstrate that the proposed method provide better saliency detection performance than the state-of-the-art methods on 5 public databases.

A novel edge-oriented framework for saliency detection enhancement

Mixed visual scenes and cluttered background commonly exist in natural images, which forms a challenge for saliency detection. In dealing with complex images, there are two kinds of deficiencies in the existing saliency detection methods: ambiguous object boundaries and fragmented salient regions. To address these two limitations, we propose a novel edge-oriented framework to improve the performance of existing salient detection methods. Our framework is based on two interesting insights: 1) human eyes are sensitive to the edges between foreground and background even there is hardly any difference in terms of saliency, 2) Guided by semantic integrity, human eyes tend to view a visual scene as several objects, rather than pixels or superpixels. The proposed framework consists of the following three parts. First, an edge probability map is extracted from an input image. Second, the edge-based over-segmentation is obtained by sharpening the edge probability map, which is ultilized to generate edge-regions using an edge-strength based hierarchical merge model. Finally, based on the prior saliency map generated by existing methods, the framework assigns each edge-region with a saliency value. Based on four publically available datasets, the experiments demonstrate that the proposed framework can significantly improve the detection results of existing saliency detection models, which is also superior to other state-of-the-art methods.

A Biologically Inspired Saliency Priority Extraction Using Bayesian Framework

In this article, the authors used saliency detection for video streaming problem to be able to transmit regions of video frames in a ranked manner based on their importance. The authors designed an empirically-based study to investigate bottom-up features to achieve a ranking system stating the saliency priority. We introduced a gradual saliency detection model using a Bayesian framework for static scenes under conditions that we had no cognitive bias. To extract color saliency, we used a new feature contrast in Lab color space as well as a k-nearest neighbor search based on k-d tree search technique to assign a ranking system into different colors according to our empirical study. To find the salient textured regions we employed contrast-based Gabor energy features and then we added a new feature as intensity variance map. We merged different feature maps and classified saliency maps using a Naive Bayesian Network to prioritize the saliency across a frame. The main goal of this work is to create the ability to assign a saliency priority for the entirety of a video frame rather than simply extracting a salient area which is widely performed.

Hierarchical visual attention model for saliency detection inspired by avian visual pathways

Visual attention is a mechanism that enables the visual system to detect potentially important objects in complex environment. Most computational visual attention models are designed with inspirations from mammalian visual systems. However, electrophysiological and behavioral evidences indicate that avian species are animals with high visual capability that can process complex information accurately in real time. Therefore, the visual system of the avian species, especially the nuclei related to the visual attention mechanism, are investigated in this paper. Afterwards, a hierarchical visual attention model is proposed for saliency detection. The optic tectum neuron responses are computed and the self-information is used to compute primary saliency maps in the first hierarchy. The &#x201C winner-take-all &#x201D network in the tecto-isthmal projection is simulated and final saliency maps are estimated with the regularized random walks ranking in the second hierarchy. Comparison results verify that the proposed model, which can define the focus of attention accurately, outperforms several state-of-the-art models. This study provides insights into the relationship between the visual attention mechanism and the avian visual pathways. The computational visual attention model may reveal the underlying neural mechanism of the nuclei for biological visual attention.

Region-of-Interest Detection Based on Statistical Distinctiveness for Panchromatic Remote Sensing Images

Region-of-interest (ROI) detection plays a significant role in the analysis and interpretation of remote sensing images (RSI), due to the huge size of satellite images and their explosive growth in quantity. However, when applied to panchromatic RSI directly, traditional saliency models cannot achieve satisfying performance for two reasons: one is the computational efficiency decrease caused by the huge image size; the other is the absence of color information for panchromatic RSI. Thus, in this letter, an ROI detection model based on statistical distinctiveness (SD) is proposed for saliency analysis and ROIs detection in panchromatic RSI. The proposed SD model incorporates both the lower order SD (LSD) and the higher order SD (HSD), in order to identify regions of interest that are highly distinctive from the rest of the scene. Finally, the saliency map is determined by fusing cue maps obtained by calculating LSD locally and HSD globally. Experimental results show that our approach achieves promising results when compared with existing state-of-the-art saliency detection models.

A novel approach to reconstruction based saliency detection via convolutional neural network stacked with auto-encoder

Visual saliency detection, aimed at the simulation of human visual system (HVS), has drawn wide attention in recent decades. Reconstruction based saliency detection models are established methods for saliency detection, which predict unexpected regions via linear combination or auto-encoder network. However, these models are ineffective in dealing with images due to the loss of spatial information caused by the conversion from images to vectors. In this paper, a novel approach is proposed to solve this problem. The core is a deep reconstruction model, i.e., convolutional neural network for reconstruction stacked with auto-encoder (CNNR for short). On the one hand, the use of CNN is able to directly take two-dimensional data as input rather than having to convert the matrix to a series of vectors as in conventional reconstruction based saliency detection methods. On the other hand, the training process of CNN is augmented with the initialization obtained by an unsupervised learning process of convolutional auto-encoder (CAE). By this way, our CNNR model can be trained on limited labeled data, with the weights of the CNN being meaningfully initialized by CAE instead of random initialization. Performance evaluations are conducted through comprehensive experiments on four benchmark datasets and the comparisons with eight state-of-the-art saliency detection models show that our proposed deep reconstruction model outperforms most of the eight state-of-the-art saliency detection models.

Prediction of mobile image saliency and quality under cloud computing environment

Recent years have witnessed the explosive growth of multimedia applications over networks and increasingly high requirements of consumers for multimedia signals in terms of quality of experience (QoE). Effective and efficient yet energy-saving saliency detection model and quality prediction method are eagerly desired, since they play critical roles in raising users' QoE and promoting the progress of green multimedia communication. Current studies of saliency detection and quality evaluation are far from ideal yet. In this paper we investigate the influence of complexity on visual saliency and quality. Complexity is an essential concept in human perception to visual stimulus, but it is substantially abstract and hard to be endowed with a clear definition. We suppose that brain systematically combines global and local features during the whole process of human perception. Global features lead a dominant position in seeking salient areas under the condition that image complexity is high, namely without obviously isolated foreground objects, whereas local features play a key role in an opposite situation. With this consideration, this paper establishes a novel framework for detecting visual saliency based on image complexity estimation before complexity-adaptive merging of global and local features. Furthermore, the concept of complexity is deployed for blind photographic image quality assessment (IQA) by means of saliency-based weighting. Features which refer to contrast, artifacts, brightness and natural scene statistics (NSS) are modified and integrated to derive a blind IQA model and predict the quality of photos. Based on the above two technologies, this paper introduces smart phones as mobile terminals, cloud platforms for speed-up and energy-saving, and wireless networks for transmission, and provides a practical mobile multimedia application. Comparative experiments validate that, within this application system, our proposed saliency detection model and blind photographic IQA method implement better than existing relevant competitors in terms of effectiveness and efficiency comparison.

RETRACTED ARTICLE: Saliency detection in stereoscopic images using adaptive Gaussian Kernel and Gabor filter

So far a number of saliency detection designs have been developed for various applications in multimedia scenario. However, certain applications of stereoscopic imaging necessitate enhancing saliency detection designs to extract the salient areas accurately. Saliency detection strategy still faces many challenges like complexity in natural images as well as lesser-scale patterns in salient objects. This paper is set to obtain the saliency detection technique in two stages: feature extraction and depth saliency detection. Gaussian Kernel model is used to extract the features and Gabor filter to get the depth saliency map. This paper optimizes two coefficients namely feature difference among image patches $$ U $$ , from feature evaluation and fine scale $$ c $$ , from which the accurate detection can be achieved. For this, a renowned optimization algorithm named grey wolf optimization (GWO) is used, which is a recently proposed intelligent optimization method inspired by hunting behavior of grey wolves. The saliency detection model detection model is implemented in MATLAB 2015a. To the next of implementation, the performance of the proposed model, GWO-3-dimensional saliency mapping is compared over other the conventional algorithms in terms of receiver operator curve, Pearson correlation coefficient, Kullback–Leibler divergence and area under the curve. The overall performance analysis in terms of above measures and a valuable statistical analysis are carried out for validating the effectiveness of proposed model. The entire analysis proves that the proposed saliency detection performs better than the conventional models.