Existing Saliency Models Research Articles

TranSalNet+: Distortion-aware saliency prediction

Predicting the saliency of images affected by distortion is a challenging but emerging research problem. Given a distorted image, we wish to accurately predict saliency as perceived by humans. A recent distortion-aware saliency benchmark – the CUDAS database – reveals the inadequacy of existing saliency models in handling distorted images. In this paper, we devise a deep learning Distortion-Aware Saliency Module (DASM) that enables capturing saliency features related to image distortions, and integrates this module into a saliency prediction architecture. To achieve the high expressive capability of DASM using supervised learning, we create a dedicated dataset that draws upon a large-scale saliency dataset and machine-generated image quality assessments. Experimental results demonstrate the superior performance of the proposed model in predicting the saliency of distorted images.

How is Visual Attention Influenced by Text Guidance? Database and Model.

The analysis and prediction of visual attention have long been crucial tasks in the fields of computer vision and image processing. In practical applications, images are generally accompanied by various text descriptions, however, few studies have explored the influence of text descriptions on visual attention, let alone developed visual saliency prediction models considering text guidance. In this paper, we conduct a comprehensive study on text-guided image saliency (TIS) from both subjective and objective perspectives. Specifically, we construct a TIS database named SJTU-TIS, which includes 1200 text-image pairs and the corresponding collected eye-tracking data. Based on the established SJTU-TIS database, we analyze the influence of various text descriptions on visual attention. Then, to facilitate the development of saliency prediction models considering text influence, we construct a benchmark for the established SJTU-TIS database using state-of-the-art saliency models. Finally, considering the effect of text descriptions on visual attention, while most existing saliency models ignore this impact, we further propose a text-guided saliency (TGSal) prediction model, which extracts and integrates both image features and text features to predict the image saliency under various text-description conditions. Our proposed model significantly outperforms the state-of-the-art saliency models on both the SJTU-TIS database and the pure image saliency databases in terms of various evaluation metrics. The SJTU-TIS database and the code of the proposed TGSal model will be released at: https://github.com/IntMeGroup/TGSal.

Assessing saliency models of observers' visual attention on acquired facial differences.

Saliency models that predict observers' visual attention to facial differences could enable psychosocial interventions to help patients and their families anticipate staring behaviors. The purpose of this study was to assess the ability of existing saliency models to predict observers' visual attention to acquired facial differences arising from head and neck cancer and its treatment. Saliency maps predicted by graph-based visual saliency (GBVS), an artificial neural network (ANN), and a face-specific model were compared to observer fixation maps generated from eye-tracking of lay observers presented with clinical facial photographs of patients with a visible or functional impairment manifesting in the head and neck region. We used a linear mixed-effects model to investigate observer and stimulus factors associated with the saliency models' accuracy. The GBVS model predicted many irrelevant regions (e.g., shirt collars) as being salient. The ANN model underestimated observers' attention to facial differences relative to the central region of the face. Compared with GBVS and ANN, the face-specific saliency model was more accurate on this task; however, the face-specific model underestimated the saliency of deviations from the typical structure of human faces. The linear mixed-effects model revealed that the location of the facial difference (midface versus periphery) was significantly associated with saliency model performance. Model performance was also significantly impacted by interobserver variability. Existing saliency models are not adequate for predicting observers' visual attention to facial differences. Extensions of face-specific saliency models are needed to accurately predict the saliency of acquired facial differences arising from head and neck cancer and its treatment.

Salient Objects in Clutter.

In this paper, we identify and address a serious design bias of existing salient object detection (SOD) datasets, which unrealistically assume that each image should contain at least one clear and uncluttered salient object. This design bias has led to a saturation in performance for state-of-the-art SOD models when evaluated on existing datasets. However, these models are still far from satisfactory when applied to real-world scenes. Based on our analyses, we propose a new high-quality dataset and update the previous saliency benchmark. Specifically, our dataset, called Salient Objects in Clutter (SOC), includes images with both salient and non-salient objects from several common object categories. In addition to object category annotations, each salient image is accompanied by attributes that reflect common challenges in common scenes, which can help provide deeper insight into the SOD problem. Further, with a given saliency encoder, e.g., the backbone network, existing saliency models are designed to achieve mapping from the training image set to the training ground-truth set. We therefore argue that improving the dataset can yield higher performance gains than focusing only on the decoder design. With this in mind, we investigate several dataset-enhancement strategies, including label smoothing to implicitly emphasize salient boundaries, random image augmentation to adapt saliency models to various scenarios, and self-supervised learning as a regularization strategy to learn from small datasets. Our extensive results demonstrate the effectiveness of these tricks. We also provide a comprehensive benchmark for SOD, which can be found in our repository: https://github.com/DengPingFan/SODBenchmark.

Multiscale Deep Network with Centerness-Aware Loss for Salient Object Detection

Deep encoder-decoder networks have been adopted for saliency detection and achieved state-of-the-art performance. However, most existing saliency models usually fail to detect very small salient objects. In this paper, we propose a multitask architecture, M2Net, and a novel centerness-aware loss for salient object detection. The proposed M2Net aims to solve saliency prediction and centerness prediction simultaneously. Specifically, the network architecture is composed of a bottom-up encoder module, top-down decoder module, and centerness prediction module. In addition, different from binary cross entropy, the proposed centerness-aware loss can guide the proposed M2Net to uniformly highlight the entire salient regions with well-defined object boundaries. Experimental results on five benchmark saliency datasets demonstrate that M2Net outperforms state-of-the-art methods on different evaluation metrics.

Fully Squeezed Multiscale Inference Network for Fast and Accurate Saliency Detection in Optical Remote-Sensing Images

Recently, salient object detection in optical remote-sensing images (RSIs) has received more and more attention. To tackle the challenges of RSIs including large-scale variation of objects, cluttered background, irregular shape of objects, and big difference in illumination, the cutting-edge convolutional neural network (CNN)-based models are proposed and have achieved an encouraging performance. However, the performance of the top-level models usually depends on the large model size and high computational cost, which limits their practical applications. To remedy the issue, we introduce a fully squeezed multiscale (FSM) module to equip the entire network. Specifically, the FSM module squeezes the feature maps from high dimension to low dimension and introduces the multiscale strategy to endow the capability of feature characterization with different receptive fields and different contexts. Based on the FSM module, we build the FSM inference network (FSMI-Net) to pop-out salient objects from optical RSIs, which is with fewer parameters and fast inference speed. Particularly, the proposed FSMI-Net only contains 3.6M parameters, and its GPU running speed is about 28 fps for <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$384 \times 384$ </tex-math></inline-formula> inputs, which is superior to the existing saliency models targeting optical RSIs. Extensive comparisons are performed on two public optical RSIs datasets, and our FSMI-Net achieves comparable detection accuracy when compared with the state-of-the-art models, where our model realizes a balance between the computational cost and detection performance.

Edge-Aware Multiscale Feature Integration Network for Salient Object Detection in Optical Remote Sensing Images

The optical remote sensing images (RSIs) show various spatial resolutions and cluttered background, where salient objects with different scales, types, and orientations are presented in diverse RSI scenes. Therefore, it is inappropriate to directly extend cutting-edge saliency detection methods for conventional RGB images to optical RSIs. Besides, the existing saliency models targeting RSIs often render imperfect saliency maps, where some of them are with coarse boundary details. To solve this problem, this article attempts to introduce the edge information to precisely detect salient objects in RSIs. Accordingly, we propose an edge-aware multiscale feature integration network (EMFI-Net) for salient object detection by conducting multiscale feature integration under the explicit and implicit assistance of salient edge cues. Specifically, our network contains two parts including the encoder and decoder. First, the encoder extracts multiscale deep features from three RSIs with different resolutions, where the high-level deep semantic features from three RSIs are integrated using a cascaded feature fusion module. Second, the encoder explicitly enriches the multiscale deep features by integrating the salient edge cues extracted by a salient edge extraction module. Meanwhile, we also implicitly deploy an edge-aware constraint to the supervision of the saliency map prediction by introducing a hybrid loss function. Finally, the decoder integrates the enriched multiscale deep features in a coarse-to-fine way, yielding a high-quality saliency map. The experiments conducted on two public optical RSI datasets clearly prove the effectiveness and superiority of the proposed EMFI-Net against the state-of-the-art saliency models.

Multi-Visual Feature Saliency Detection for Sea-Surface Targets through Improved Sea-Sky-Line Detection

To visually detect sea-surface targets, the objects of interest must be effectively and rapidly isolated from the background of sea-surface images. In contrast to traditional image detection methods, which employ a single visual feature, this paper proposes a significance detection algorithm based on the fusion of multi-visual features after detecting the sea-sky-lines. The gradient edges of the sea-surface images are enhanced using a Gaussian low-pass filter to eliminate the effect of the image gradients pertaining to the clouds, wave points, and illumination. The potential region and points of the sea-sky-line are identified. The sea-sky-line is fitted through polynomial iterations to obtain a sea-surface image containing the target object. The saliency subgraphs of the high and low frequency, gradient texture, luminance, and color antagonism features are fused to obtain an integrated saliency map of the sea-surface image. The saliency target area of the sea surface is segmented. The effectiveness of the proposed method was verified. The average detection rate and time for the sea-sky-line detection were 96.3% and 1.05 fps, respectively. The proposed method outperformed the existing saliency models on the marine obstacle detection dataset and Singapore maritime dataset, with mean absolute errors of 0.075 and 0.051, respectively.

Can we accurately predict where we look at paintings?

The objective of this study is to investigate and to simulate the gaze deployment of observers on paintings. For that purpose, we built a large eye tracking dataset composed of 150 paintings belonging to 5 art movements. We observed that the gaze deployment over the proposed paintings was very similar to the gaze deployment over natural scenes. Therefore, we evaluate existing saliency models and propose a new one which significantly outperforms the most recent deep-based saliency models. Thanks to this new saliency model, we can predict very accurately what are the salient areas of a painting. This opens new avenues for many image-based applications such as animation of paintings or transformation of a still painting into a video clip.

CMP-based saliency model for stereoscopic omnidirectional images

Although many saliency models for images had been proposed, the saliency model for stereoscopic omnidirectional image (SOI) is still a very important but not yet deeply studied topic. In addition, one of main disadvantages of the existing saliency models is difficult to simulate the characteristics of a wide field of view (FOV) of SOIs. To solve the problems, this paper proposes a novel cube map projection (CMP) based saliency model for SOIs. Considering that the equi-rectangular projection (ERP) representation of SOI will result in obvious stretch distortion, and what the human eyes view are the viewport images by using head mounted display (HMD), we construct the saliency model of the SOI in its less shape distorted CMP plane. First, the SOI with ERP format is converted into the CMP format. In order to reduce the boundary effect between the faces in CMP plane, its four horizontal faces are moved to obtain the horizontal global face and the horizontal local faces for which the corresponding saliency maps are subsequently computed by combining color distance and depth distance in the graph model. Then, to establish the correlations of these faces, the horizontal face saliency map is calculated by weighting the horizontal global face saliency map to the horizontal local face saliency map. Meanwhile, the vertical face saliency map is defined and obtained. Finally, the horizontal face saliency map and vertical face saliency map are spliced and re-projected to ERP plane to obtain the final saliency map of the ERP-mapped SOI. The experiments on the public ODI database are performed to compare the proposed methods with the state-of-the-art methods, and the results show that the proposed method achieves better performance of estimating SOI saliency map in terms of six well-known quantitative metrics.

PiCANet: Pixel-wise Contextual Attention Learning for Accurate Saliency Detection.

Existing saliency models typically incorporate contexts holistically. However, for each pixel, usually only part of its context region contributes to saliency prediction, while other parts are likely either noise or distractions. In this paper, we propose a novel pixel-wise contextual attention network (PiCANet) to selectively attend to informative context locations at each pixel. The proposed PiCANet generates an attention map over the contextual region of each pixel and construct attentive contextual features via selectively incorporating the features of useful context locations. We present three formulations of the PiCANet via embedding the pixel-wise contextual attention mechanism into the pooling and convolution operations with attending to global or local contexts. All the three models are fully differentiable and can be integrated with convolutional neural networks with joint training. In this work, we introduce the proposed PiCANets into a U-Net model for salient object detection. The generated global and local attention maps can learn to incorporate global contrast and regional smoothness, which help localize and highlight salient objects more accurately and uniformly. Experimental results show that the proposed PiCANets perform effectively for saliency detection against the state-of-the-art methods. Furthermore, we demonstrate the effectiveness and generalization ability of the PiCANets on semantic segmentation and object detection with improved performance.

Saliency-aware inter-image color transfer for image manipulation

This paper proposes a novel saliency-aware inter-image color transfer method to perform image manipulation. Specifically, given the source image, the candidate images are first retrieved from a group of images with the same semantic category, and the corresponding saliency maps are obtained using an existing saliency model. Then, the inter-image color transfer method is proposed to transfer the colors of the high-saliency region in each candidate image to the target object region in the source image, for generating the manipulated image. Finally, from a set of manipulated images, the one with the highest weighted F-measure of its saliency map is selected as the final result. Experimental results show that the proposed method not only highlights objects effectively but also preserves the naturalness of images well, and consistently outperforms other image manipulation methods when viewing the manipulated images with or without the source image as the reference.

Salient object detection by local and global manifold regularized SVM model

In this paper, a bottom-up saliency detection algorithm is proposed by introducing the manifold regularized Support Vector Machine (SVM) model. The local manifold regularization can well capture local saliency cues. However, its local-to-global mechanism often ignores the completeness of salient objects especially when the contrast between the foreground and background is low, while the global regularization can discover the long-range semantic structure but easily neglect the details. To comprehensively utilize their advantages, we respectively construct a local regularizer (LR) and a global regularizer (GR), and incorporate them into the SVM models, which are then self-trained across superpixels by pseudo labeled ones. The LR-SVM and GR-SVM models are further refined by updating training sample with higher-level object proposals. Extensive experiments on five benchmark datasets demonstrate that the proposed algorithm performs favorably against the state-of-the-art saliency detection methods. In addition, we show that the manifold regularized SVM model can be easily applied to some existing saliency models and achieve significant performance improvement.

Mutual Guidance-Based Saliency Propagation for Infrared Pedestrian Images

Saliency detection is important in computer vision. However, most of the existing saliency models are designed for visible images. It is still a challenging problem to apply saliency detection algorithms on infrared images. In this paper, an effective propagation based saliency detection method for infrared pedestrian images is proposed. Firstly, based on the thermal characteristics of infrared images and thermal radiation models, a thermal analysis based saliency (TAS) is introduced. TAS measures the stableness of pedestrians based on maximally stable extremal regions, which is further improved by an intensity filter. Then, by taking into account the appearance characteristic of pedestrians, an appearance analysis weighted saliency (AAS) is proposed which combines the intensity and shape features of pedestrians to improve the intensity contrast. Finally, besides the commonly used intra-scale neighborhood, an inter-scale neighborhood is introduced to jointly construct a mutual guidance-based saliency propagation model. This model could simultaneously integrate the saliency features and improve the saliency performance. Two datasets DIP and IMS with 600 infrared pedestrian images are published. Then, extensive experiments and comparisons with state-of-the-art methods demonstrate the effectiveness of the proposed saliency method for infrared pedestrian images.

Improving Video Saliency Detection via Localized Estimation and Spatiotemporal Refinement

Video saliency detection aims to pop out the most salient regions in every frame of a video. Up to now, many efforts have been made from various aspects for video saliency detection. Unfortunately, the existing video saliency models are very likely to fail in challenging videos with complicated motions and complex scenes. Therefore, in this paper, we propose a novel framework to improve the saliency detection results generated by existing video saliency models. The proposed framework consists of three key steps including localized estimation, spatiotemporal refinement, and saliency update. Specifically, the initial saliency map of each frame in a video is first generated by using an existing saliency model. Then, by considering the temporal consistency and strong correlation among adjacent frames, the localized estimation models, which are generated by training the random forest regressor within a local temporal window, are employed to generate the temporary saliency map. Finally, by taking the appearance and motion information of salient objects into consideration, the spatiotemporal refinement step is deployed to further improve the temporary saliency map and generate the final saliency map. Furthermore, such an improved saliency map is then utilized to update the initial saliency map and provide reliable cues for saliency detection in the next frame. The experimental results on four challenging datasets demonstrate that the proposed framework is able to consistently and significantly improve the saliency detection performance of various video saliency models, thereby achieving the state-of-the-art performance.

Spatial–temporal multi-task learning for salient region detection

This paper proposes a novel multi-task learning based salient region detection method by fusing spatial and temporal features. Salient region detection has been widely used in various computer vision tasks, being as a general preprocessor to identify interest objects. Despite the recent successes, existing saliency models still lag behind the performance of human when visually perceives dynamic scenes. Most of the existing models largely rely on various spatial features. However, these spatial feature based methods have several deficiencies: (i) they can hardly adapt to the situation where moving objects are included, and (ii) they cannot model the human vision process in dynamic scenes. Recently, some saliency models introduce temporal features in their detecting process, such as the optical flow and stacking frames. The potential of temporal features for saliency optimization has been demonstrated. However, since temporal features in these models are merely used as a compensation to static features, the advantages of temporal features have not yet been fully explored. Aiming to comprehensively address these issues above, our method fuses spatial and temporal features, and learns the mapping relationship from various features to salient regions using our multi-task learning framework. The final salient region is generated by our unified Bayesian framework. The experimental results demonstrated that our proposed approach outperforms previous methods.

Saliency Optimization and Integration Via Iterative Bootstrap Learning

This paper proposes an effective method to elevate the performance of saliency detection via iterative bootstrap learning, which consists of two tasks including saliency optimization and saliency integration. Specifically, first, multiscale segmentation and feature extraction are performed on the input image successively. Second, prior saliency maps are generated using existing saliency models, which are used to generate the initial saliency map. Third, prior maps are fed into the saliency regressor together, where training samples are collected from the prior maps at multiple scales and the random forest regressor is learned from such training data. An integration of the initial saliency map and the output of saliency regressor is deployed to generate the coarse saliency map. Finally, in order to improve the quality of saliency map further, both initial and coarse saliency maps are fed into the saliency regressor together, and then the output of the saliency regressor, the initial saliency map as well as the coarse saliency map are integrated into the final saliency map. Experimental results on three public data sets demonstrate that the proposed method consistently achieves the best performance and significant improvement can be obtained when applying our method to existing saliency models.

GBVS360, BMS360, ProSal: Extending existing saliency prediction models from 2D to omnidirectional images

In this paper it is studied how existing visual saliency models designed for 2D images can be extended and applied to Omnidirectional images in the equirectangular format. Three different frameworks, BMS360, GBVS360 and Projected Saliency were designed to address this task. In the particular case of BMS360 and GBVS360, the 2D models Boolean Map Saliency (BMS) and Graph-Based Visual Saliency (GBVS) were rigorously extended to integrate the specific properties of equirectangular images to model visual attention in omnidirectional images watched using Head Mounted Displays (HMDs). With the proposed extensions, the saliency prediction performance in comparison to their original design could significantly be improved. Another key result of this paper comes from the design of a framework called projected saliency which allows applying existing saliency models on equirectangular images without requiring in-depth adaptation of the prediction algorithms. The projected saliency framework enabled to study the interaction of features in the feature activation process of computational saliency models. It shows that the activation should not be done per-viewport, but should account for neighboring regions. As a consequence, we argue that the feature activation process should be performed at a global level instead of a local one. Here, the use of a local approach with a large FOV was found to be a good compromise between global activation and appropriate feature computation to directly port existing saliency models designed for rectilinear images to the domain of equirectangular images. In this paper an adaptive equatorial prior is also described. This equatorial prior allows to account for pitch shift of the horizon line. Finally, results show that the newly designed BMS360 and GVBS360 outperform even recent state-of-the-art models at head motion prediction and are competitive at head/eye saliency-map prediction. Moreover, all models and their code are made publicly available on GitHub allowing to further iterate on this work.

Saliency integration driven by similar images

This paper proposes a saliency integration approach via the use of similar images to elevate saliency detection performance. Given the input image, a group of similar images are first retrieved, and meanwhile, the corresponding multiple saliency maps of the input image are generated by using existing saliency models. Then, the saliency fusion map is generated by using an adaptive fusion method to integrate such saliency maps, for which the fusion weights are measured by the corresponding similarity between each similar image and the input image. Next, an inter-image graph, for each pair of input image and similar image, is constructed to propagate the confident saliency values from the similar image to the input image, yielding the saliency propagation map. Finally, the saliency fusion map and the saliency propagation map are integrated to obtain the final saliency map. Experimental results on two public datasets demonstrate that the proposed approach achieves the better saliency detection performance compared to the existing saliency models and other saliency integration approaches.

Stereoscopic saliency model using contrast and depth-guided-background prior

Abstract Many successful models of saliency have been proposed to detect salient regions for 2D images. Because stereopsis, with its distinctive depth information, influences human viewing, it is necessary for stereoscopic saliency detection to consider depth information as an additional cue. In this paper, we propose a 3D stereoscopic saliency model based on both contrast and depth-guided-background prior. First, a depth-guided-background prior is specifically detected from a disparity map apart from the conventional prior, assuming boundary super-pixels as background. Then, saliency based on disparity with the help of the proposed prior is proposed to prioritize the contrasts among super-pixels. In addition, a scheme to combine the contrast of disparity and the contrast of color is presented. Finally, 2D spatial dissimilarity features are further employed to refine the saliency map. Experimental results on the PSU stereo saliency benchmark dataset (SSB) show that the proposed method performs better than existing saliency models.