Final Saliency Map Research Articles

Abs-CAM: a gradient optimization interpretable approach for explanation of convolutional neural networks

The black-box nature of deep neural networks severely hinders its performance improvement and application in specific scenes. In recent years, class activation mapping-based method has been widely used to interpret the internal decisions of models in computer vision tasks. However, when this method uses backpropagation to obtain gradients, it will cause noise in the saliency map and even locate features that are irrelevant to decisions. In this paper, we propose an absolute value class activation mapping-based (Abs-CAM) method, which optimizes the gradients derived from the backpropagation and turns all of them into positive gradients to enhance the visual features of output neurons’ activation and improve the localization ability of the saliency map. The framework of Abs-CAM is divided into two phases: generating initial saliency map and generating final saliency map. The first phase improves the localization ability of the saliency map by optimizing the gradient, and the second phase linearly combines the initial saliency map with the original image to enhance the semantic information of the saliency map. We conduct qualitative and quantitative evaluation of the proposed method, including Deletion, Insertion, and Pointing Game. The experimental results show that the Abs-CAM can obviously eliminate the noise in the saliency map, and can better locate the features related to decisions, and is superior to the previous methods in recognition and localization tasks.

Saliency detection for surface defects of ceramic tile

Considering the lack of techniques for automatically detecting the surface defects of ceramic tiles, owing to the numerous types and complex surface textures of ceramic tiles, a salient region detection (SRD) algorithm is proposed. After the superpixel segmentation of the surface image of ceramic tiles, colour and structure saliency maps were constructed by taking the differences between the defects and backgrounds, in terms of colour and structure, as the salient values. The final saliency map for the surface defects was then obtained through the fusion of these two maps. The salient regions in the map were determined using the connected domain method and were then marked using minimum circumscribed rectangles for detecting the surface defects. The feasibility and effectiveness of the proposed SRD algorithm was verified through comparative experiments. The experimental results show that the detection accuracy of the algorithm for unknown types of ceramic tiles is 92.60%, which was 5.40% higher than that of the ResNet-50 algorithm, which was the highest for known types of ceramic tiles.

Saliency Detection Using a Bio-inspired Spiking Neural Network Driven by Local and Global Saliency

ABSTRACT The detection of the most salient parts of images as objects in salient object detection tasks mimics human behavior, which is useful for a variety of computer vision applications. In this paper, the Local and Global Saliency Driven Dual-Channel Pulse Coupled Neural Network (LGSD-DCPCNN) model is used to provide a novel strategy for saliency detection. To achieve visually homogeneous sections and save computation costs, the input image is first subjected to superpixel segmentation. The global and local saliency maps are then created using the segmented image’s position, color, and textural properties. The LGSD-DCPCNN network is activated using these saliency maps to extract visually consistent features from the input maps to provide the final saliency map. An extensive qualitative and quantitative performance study is undertaken to assess the efficacy of the proposed method. When compared to state-of-the-art approaches, the experimental results show a considerable improvement in the detection of salient regions. Quantitative analysis of the proposed method reveals a significant improvement in the area under the ROC curve (AUC) score, F-measure score, and mean absolute error (MAE) score. The qualitative analysis describes the proposed algorithm’s ability to detect multiple salient objects accurately while maintaining significant border preservation.

Multi-Color Space Network for Salient Object Detection.

The salient object detection (SOD) technology predicts which object will attract the attention of an observer surveying a particular scene. Most state-of-the-art SOD methods are top-down mechanisms that apply fully convolutional networks (FCNs) of various structures to RGB images, extract features from them, and train a network. However, owing to the variety of factors that affect visual saliency, securing sufficient features from a single color space is difficult. Therefore, in this paper, we propose a multi-color space network (MCSNet) to detect salient objects using various saliency cues. First, the images were converted to HSV and grayscale color spaces to obtain saliency cues other than those provided by RGB color information. Each saliency cue was fed into two parallel VGG backbone networks to extract features. Contextual information was obtained from the extracted features using atrous spatial pyramid pooling (ASPP). The features obtained from both paths were passed through the attention module, and channel and spatial features were highlighted. Finally, the final saliency map was generated using a step-by-step residual refinement module (RRM). Furthermore, the network was trained with a bidirectional loss to supervise saliency detection results. Experiments on five public benchmark datasets showed that our proposed network achieved superior performance in terms of both subjective results and objective metrics.

Ship detection in SAR images by saliency analysis of multiscale superpixels

ABSTRACT This paper proposes a new ship detection algorithm via saliency analysis on multiscale superpixels. First, Synthetic Aperture Radar (SAR) imagery is over-segmented into many homogenous regions, called superpixels. In contrast to prior work, these superpixels are generated with various scales to handle size variation of ships in the SAR imagery. At each scale, local contrast and ship geometrical characteristics are considered to enhance the saliency of ship pixels. Then, the final saliency map is computed by integrating saliency values of all scales. Experimental results on two real SAR images demonstrate the efficiency of the proposed algorithm.

Salient Object Detection by LTP Texture Characterization on Opposing Color Pairs under SLICO Superpixel Constraint.

The effortless detection of salient objects by humans has been the subject of research in several fields, including computer vision, as it has many applications. However, salient object detection remains a challenge for many computer models dealing with color and textured images. Most of them process color and texture separately and therefore implicitly consider them as independent features which is not the case in reality. Herein, we propose a novel and efficient strategy, through a simple model, almost without internal parameters, which generates a robust saliency map for a natural image. This strategy consists of integrating color information into local textural patterns to characterize a color micro-texture. It is the simple, yet powerful LTP (Local Ternary Patterns) texture descriptor applied to opposing color pairs of a color space that allows us to achieve this end. Each color micro-texture is represented by a vector whose components are from a superpixel obtained by the SLICO (Simple Linear Iterative Clustering with zero parameter) algorithm, which is simple, fast and exhibits state-of-the-art boundary adherence. The degree of dissimilarity between each pair of color micro-textures is computed by the FastMap method, a fast version of MDS (Multi-dimensional Scaling) that considers the color micro-textures’ non-linearity while preserving their distances. These degrees of dissimilarity give us an intermediate saliency map for each RGB (Red–Green–Blue), HSL (Hue–Saturation–Luminance), LUV (L for luminance, U and V represent chromaticity values) and CMY (Cyan–Magenta–Yellow) color space. The final saliency map is their combination to take advantage of the strength of each of them. The MAE (Mean Absolute Error), MSE (Mean Squared Error) and F measures of our saliency maps, on the five most used datasets show that our model outperformed several state-of-the-art models. Being simple and efficient, our model could be combined with classic models using color contrast for a better performance.

An Improved Saliency Detection Algorithm Based on Edge Boxes and Bayesian Model

Visual saliency detection aims to extract salient objects from the original image, making it less complicated to process the image. This paper combines an edge box algorithm with Bayesian theory to detect salient objects. The proposed saliency detection algorithm transforms the process of traditional detection method, and prioritizes the positioning of significant objects. Firstly, the Harris corners of the original image were calculated, and clustered by the improved clustering algorithm, yielding the number of salient objects in the image. Then, all possible positions of salient objects in the image were framed by the edge box algorithm, and the boxes were sorted in descending order of the score. According to the number N of clusters of the image corners, the N top-ranking boxes were selected to determine the salient regions. In this way, the position and number of salient objects were clarified. Based on the selected salient regions, the final saliency map was calculated by improved geodesic distance and Bayesian model. Experimental results show that our approach performed better than 11 existing algorithms in both simple and relatively complex scenes. In terms of objective performance, the accuracy and recall of our algorithm on MSRA10k, ECSSD, DUT-OMRON and SED2 datasets were higher than that of the other algorithms.

Multiscale Balanced-Attention Interactive Network for Salient Object Detection

The purpose of saliency detection is to detect significant regions in the image. Great progress on salient object detection has been made using from deep-learning frameworks. How to effectively extract and integrate multiscale information with different depths is an open problem for salient object detection. In this paper, we propose a processing mechanism based on a balanced attention module and interactive residual module. The mechanism addressed the acquisition of the multiscale features by capturing shallow and deep context information. For effective information fusion, a modified bi-directional propagation strategy was adopted. Finally, we used the fused multiscale information to predict saliency features, which were combined to generate the final saliency maps. The experimental results on five benchmark datasets show that the method is on a par with the state of the art for image saliency datasets, especially on the PASCAL-S datasets, where the MAE reaches 0.092, and on the DUT-OMROM datasets, where the F-measure reaches 0.763.

Visual Saliency via Multiscale Analysis in Frequency Domain and Its Applications to Ship Detection in Optical Satellite Images.

This article proposes a bottom-up visual saliency model that uses the wavelet transform to conduct multiscale analysis and computation in the frequency domain. First, we compute the multiscale magnitude spectra by performing a wavelet transform to decompose the magnitude spectrum of the discrete cosine coefficients of an input image. Next, we obtain multiple saliency maps of different spatial scales through an inverse transformation from the frequency domain to the spatial domain, which utilizes the discrete cosine magnitude spectra after multiscale wavelet decomposition. Then, we employ an evaluation function to automatically select the two best multiscale saliency maps. A final saliency map is generated via an adaptive integration of the two selected multiscale saliency maps. The proposed model is fast, efficient, and can simultaneously detect salient regions or objects of different sizes. It outperforms state-of-the-art bottom-up saliency approaches in the experiments of psychophysical consistency, eye fixation prediction, and saliency detection for natural images. In addition, the proposed model is applied to automatic ship detection in optical satellite images. Ship detection tests on satellite data of visual optical spectrum not only demonstrate our saliency model's effectiveness in detecting small and large salient targets but also verify its robustness against various sea background disturbances.

Two-Stage Edge Reuse Network for Salient Object Detection of Strip Steel Surface Defects

Recently, researchers have paid more attention to salient object detection of strip steel surface defects, but there are lots of difficulties, such as a mass of noise, blurry defect boundaries, complex backgrounds, and various types of defects. Existing image salient object detection methods fail to overcome the above challenging scenes. To address them, in this paper, we propose a novel Two-Stage Edge Reuse Network (TSERNet), which consists of two stages, <i>i.e</i>., prediction and refinement. In the first prediction stage, we construct a primary net based on the encoder-decoder architecture. The encoder not only extracts multi-scale features but also generates an edge map. In the decoder, we propose a novel Edge-aware Foreground-Background Integration (EFBI) module to distinguish foreground and background through edge features and reverse attention mechanism, and exploit the decoder to generate an initial saliency map. In the second refinement stage, we construct a sub-net based on the same architecture as that of the first stage. Its encoder extracts features from the initial saliency map, and its decoder deploys the Edge-aware Refinement Module (ERM), which reuses the edge map generated from the first stage, to enhance these features for purifying the initial saliency map, resultting in the final saliency map. Comprehensive experiments on the public dataset show that our proposed TSERNet is consistently superior to 22 relevant state-of-the-art methods. The code and results of our method are available at https://github.com/monxxcn/TSERNet.

Dense Attention-Guided Cascaded Network for Salient Object Detection of Strip Steel Surface Defects

Recently, more and more researchers have paid attention to the surface defect detection of strip steel. However, the performance of existing methods usually fails to detect the defect regions from some complex scenes, especially with the noise disturbance and diverse defect types. Therefore, this article proposes an end-to-end dense attention-guided cascaded network (DACNet) to detect salient objects (i.e., defects) on the strip steel surface, where the proposed DACNet is a U-shape network including an encoder and a decoder. The encoder first deploys multiresolution convolutional branches (i.e., high/medium/low) in a cascaded way. Concretely, the cascaded feature integration (CFI) unit fuses the deep features from the last convolutional blocks of multiresolution branches, yielding the enhanced high-level deep semantic feature. Subsequently, coupled with the multilevel deep features from high-resolution branch, the new multiscale deep features are capable of characterizing various defects. Then, driven by the dense attention mechanism which enables the deeper attention cues flow into decoding stages, the decoder progressively integrates the multiscale deep features into the final saliency map, where the dense attention is designed to steer deep features pay more concerns to the defect regions. Comprehensive experiments are conducted on the public strip steel datasets, and the experimental results demonstrate that our model consistently outperforms the state-of-the-art models in all evaluation metrics.

RGB-D Image Saliency Detection Based on Cross-Model Feature Fusion

<p indent=0mm>In order to solve the problem that saliency detection based on RGB images cannot accurately detect salient object in scenes such as multiple targets or small targets, a novel saliency detection network based on RGB-D cross-modal feature fusion is proposed. The network takes the improved fully convolutional network (FCN) as the dual-stream backbone network, extracts the color and depth features and makes predictions respectively, and finally uses the inception structure fusion to generate the final saliency map. Aiming at the problem that the actual receptive field of the original FCN is far lower than the theoretical receptive field, and the global image information is not really used, a dual-branch structure global and local feature extraction block is designed. The global feature branch is used to extract global information and guide local feature extraction. This builds an improved FCN. In addition, considering the differences between color and depth features at different levels, a cross-modal feature fusion module is proposed, which uses dot product to selectively fuse color and depth features. Compared with addition and cascade, it uses dot multiplication. It can effectively reduce noise and redundant information. Comprehensive experiments on three benchmark datasets demonstrate that compared with 21 mainstream networks, this model is basically in the top three levels in terms of <italic>S</italic> value, <italic>F</italic> value and MAE. At the same time, the size of the model is analyzed. In comparison, its size is only 4.7% of MMCI, which has a decrease of 22.8% compared with the existing smallest model A2dele.

A Computational-Cognitive Model of Visual Attention in Dynamic Environments

Background and Objectives: Visual attention is a high order cognitive process of human brain which defines where a human observer attends. Dynamic computational visual attention models are modeled on the behavior of the human brain and can predict what areas a human will pay attention to when viewing a scene such as a video. However, several types of computational models have been proposed to provide a better understanding of saliency maps in static and dynamic environments, most of these models are used for specific scenes. In this paper, we propose a model that can generate saliency maps in a variety of dynamic environments with complex scenes.Methods: We used a deep learner as a mediating network to combine basic saliency maps with appropriate weighting. Each of these basic saliency maps covers an important feature of human visual attention, and ultimately the final saliency map is very similar to human visual behavior.Results: The proposed model is run on two datasets and the generated saliency maps are evaluated by different criteria such as ROC, CC, NSS, SIM and KLdiv. The results show that the proposed model has a good performance compared to other similar models.Conclusion: The proposed model consists of three main parts, including basic saliency maps, gating network, and combinator. This model was implemented on the ETMD dataset and the resulting saliency maps (visual attention areas) were compared with some other models in this field by evaluation criteria and their results were evaluated. The results obtained from the proposed model are acceptable and based on the accepted evaluation criteria in this area, it performs better than similar models.

Visual Saliency Estimation through Manifold Learning

Saliency detection has been a desirable way for robotic vision to find the most noticeable objects in a scene. In this paper, a robust manifold-based saliency estimation method has been developed to help capture the most salient objects in front of robotic eyes, namely cameras. In the proposed approach, an image is considered as a manifold of visual signals (stimuli) spreading over a connected grid, and local visual stimuli are compared against the global image variation to model the visual saliency. With this model, manifold learning is then applied to minimize the local variation while keeping the global contrast, and turns the RGB image into a multi-channel image. After the projection through manifold learning, histogram-based contrast is then computed for saliency modeling of all channels of the projected images, and mutual information is introduced to evaluate each single-channel saliency map against prior knowledge to provide cues for the fusion of multiple channels. In the last step, the fusion procedure combines all single-channel saliency maps according to their mutual information score, and generates the final saliency map. In our experiment, the proposed method is evaluated using one of the largest publicly available image datasets. The experimental results demonstrate that our algorithm consistently outperforms the state-of-the-art unsupervised saliency detection methods, yielding higher precision and better recall rates. Furthermore, the proposed method is tested on a video-type test dataset where a moving camera is trying to catch up with the walking person---a salient object in the video sequence. Our experimental results show that the proposed approach can successful accomplish this task, revealing its potential use for similar robotic applications.

Multiple Kernel Boosting Saliency Detection of Flame Image of Sintering Section

<p indent=0mm>The flame image of the sintering section is usually interfered by smoke and halo, which would cause the blurring of flame edge and material layer in the image. In order to solve the problem that traditional saliency detection method based on two-dimensional image features is difficult to effectively obtain the actual saliency of the cross-sectional flame image, a saliency detection method based on boundary connectivity and multi-kernel Boosting is proposed. Firstly, the color de-correlation is used in the process of image color space conversion. Boundary connectivity and dark channel prior principle are used to obtain the initial saliency map. Secondly, the super-pixel region information, regional variance and regional contrast of the flame image are used to construct region descriptor, the multiple kernel Boosting algorithm based on support vector regression is used to generate the complementary saliency map on 4 scales. Finally, the initial saliency map and the complementary saliency map are fused to obtain the final saliency map. 600 flame images including manual labeling are used to compare the proposed method with other 5 existing methods and each stage of the proposed method is analyzed. <italic>P</italic>-<italic>R</italic> curve, <italic>F</italic>-measure, mean absolute error and running time are taken as evaluation indexes. The experimental results show that the proposed method is superior to the other 5 methods, and the detection performance of each stage is gradually enhanced, which lays a foundation for improving the effective information extraction of sintering flame section image.

Markov Chain and Adaboost Image Saliency Detection Algorithm Based on Conditional Random Field

The traditional salient object detection algorithms are used to apply the underlying features and prior knowledge of the images. Based on conditional random field Markov chain and Adaboost image saliency detection technology, a saliency detection method is proposed to effectively reduce the error caused by the target approaching the edge, which mainly includes the use of absorption Markov chain model to generate the initial saliency map. In this model, the transition probability of each node is defined by the difference of color and texture between each super pixel, and the absorption time of the transition node is calculated as the significant value of each super pixel. A strong classifier optimization model based on Adaboost iterative algorithm is designed.The initial saliency map is processed by the classifier to obtain an optimized saliency map, which highlights the global contrast. In order to extract the saliency region of the final saliency map, a method using conditional random field is designed to segment and extract the saliency region. The results show that the saliency area detected by this method is prominent, the overall contour is clear and has high resolution. At the same time, this method has better performance in accuracy recall curve and histogram.

3MNet: Multi-task, multi-level and multi-channel feature aggregation network for salient object detection.

Salient object detection is a hot spot of current computer vision. The emergence of the convolutional neural network (CNN) greatly improves the existing detection methods. In this paper, we present 3MNet, which is based on the CNN, to make the utmost of various features of the image and utilize the contour detection task of the salient object to explicitly model the features of multi-level structures, multiple tasks and multiple channels, so as to obtain the final saliency map of the fusion of these features. Specifically, we first utilize contour detection task for auxiliary detection and then utilize use multi-layer network structure to extract multi-scale image information. Finally, we introduce a unique module into the network to model the channel information of the image. Our network has produced good results on five widely used datasets. In addition, we also conducted a series of ablation experiments to verify the effectiveness of some components in the network.

Visual attention prediction for Autism Spectrum Disorder with hierarchical semantic fusion

Visual attention for the diagnosis of Autism Spectrum Disorder (ASD) which is a kind of mental disorder has attracted the interests of increasing number of researchers. Although multiple visual attention prediction models have been proposed, this problem is still open. In this paper, considering the shift of visual attention, we propose that an image can be viewed as a pseudo sequence. Besides, we propose a novel visual attention prediction method for ASD with hierarchical semantic fusion (ASD-HSF). Specifically, the proposed model mainly contains a Spatial Feature Module (SFM) and a Pseudo Sequential Feature Module (PSFM). SFM is designed to extract spatial semantic features with a fully convolutional network, while PSFM implemented by two Convolutional Long Short-Term Memory networks (ConvLSTMs) is applied to learn pseudo sequential features. And the outputs of these two modules are fused to extract the final saliency map which simultaneously includes spatial semantic information and pseudo sequential information. Experimental results show that the proposed model not only outperforms ten state-of-the-art general saliency prediction counterparts, but also reaches the first and the second ranks under four metrics and the rest ones of ASD saliency prediction respectively.

CHELM: Convex Hull based Extreme Learning Machine for salient object detection

Machine learning based saliency detection methods have achieved better performance than traditional methods. Here, we propose a machine learning based method that utilizes Convex Hull and Extreme Learning Machine (ELM) for detecting salient object(s) in an image. The novelty of the proposed method lies in the generation of training set without using human annotations. Initially, an input image is segmented using SLIC algorithm at different scales to produce multiscale segmented images. This is followed by estimating two different saliency priors viz. (a) Convex Hull center prior and (b) contrast prior for each segmented image. These priors exploit foreground center and spatially weighted contrast respectively. Both of these estimated priors help in computing initial saliency of each segment across all scales. For each scale, the initial saliency map along with the Convex Hull based label map is employed on a segmented image to determine the positive (salient) and negative (background) training set. Distinctive features for each segment belonging to the training set are extracted and then passes to the Extreme Learning Machine for learning the ELM model. Afterwards, multiscale saliency maps of an image are found by applying the learned ELM model on distinctive features extracted from each segment across multiple scales. These multiscale saliency maps are linearly combined to obtain the final saliency map. The effectiveness of the proposed method is supported through extensive experimental results performed on six publicly available datasets viz. MSRA10K, DUT-OMRON, ECSSD, PASCAL-S, SED2, and THUR15K. The performance of the proposed method was compared with 11 state-of-the-art methods in terms of Precision, Recall, F-Measure, Receiver Operating Characteristics (ROC), and Area under the curve (AUC). The proposed method outperforms or comparable with compared methods in terms of all the performance measures.

Saliency prediction on omnidirectional images with attention-aware feature fusion network

Recent years have witnessed rapid development of deep learning technology and its successful application in the saliency prediction of traditional 2D images. However, when using deep neural network (DNN) models to perform saliency prediction on omnidirectional images (ODIs), there are two critical issues: (1) The datasets for ODIs are small-scale that cannot support the training DNN-based models. (2) It is challenging to perform saliency prediction in that some ODIs contain complex background clutters. In order to solve these two problems, we propose a novel Attention-Aware Features Fusion Network (AAFFN) model which is first trained with traditional 2D images and then transferred to the ODIs for saliency prediction. Specifically, our proposed AAFFN model consists of three modules: a Part-guided Attention (PA) module, a Visibility Score (VS) module, and a Attention-Aware Features Fusion (AAFF) module. The PA module is used to extract precise features to estimate attention of the finer part on ODIs, and eliminate the influence of cluttered background. Meanwhile, the VS module is introduced to measure the proportion of the foreground and background parts and generate visibility scores in the feature learning process. Finally, in the AAFF module, we utilize the weighted fusion of attention maps and visibility scores to generate the final saliency map. Extensive experiments and ablation analysis demonstrate that the proposed model achieves superior performance and outperforms other state-of-the-art methods on public benchmark datasets.