Multi-scale Saliency Research Articles

Geometry Auxiliary Salient Object Detection for Light Fields via Graph Neural Networks.

Light field imaging, originated from the availability of light field capture technology, offers a wide range of applications in the field of computational vision. The capability of predicting salient objects of light fields remains technologically challenging due to its complicated geometry structure. In this paper, we propose a light field salient object detection approach that formulates the geometric coherence among multiple views of light fields as graphs, where the angular/central views represent the nodes and their relations compose the edges. The spatial and disparity correlations between multiple views are effectively explored through multi-scale graph neural networks, enabling the more comprehensive understanding of light field content and more representative and discriminative saliency features generation. Moreover, a multi-scale saliency feature consistency learning module is embedded to enhance the saliency features. Finally, an accurate salient object map is produced for the light field based upon the extracted features. In addition, we establish a new light field salient object detection dataset (CITYU-Lytro) that contains 817 light fields with diverse contents and their corresponding annotations, aiming to further promote the research on light field salient object detection. Quantitative and qualitative experiments demonstrate that the proposed method performs favorably compared with the state-of-the-art methods on the benchmark datasets.

Salient Object Detection Based on Multiscale Segmentation and Fuzzy Broad Learning

Abstract Saliency detection has been a hot topic in the field of computer vision. In this paper, we propose a novel approach that is based on multiscale segmentation and fuzzy broad learning. The core idea of our method is to segment the image into different scales, and then the extracted features are fed to the fuzzy broad learning system (FBLS) for training. More specifically, it first segments the image into superpixel blocks at different scales based on the simple linear iterative clustering algorithm. Then, it uses the local binary pattern algorithm to extract texture features and computes the average color information for each superpixel of these segmentation images. These extracted features are then fed to the FBLS to obtain multiscale saliency maps. After that, it fuses these saliency maps into an initial saliency map and uses the label propagation algorithm to further optimize it, obtaining the final saliency map. We have conducted experiments based on several benchmark datasets. The results show that our solution can outperform several existing algorithms. Particularly, our method is significantly faster than most of deep learning-based saliency detection algorithms, in terms of training and inferring time.

STA-Net: spatial-temporal attention network for video salient object detection

This paper conducts a systematic study on the role of spatial and temporal attention mechanism in the video salient object detection (VSOD) task. We present a two-stage spatial-temporal attention network, named STA-Net, which makes two major contributions. In the first stage, we devise a Multi-Scale-Spatial-Attention (MSSA) module to reduce calculation cost on non-salient regions while exploiting multi-scale saliency information. Such a sliced attention method offers an individual way to efficiently exploit the high-level features of the network with an enlarged receptive field. The second stage is to propose a Pyramid-Saliency-Shift-Aware (PSSA) module, which puts emphasis on the importance of dynamic object information since it offers a valid shift cue to confirm salient object and capture temporal information. Such a temporal detection module is able to encourage precise salient region detection. Exhaustive experiments show that the proposed STA-Net is effective for video salient object detection task, and achieves compelling performance in comparison with state-of-the-art.

Calcification segmentation based on a different scales superpixels saliency detection algorithm

Accurate detection of breast tumor calcifications is of great significance in assisting doctors’ diagnosis to improve the accuracy of breast cancer early detection. In this article, a different scale of superpixels saliency detection algorithm is used to segment calcifications in breast tumor ultrasound images based on a simple linear iterative cluster. First, a multi-scale saliency segmentation algorithm was used to divide the tumor region of different sizes and weak calcification (Wca) was extracted according to uneven gray distribution and texture contrast between regions. Second, based on single-scale superpixel segmentation of the original image, the strong calcification extraction map was calculated by measuring gray value difference and calcification gray distance features. Finally, the final calcification extraction map was obtained by combining the strong and weak calcification extraction maps. The detection algorithm proposed in this article could effectively detect calcifications in breast ultrasound images.

A Deep Multiscale Fusion Method via Low-Rank Sparse Decomposition for Object Saliency Detection Based on Urban Data in Optical Remote Sensing Images

The urban data provides a wealth of information that can support the life and work for people. In this work, we research the object saliency detection in optical remote sensing images, which is conducive to the interpretation of urban scenes. Saliency detection selects the regions with important information in the remote sensing images, which severely imitates the human visual system. It plays a powerful role in other image processing. It has successfully made great achievements in change detection, object tracking, temperature reversal, and other tasks. The traditional method has some disadvantages such as poor robustness and high computational complexity. Therefore, this paper proposes a deep multiscale fusion method via low-rank sparse decomposition for object saliency detection in optical remote sensing images. First, we execute multiscale segmentation for remote sensing images. Then, we calculate the saliency value, and the proposal region is generated. The superpixel blocks of the remaining proposal regions of the segmentation map are input into the convolutional neural network. By extracting the depth feature, the saliency value is calculated and the proposal regions are updated. The feature transformation matrix is obtained based on the gradient descent method, and the high-level semantic prior knowledge is obtained by using the convolutional neural network. The process is iterated continuously to obtain the saliency map at each scale. The low-rank sparse decomposition of the transformed matrix is carried out by robust principal component analysis. Finally, the weight cellular automata method is utilized to fuse the multiscale saliency graphs and the saliency map calculated according to the sparse noise obtained by decomposition. Meanwhile, the object priors knowledge can filter most of the background information, reduce unnecessary depth feature extraction, and meaningfully improve the saliency detection rate. The experiment results show that the proposed method can effectively improve the detection effect compared to other deep learning methods.

CoCNN: RGB-D deep fusion for stereoscopic salient object detection

Many convolutional neural network (CNN)-based approaches for stereoscopic salient object detection involve fusing either low-level or high-level features from the color and disparity channels. The former method generally produces incomplete objects, whereas the latter tends to blur object boundaries. In this paper, a coupled CNN (CoCNN) is proposed to fuse color and disparity features from low to high layers in a unified deep model. It consists of three parts: two parallel multilinear span networks, a cascaded span network and a conditional random field module. We first apply the multilinear span network to compute multiscale saliency predictions based on RGB and disparity individually. Each prediction, learned under separate supervision, utilizes the multilevel features extracted by the multilinear span network. Second, a proposed cascaded span network, under deep supervision, is designed as a coupling unit to fuse the two feature streams at each scale and integrate all fused features in a supervised manner to construct a saliency map. Finally, we formulate a constraint in the form of a conditional random field model to refine the saliency map based on the a priori assumption that objects with similar saliency values have similar colors and disparities. Experiments conducted on two commonly used datasets demonstrate that the proposed method outperforms previous state-of-the-art methods.

A novel image recognition approach using multiscale saliency model and GoogLeNet

It is very good to apply the saliency model in the visual selective attention mechanism to the preprocessing process of image recognition. However, the mechanism of visual perception is still unclear, so this visual saliency model is not ideal. To this end, this paper proposes a novel image recognition approach using multiscale saliency model and GoogLeNet. First, a multi-scale convolutional neural network was taken advantage of constructing multiscale salient maps, which could be used as filters. Second, an original image was combined with the salient maps to generate the filtered image, which highlighted the salient regions and suppressed the background in the image. Third, the image recognition task was implemented by adopting the classical GoogLeNet model. In this paper, many experiments were completed by comparing four commonly used evaluation indicators on the standard image database MSRA10K. The experimental results show that the recognition results of the test images based on the proposed method are superior to some stateof- the-art image recognition methods, and are also more approximate to the results of human eye observation.

结合多尺度视觉显著性的舰船目标检测

结合多尺度视觉显著性的舰船目标检测

Boundary-aware RGBD Salient Object Detection with Cross-modal Feature Sampling.

Mobile devices usually mount a depth sensor to resolve ill-posed problems, like salient object detection on cluttered background. The main barrier of exploring RGBD data is to handle the information from two different modalities. To cope with this problem, in this paper, we propose a boundary-aware cross-modal fusion network for RGBD salient object detection. In particular, to enhance the fusion of color and depth features, we present a cross-modal feature sampling module to balance the contribution of the RGB and depth features based on the statistics of their channel values. In addition, in our multi-scale dense fusion network architecture, we not only incorporate edge-sensitive losses to preserve the boundary of the detected salient region, but also refine its structure by merging the estimated saliency maps of different scales. We accomplish the multi-scale saliency map merging using two alternative methods which produce refined saliency maps via per-pixel weighted combination and an encoder-decoder network. Extensive experimental evaluations demonstrate that our proposed framework can achieve the state-of-the-art performance on several public RGBD-based datasets.

Multi-Scale Infrared Small Target Detection Method via Precise Feature Matching and Scale Selection Strategy

Infrared small target detection is a crucial and challenging topic for various applications. In recent years, the spectrum scale space (SSS) algorithm has shown considerable potential in the field of target detection. However, the SSS algorithm is prone to high false alarm rates in infrared small target detection scenarios with complex background. This paper proposes an improved SSS (ISSS) algorithm via precise feature matching and scale selection strategy for efficient infrared small target detection, which includes background suppression, feature matching and optimal scale selection three stages. In the background suppression stage, a matrix decomposition method named inexact augmented Lagrange multiplier (IALM) algorithm is used to extract the sparse image matrix from the original image as the target foreground image. In the feature matching stage, the 16 elaborate Gaussian kernel functions convolve with the the amplitude spectrum of target foreground image to generate 16 scale saliency maps that precisely match the feature of small targets. In the optimal scale selection stage, a few proper candidate scale maps are screened out according to the difference between the pixel values of the target area and the background clutters, in which the target area was more highlighted, and the scale map corresponding to the maximum value of local information entropy of the candidate saliency map is the final detection result map. We mainly made three contributions: First, IALM algorithm is utilized as a preprocessing step, and we have verified it is indispensable in eliminating most backgrounds with self-correlation property. Second, an elaborate scale division strategy is proposed to obtain multi-scale saliency maps that match the feature of infrared small targets precisely. Third, the gray value difference and the maximum value of local information entropy are defined and used as the judgment criteria for optimal scale selection. Extensive experimental results demonstrate that the proposed method outperforms state-of-the-art techniques, especially on infrared images with thick clouds and high-brightness buildings.

Uneven Illumination Surface Defects Inspection Based on Saliency Detection and Intrinsic Image Decomposition

Surface defect detection based on computer vision remains a challenging task due to the uneven illumination, low contrast and miscellaneous patterns of defects. Current methods usually present undesirable detection accuracy and lack adaptability for the various scenes. In the paper, the novel uneven illumination surface defects inspection (UISDI) method is proposed to address these issues. First, the multi-scale saliency detection (MSSD) method is proposed to construct a coarse defect map and obtain the corresponding background regions. Second, a novel background similarity prior-based intrinsic image decomposition model (BSIID) is applied to divide the defect image into a non-defective shading layer and a defective reflectance layer. An accelerated optimization solution is proposed to solve the minimization problem of the intrinsic image decomposition model. Last, the enhanced defect image is obtained by filtering the reflectance image and is then utilized to accurately segment the defect region from the coarse defect map. The experiments conducted using four real-world defect datasets demonstrate that the proposed method outperforms state-of-the-art methods.

Image compression method based on multi-scale saliency region detection

Image compression method based on multi-scale saliency region detection

A novel biologically-inspired target detection method based on saliency analysis for synthetic aperture radar (SAR) imagery

Saliency Object Detection (SOD) models driven by the biologically-inspired Focus of Attention (FOA) mechanism can result in highly accurate saliency maps. However, their application in high-resolution Synthetic Aperture Radar (SAR) images entails a number of intractable problems due to complex backgrounds. In this paper, we propose a novel hierarchical self-diffusion saliency (HSDS) method for detecting vehicle targets in large scale SAR images. To reduce the influence of cluttered returns on saliency analysis, we learn a weight vector from the training set to capture optimal initial saliency of the superpixels during saliency diffusion. By accounting for the multiple sizes of background objects, the saliency analysis is implemented in multi-scale space, and a saliency fusion strategy employed to integrate the multi-scale saliency maps. Simulation experiments demonstrate that our proposed method can produce a more accurate and stable detection performance, with decreased false alarms, compared to benchmark approaches.

A novel deep network and aggregation model for saliency detection

Recent deep learning-based methods for saliency detection have proved the effectiveness of integrating features with different scales. They usually design various complex architectures of network, e.g., multiple networks, to explore the multi-scale information of images, which is expensive in computation and memory. Feature maps produced with different subsampling convolutional layers have different spatial resolutions; therefore, they can be used as the multi-scale features to reduce the costs. In this paper, by exploiting the in-network feature hierarchy of convolutional networks, we propose a novel multi-scale network for saliency detection (MSNSD) consisting of three modules, i.e., bottom-up feature extraction, top-down feature connection and multi-scale saliency prediction. Moreover, to further boost the performance of MSNSD, an input image-aware saliency aggregation method is proposed based on the ridge regression, which combines MSNSD with some well-performed handcrafted shallow models. Extensive experiments on several benchmarks show that the proposed MSNSD outperforms the state-of-the-art saliency methods with less computational and memory complexity. Meanwhile, our aggregation method for saliency detection is effective and efficient to combine deep and shallow models and make them complementary to each other.

Multiscale saliency detection method for ship targets in synthetic aperture radar images

Ship detection is an important direction of synthetic aperture radar (SAR) image application in maritime surveillance. A multi-scale optimisation threshold saliency detection is developed in this letter, which is used to detect ships targets of SAR images. The SAR image is first decomposed into a pyramid image sequence. Then the saliency detection is performed using the spectral residual method for each layer in the sequence, and the salient sub-graph that contains ship targets is obtained. Finally, each sub-graph is fused and the optimisation threshold segmentation method that applies to the saliency map gives the final detection result. Experimental results show that the proposed approach has low complexity and high detection accuracy and greatly reduces the dependence on prior knowledge.

Video Saliency Prediction using Spatiotemporal Residual Attentive Networks.

This paper proposes a novel residual attentive learning network architecture for predicting dynamic eye-fixation maps. The proposed model emphasizes two essential issues, i.e, effective spatiotemporal feature integration and multi-scale saliency learning. For the first problem, appearance and motion streams are tightly coupled via dense residual cross connections, which integrate appearance information with multi-layer, comprehensive motion features in a residual and dense way. Beyond traditional two-stream models learning appearance and motion features separately, such design allows early, multi-path information exchange between different domains, leading to a unified and powerful spatiotemporal learning architecture. For the second one, we propose a composite attention mechanism that learns multi-scale local attentions and global attention priors end-to-end. It is used for enhancing the fused spatiotemporal features via emphasizing important features in multi-scales. A lightweight convolutional Gated Recurrent Unit (convGRU), which is flexible for small training data situation, is used for long-term temporal characteristics modeling. Extensive experiments over four benchmark datasets clearly demonstrate the advantage of the proposed video saliency model over other competitors and the effectiveness of each component of our network. Our code and all the results will be available at https://github.com/ashleylqx/STRA-Net.

Ship detection and tracking method for satellite video based on multiscale saliency and surrounding contrast analysis

In port surveillance, monitoring based on satellite video is a valuable supplement to a ground monitoring system because of its wide monitoring range. Therefore, automatic ship detection and tracking based on satellite video is an important research field. However, because of the small size of ships without texture and the interference of sea noise, it is also a challenging subject. An approach of automatic detection and tracking moving ships of different sizes using satellite video is presented. First, motion compensation between two frames is realized. Then, saliency maps of multiscale differential image are combined to create dynamic multiscale saliency map (DMSM), which is more suitable for the detection of ships of different sizes. Third, candidate motion regions are segmented from DMSM, and moving ships can be detected after the false alarms are removed based on the surrounding contrast. Fourth, important elements such as centroid distance, area ratio, and histogram distance from moving ships are used to perform ship matching. Finally, ship association and tracking are realized by using the intermediate frame in every three adjacent frames. Experimental results on satellite sequences indicate that our method can effectively detect and track ships and obtain the target track, which is superior in terms of the defined recall and precision compared with other classical target tracking methods.

Unsupervised change detection method based on saliency analysis and convolutional neural network

Due to great advantages in deep features representation and classification for image data, deep learning is becoming increasingly popular for change detection (CD) in the remote-sensing community. An unsupervised CD method is proposed by combining deep features representation, saliency detection, and convolutional neural network (CNN). First, bitemporal images are fed into the pretrained CNN model for deep features extraction and difference image generation. Second, multiscale saliency detection is adopted to implement the uncertainty analysis for the difference image, where image pixels can be categorized into three classes: changed, unchanged, and uncertain. Then, a flexible CNN model is constructed and trained using the interested changed and unchanged pixels, and the change type of the uncertain pixels can be determined by the CNN model. Finally, object-based refinement and multiscale fusion strategies are utilized to generate the final change map. The effectiveness and reliability of our CD method are verified on three very high-resolution datasets, and the experimental results show that our proposed approach outperforms the other state-of-the-art CD methods in terms of five quantitative metrics.

Multi-level and multi-scale deep saliency network for salient object detection

Traditional saliency model usually utilize handcrafted image features and various prior knowledge to pop out salient regions from complex surroundings. In this paper, we propose a novel FCN-like deep convolutional neural network for pixel-wise salient object detection. Our deep network automatically learns multi-level feature from different convolutional layers of a pre-trained convolutional neural network. Moreover, deeper side outputs are connected to the shallower ones, which provides a better feature representation and helps shallow side outputs to accurately locate salient regions. In addition, we adopt a weighted-fusion module to combine different side outputs for utilizing multi-scale and multi-level features. Finally, a fully connected CRF model can be optimally incorporated to improve spatial coherence and contour localization in the fused saliency map. Both qualitative and quantitative evaluations on four publicly available datasets demonstrate the robustness and efficiency of our proposed approach against 17 state-of-the-art methods.

Image saliency and co-saliency detection by low-rank multiscale fusion

Saliency and co-saliency detection aim to distinguish conspicuous foreground objects from single and multiple images, thus are essential in many multimedia and vision applications. To achieve balanced efficiency and accuracy, most recent successful saliency detectors are based on superpixels. However, saliency detection with single-scale superpixel segmentation may fail in capturing intrinsic salient objects of complex natural scenes with small-scale high-contrast backgrounds. To tackle this problem, we present a simple strategy using multiscale superpixels to jointly detect salient object via low-rank optimisation. Specifically, we first build a multiscale superpixel pyramid and derive the corresponding saliency map by multimodal saliency features and priors at each single scale. Then, we use joint low-rank analysis of multiscale saliency maps to obtain a more reliable and adaptively-fused saliency map, which properly takes all scales saliency into account. We further propose a GMM generative co-saliency prior to enable the above approach to detect co-salient objects from multiple images. Extensive experiments on benchmark datasets validate the effectiveness and superiority of the proposed saliency and co-saliency detector over state-of-the-arts.