Visual Saliency Analysis Research Articles

Pan-Sharpening Based on Joint Visual Saliency Analysis and Parallel Bidirectional Network

In remote sensing (RS) images, the demands for spectral and spatial quality of different regions are different, which means the unified fusion strategy on the whole image is not suitable for pan-sharpening task. Saliency, derived from visual attention mechanism, provides an effective way to satisfy these demands. Inspired by this, we propose a novel pan-sharpening method based on joint visual saliency analysis and parallel bidirectional network (JSPBN). Firstly, considering the complex scenes and uneven distribution of targets in RS images, we develop a Bayesian optimization based joint visual saliency analysis (B-JVSA) method that integrates prior saliency based on global color contrast with likelihood saliency based on joint co-occurrence histogram, which can highlight common salient regions while suppressing individual ones and irrelevant background by exploring the correlation among multiple RS images. Secondly, we construct a parallel bidirectional feature pyramid (PBFP) network to obtain coarse fusion features, fully considering individual characteristics of panchromatic images and multispectral images. Finally, we design a saliency-aware layer (SAL) according to B-JVSA to further refine the fusion effect in salient regions and non-salient regions. With the help of SAL, diverse strategies for certain regions are learned through two independent residual dense networks and thereby generating accurate fusion results. Experimental results show that our proposal performs better than the competing methods in both spatial quality enhancement and spectral fidelity preservation.

Dense Attention Fluid Network for Salient Object Detection in Optical Remote Sensing Images.

Despite the remarkable advances in visual saliency analysis for natural scene images (NSIs), salient object detection (SOD) for optical remote sensing images (RSIs) still remains an open and challenging problem. In this paper, we propose an end-to-end Dense Attention Fluid Network (DAFNet) for SOD in optical RSIs. A Global Context-aware Attention (GCA) module is proposed to adaptively capture long-range semantic context relationships, and is further embedded in a Dense Attention Fluid (DAF) structure that enables shallow attention cues flow into deep layers to guide the generation of high-level feature attention maps. Specifically, the GCA module is composed of two key components, where the global feature aggregation module achieves mutual reinforcement of salient feature embeddings from any two spatial locations, and the cascaded pyramid attention module tackles the scale variation issue by building up a cascaded pyramid framework to progressively refine the attention map in a coarse-to-fine manner. In addition, we construct a new and challenging optical RSI dataset for SOD that contains 2,000 images with pixel-wise saliency annotations, which is currently the largest publicly available benchmark. Extensive experiments demonstrate that our proposed DAFNet significantly outperforms the existing state-of-the-art SOD competitors. https://github.com/rmcong/DAFNet_TIP20.

Detection of Bobbin Yarn Surface Defects by Visual Saliency Analysis

In order to solve the problem of unstable quality of the traditional bobbin yarn surface defect detection, a method for detecting the surface defect of bobbin yarn using visual saliency analysis is proposed. Firstly, the Difference of Gaussian algorithm is used to suppress the texture of the image, and the contrast between the defect area and the background is enhanced. Then, the method of wavelet threshold is applied to filter the noise in the images after the Difference of Gaussian, and the detailed features of the defect area are preserved while the noise interference is eliminated. Finally, the frequency tuned visual saliency algorithm is used to extract and segment the defective regions accurately. After the process about the Difference of Gaussian and wavelet threshold denoising, the original image of the bobbin yarn can make the image highlight the feature information of the defect area on the basis of ensuring elimination of the background texture and noise, and facilitate the detection of the defect area by combining visual saliency analysis. In the detection of the accuracy and integrity of the defect area, the detection method of this paper is better than the control method, and the defect segmentation result is more accurate. The proposed method for detecting the surface defect of the bobbin can not only completely eliminate the influence of the background texture of the bobbin surface, but also accurately and completely detect the surface defect of the bobbin, and the defect details remain intact. In this paper, the effectiveness of the algorithm is further verified by objective tests. The precision and recall rate of the test are 95.15 % and 98.00 %, respectively, which can meet the needs of actual detection.

A novel remote-sensing image fusion method based on hybrid visual saliency analysis

ABSTRACTIn practical applications, different regions in images may practically have different demands for the spatial and spectral resolution. However, most existing methods execute a unified fusion processing of the whole image with no consideration of such diverse demands. In this article, a new fusion method for remote-sensing images based on saliency analysis is proposed for addressing the issue. By introducing the hybrid visual saliency analysis, regions in the panchromatic (Pan) image and multispectral image would be automatically partitioned into two parts: salient and non-salient regions. Then, a subregion fusion strategy is applied to fuse the non-salient and salient regions, respectively. As for salient regions such as residential areas and roads, the adaptive intensity–hue–saturation (adaptive IHS) method is implemented for its merit of effective improvement in spatial quality. For non-salient regions such as farmland, forest, and grassland, the dual-tree complex wavelet transform is used in the process of spatial detail extraction, and the combination coefficients yielded by the adaptive IHS method are integrated to suppress the spectral distortion. Experimental results demonstrate that our proposal provides state-of-the-art performance as well as achieves a better balance between spatial injection and spectral maintenance in different regions.

Learning Optimal Seeds for Ranking Saliency

A variety of methods have been developed for visual saliency analysis, and it is a challenge to detect the most important scene from the input image. In this paper, to improve the shortage that the spatial connectivity of every node in model only via the k-regular graph and the idealistic boundary prior assumption is used in graph-based manifold ranking, we present a new optimal seed method to get saliency map. First, we evaluate the salience value of each region by global contrast-based spatial and color feature. Second, the salience values of the first stage are used to optimize the background and foreground queries (seeds); meanwhile, we tackle boundary cues from hierarchical graph to optimize background seeds. Then, we derive each stage saliency measure by the classical manifold ranking after obtaining optimal seeds. Finally, the final saliency map is obtained by combining the saliency results of two stages. Our algorithm is tested on the five public datasets and compared with nine state-of-the-art methods; the quantitative evaluation indicates that our method is effective and efficient. Our method can handle complex images with different details and can produce more accurate saliency maps than other state-of-the-art approaches.

Attentive Systems: A Survey

Visual saliency analysis detects salient regions/objects that attract human attention in natural scenes. It has attracted intensive research in different fields such as computer vision, computer graphics, and multimedia. While many such computational models exist, the focused study of what and how applications can be beneficial is still lacking. In this article, our ultimate goal is thus to provide a comprehensive review of the applications using saliency cues, the so-called attentive systems. We would like to provide a broad vision about saliency applications and what visual saliency can do. We categorize the vast amount of applications into different areas such as computer vision, computer graphics, and multimedia. Intensively covering 200+ publications we survey (1) key application trends, (2) the role of visual saliency, and (3) the usability of saliency into different tasks.

Accurate saliency detection based on depth feature of 3D images

This paper presents an accurate saliency detection algorithm customized for 3D images which contain abundant depth cue. Firstly, depth feature is calculated based on the sharp regions’ positions within the focal stack. Then, we compute the coarse saliency map by subtracting the background region from the all-focus image according to the depth feature. Finally, we employ the contrast information in the coarse saliency map to obtain the final result. Experiments on light field dataset demonstrate that our approach favorably outperforms five state-of-the-art methods in terms of precision, recall and F-Measure. Moreover, the depth feature is validated to be a valuable complement to existing visual saliency analysis under the circumstance that the background regions are complex or similar to salient object regions.

Airport Detection and Aircraft Recognition Based on Two-Layer Saliency Model in High Spatial Resolution Remote-Sensing Images

Efficient airport detection and aircraft recognition are essential due to the strategic importance of these regions and targets in economic and military construction. In this paper, a novel airport detection and aircraft recognition method that is based on the two-layer visual saliency analysis model and support vector machines (SVMs) is proposed for high-resolution broad-area remote-sensing images. In the first layer saliency (FLS) model, we introduce a spatial-frequency visual saliency analysis algorithm that is based on a CIE Lab color space to reduce the interference of backgrounds and efficiently detect well-defined airport regions in broad-area remote-sensing images. In the second layer saliency model, we propose a saliency analysis strategy that is based on an edge feature preserving wavelet transform and high-frequency wavelet coefficient reconstruction to complete the pre-extraction of aircraft candidates from airport regions that are detected by the FLS and crudely extract as many aircraft candidates as possible for additional classification in detected airport regions. Then, we utilize feature descriptors that are based on a dense SIFT and Hu moment to accurately describe these features of the aircraft candidates. Finally, these object features are inputted to the SVM, and the aircraft are recognized. The experimental results indicate that the proposed method not only reliably and effectively detects targets in high-resolution broad-area remote-sensing images but also produces more robust results in complex scenes.

Evaluation of visual saliency analysis algorithms in noisy images

Most existing visual saliency analysis algorithms assume that the input image is clean and does not have any disturbances. However, this situation is not always the case. In this paper, we provide an extensive evaluation of visual saliency analysis algorithms in noisy images. We analyze the noise immunity of saliency analysis algorithms by evaluating the performances of the algorithms in noisy images with increasing noise scales and by studying the effects of applying different denoising methods before performing saliency analysis. We use 10 state-of-the-art saliency analysis algorithms and 7 typical image denoising methods on 4 eye fixation datasets and 2 salient object detection datasets. Our experiments show that the performances of saliency analysis algorithms decrease with increasing image noise scales in general. An exception is that the nonlinear features (NF) integrated algorithm shows good noise immunity. We also find that image denoising methods can greatly improve the noise immunity of the algorithms. Our results show that the combination of NF and Median denoising method works best on eye fixation datasets and the combination of saliency optimization (SO) and color block-matching and 3D filtering (C-BM3D) method works best on salient object detection datasets. The combination of SO and Average denoising method works best for applications wherein time efficiency is a major concern for both types of datasets.

Region of interest extraction based on multiscale visual saliency analysis for remote sensing images

Region of interest (ROI) extraction is an important component of remote sensing image processing. However, traditional ROI extraction methods are usually prior knowledge-based and depend on classification, segmentation, and a global searching solution, which are time-consuming and computationally complex. We propose a more efficient ROI extraction model for remote sensing images based on multiscale visual saliency analysis (MVS), implemented in the CIE L*a*b* color space, which is similar to visual perception of the human eye. We first extract the intensity, orientation, and color feature of the image using different methods: the visual attention mechanism is used to eliminate the intensity feature using a difference of Gaussian template; the integer wavelet transform is used to extract the orientation feature; and color information content analysis is used to obtain the color feature. Then, a new feature-competition method is proposed that addresses the different contributions of each feature map to calculate the weight of each feature image for combining them into the final saliency map. Qualitative and quantitative experimental results of the MVS model as compared with those of other models show that it is more effective and provides more accurate ROI extraction results with fewer holes inside the ROI.

Spatiochromatic Context Modeling for Color Saliency Analysis.

Visual saliency is one of the most noteworthy perceptual abilities of human vision. Recent progress in cognitive psychology suggests that: 1) visual saliency analysis is mainly completed by the bottom-up mechanism consisting of feedforward low-level processing in primary visual cortex (area V1) and 2) color interacts with spatial cues and is influenced by the neighborhood context, and thus it plays an important role in a visual saliency analysis. From a computational perspective, the most existing saliency modeling approaches exploit multiple independent visual cues, irrespective of their interactions (or are not computed explicitly), and ignore contextual influences induced by neighboring colors. In addition, the use of color is often underestimated in the visual saliency analysis. In this paper, we propose a simple yet effective color saliency model that considers color as the only visual cue and mimics the color processing in V1. Our approach uses region-/boundary-defined color features with spatiochromatic filtering by considering local color-orientation interactions, therefore captures homogeneous color elements, subtle textures within the object and the overall salient object from the color image. To account for color contextual influences, we present a divisive normalization method for chromatic stimuli through the pooling of contrary/complementary color units. We further define a color perceptual metric over the entire scene to produce saliency maps for color regions and color boundaries individually. These maps are finally globally integrated into a one single saliency map. The final saliency map is produced by Gaussian blurring for robustness. We evaluate the proposed method on both synthetic stimuli and several benchmark saliency data sets from the visual saliency analysis to salient object detection. The experimental results demonstrate that the use of color as a unique visual cue achieves competitive results on par with or better than 12 state-of-the-art approaches.

Video accessibility enhancement for hearing-impaired users

There are more than 66 million people suffering from hearing impairment and this disability brings them difficulty in video content understanding due to the loss of audio information. If the scripts are available, captioning technology can help them in a certain degree by synchronously illustrating the scripts during the playing of videos. However, we show that the existing captioning techniques are far from satisfactory in assisting the hearing-impaired audience to enjoy videos. In this article, we introduce a scheme to enhance video accessibility using a Dynamic Captioning approach, which explores a rich set of technologies including face detection and recognition, visual saliency analysis, text-speech alignment, etc. Different from the existing methods that are categorized as static captioning, dynamic captioning puts scripts at suitable positions to help the hearing-impaired audience better recognize the speaking characters. In addition, it progressively highlights the scripts word-by-word via aligning them with the speech signal and illustrates the variation of voice volume. In this way, the special audience can better track the scripts and perceive the moods that are conveyed by the variation of volume. We implemented the technology on 20 video clips and conducted an in-depth study with 60 real hearing-impaired users. The results demonstrated the effectiveness and usefulness of the video accessibility enhancement scheme.

AdOn: toward contextual overlay in-video advertising

This paper presents a contextual video advertising system, called AdOn, which supports intelligent overlay in-video advertising. Unlike most current ad-networks such as Youtube that overlay the ads at fixed locations in the videos (e.g., on the bottom fifth of videos 15 s in), AdOn is able to automatically detect a set of spatio-temporal non-intrusive locations and associate the contextually relevant ads with these locations. The overlay ad locations are obtained on the basis of video structuring, face and text detection, as well as visual saliency analysis, so that the intrusiveness to the users can be minimized. The ads are selected according to content-based multimodal relevance so that the relevance can be maximized. AdOn represents one of the first attempts towards contextual overlay video advertising by leveraging information retrieval and multimedia content analysis techniques. The experiments conducted on a video database with more than 100 video programs and 7,000 ad products indicated that AdOn is superior to existing advertising approaches in terms of ad relevance and user experience.