Foreground Saliency Research Articles

A new framework for improving semantic segmentation in aerial imagery

High spatial resolution (HSR) remote sensing imagery presents a rich tapestry of foreground-background intricacies, rendering semantic segmentation in aerial contexts a formidable and vital undertaking. At its core, this challenge revolves around two pivotal questions: 1) Mitigating Background Interference and Enhancing Foreground Clarity. 2) Accurate Segmentation in Dense Small Object Cluster. Conventional semantic segmentation methods primarily cater to the segmentation of large-scale objects in natural scenes, yet they often falter when confronted with aerial imagery’s characteristic traits such as vast background areas, diminutive foreground objects, and densely clustered targets. In response, we propose a novel semantic segmentation framework tailored to overcome these obstacles. To address the first challenge, we leverage PointFlow modules in tandem with the Foreground-Scene (F-S) module. PointFlow modules act as a barrier against extraneous background information, while the F-S module fosters a symbiotic relationship between the scene and foreground, enhancing clarity. For the second challenge, we adopt a dual-branch structure termed disentangled learning, comprising Foreground Precedence Estimation and Small Object Edge Alignment (SOEA). Our foreground saliency guided loss optimally directs the training process by prioritizing foreground examples and challenging background instances. Extensive experimentation on the iSAID and Vaihingen datasets validates the efficacy of our approach. Not only does our method surpass prevailing generic semantic segmentation techniques, but it also outperforms state-of-the-art remote sensing segmentation methods.

Infrared small target detection using Homogeneity-weighted local patch saliency

Detecting infrared small targets (IRSTs) submerged in the complex back- grounds with heavy clutters is a challenging work of great significance in many fields. Inspired by representation capability of the superpixel, this paper presents a human visual system (HVS) based method called Homogeneity weighted local patch saliency (HWLPS). Firstly, the image is transformed into a superpixel graph and a compound flux field of the thermal intensity and gradient. Next, IRST component saliency and superpixel homogeneity are quantified by the morphing gradient direction diversity (GDD) metric and the inter-patch homogeneity (IPH) successively. Finally, the local contrast aggregation fuses the morphing GDD and the foreground saliency of image patches according to the superpixel graph. Experiments on various real IR image datasets show that, the proposed algorithm outperforms the state-of-the-art methods with robustness and stability against intricate back- grounds.

RSSFormer: Foreground Saliency Enhancement for Remote Sensing Land-Cover Segmentation.

High spatial resolution (HSR) remote sensing images contain complex foreground-background relationships, which makes the remote sensing land cover segmentation a special semantic segmentation task. The main challenges come from the large-scale variation, complex background samples and imbalanced foreground-background distribution. These issues make recent context modeling methods sub-optimal due to the lack of foreground saliency modeling. To handle these problems, we propose a Remote Sensing Segmentation framework (RSSFormer), including Adaptive TransFormer Fusion Module, Detail-aware Attention Layer and Foreground Saliency Guided Loss. Specifically, from the perspective of relation-based foreground saliency modeling, our Adaptive Transformer Fusion Module can adaptively suppress background noise and enhance object saliency when fusing multi-scale features. Then our Detail-aware Attention Layer extracts the detail and foreground-related information via the interplay of spatial attention and channel attention, which further enhances the foreground saliency. From the perspective of optimization-based foreground saliency modeling, our Foreground Saliency Guided Loss can guide the network to focus on hard samples with low foreground saliency responses to achieve balanced optimization. Experimental results on LoveDA datasets, Vaihingen datasets, Potsdam datasets and iSAID datasets validate that our method outperforms existing general semantic segmentation methods and remote sensing segmentation methods, and achieves a good compromise between computational overhead and accuracy. Our code is available at https://github.com/Rongtao-Xu/RepresentationLearning/tree/main/RSSFormer-TIP2023.

UAV Image Haze Removal Based on Saliency- Guided Parallel Learning Mechanism

Current haze removal methods for unmanned aerial vehicle (UAV) images are mostly based on natural image dehazing methods, which ignore the particular imaging mechanism of UAVs. They often fail to restore regions with rich spectral and textural information. In this letter, we propose a saliency-guided parallel learning mechanism for UAV image haze removal. First, we design a saliency-guided parallel dehazing module with two parallel paths. The residual feature extraction path obtains deep-level features to realize global dehazing effectively. The key feature enhancement path, which comprises saliency dense blocks, realizes local textural preservation and spectral restoration. Second, a sporadic foreground saliency detection method is proposed for UAV images with sporadic objects. The saliency map guides the learning of significant spectral and textural information in hazy images. Finally, a multiscale reconstruction module is introduced to more accurately estimate small-scale textural details and large-scale spectral information. Experimental results show that the proposed method has better detail performance and visual effects than state-of-the-art methods.

Subjective low-light image enhancement based on a foreground saliency map model

Subjective low-light image enhancement based on a foreground saliency map model

Distilling Localization for Self-Supervised Representation Learning

Recent progress in contrastive learning has revolutionized unsupervised representation learning. Concretely, multiple views (augmentations) from the same image are encouraged to map to close embeddings, while views from different images are pulled apart.In this paper, through visualizing and diagnosing classification errors, we observe that current contrastive models are ineffective at localizing the foreground object, limiting their ability to extract discriminative high-level features. This is due to the fact that view generation process considers pixels in an image uniformly.To address this problem, we propose a data-driven approach for learning invariance to backgrounds. It first estimates foreground saliency in images and then creates augmentations by copy-and-pasting the foreground onto a variety of back-grounds. The learning still follows an instance discrimination approach, so that the representation is trained to disregard background content and focus on the foreground. We study a variety of saliency estimation methods, and find that most methods lead to improvements for contrastive learning. With this approach, significant performance is achieved for self-supervised learning on ImageNet classification, and also for object detection on PASCAL VOC and MSCOCO.

Uncooperative Spacecraft Pose Estimation Using Monocular Monochromatic Images

Imaging cameras are cost-effective sensors for spacecraft navigation. Image-driven techniques to extract the target spacecraft from its background are efficient and do not require pretraining. In this paper, we introduce several image-driven foreground extraction methods, including combining the difference of Gaussian-based scene detection and graph manifold ranking-based foreground saliency generation. We successfully apply our foreground extraction method on infrared images from the STS-135 flight mission captured by the space shuttle’s Triangulation and LIDAR Automated Rendezvous and Docking System (TriDAR) thermal camera. Our saliency approach demonstrates state-of-the-art performance and provides an order of magnitude reduction in processing speed from the traditional methods. Furthermore, we develop a new uncooperative spacecraft pose estimation method by combining our foreground extraction technique with the level-set region-based pose estimation with novel initialization and gradient descent enhancements. Our method is validated using synthetically generated Envisat, Radarsat model, and International Space Station motion sequences. The proposed process is also validated with real rendezvous flight images of the International Space Station.

Explore double-opponency and skin color for saliency detection

Recent advances in salient detection have exploited the foreground or background information to assist other saliency cues such as contrast to achieve state-of-the-art results. However the problem remains challenging. For example, human skin color is easily overlooked during saliency detection. Simulating the human visual mechanism to improve the current algorithm, we propose a saliency model based on the color-opponent mechanisms of a certain type of color-sensitive double-opponent (DO) cells in the primary visual cortex (V1) of human visual system. Firstly, DO cells with concentric receptive fields (RFs) can detect region contrast for yielding foreground saliency map. Our approach is intuitive and easy to interpret, and it allows fast implementation. Then, skin saliency map is built in a way that is different from high-level factors such as face detection, combining skin region and spatial Euclidean distance weight in the RGB space, and the significant skin features can be obtained effectively. Finally, a linear fusion strategy is proposed to integrate different saliency maps. Experimental results with three well-known benchmark databases demonstrate that the proposed method can achieve competitive performance when compared to state-of-the-art methods. Saliency regions contain important skin features compared to other traditional methods.

Thermal Infrared Small Ship Detection in Sea Clutter Based on Morphological Reconstruction and Multi-Feature Analysis

The existing thermal infrared (TIR) ship detection methods may suffer serious performance degradation in the situation of heavy sea clutter. To cope with this problem, a novel ship detection method based on morphological reconstruction and multi-feature analysis is proposed in this paper. Firstly, the TIR image is processed by opening- or closing-based gray-level morphological reconstruction (GMR) to smooth intricate background clutter while maintaining the intensity, shape, and contour features of ship target. Then, considering the intensity and contrast features, the fused saliency detection strategy including intensity foreground saliency map (IFSM) and brightness contrast saliency map (BCSM) is presented to highlight potential ship targets and suppress sea clutter. After that, an effective contour descriptor namely average eigenvalue measure of structure tensor (STAEM) is designed to characterize candidate ship targets, and the statistical shape knowledge is introduced to identify true ship targets from residual non-ship targets. Finally, the dual method is adopted to simultaneously detect both bright and dark ship targets in TIR image. Extensive experiments show that the proposed method outperforms the compared state-of-the-art methods, especially for infrared images with intricate sea clutter. Moreover, the proposed method can work stably for ship target with unknown brightness, variable quantities, sizes, and shapes.

Salient object detection using background subtraction, Gabor filters, objectness and minimum directional backgroundness

Salient object detection is the process of identifying essential objects in an image. This paper solves this problem using background subtraction, Gabor filters, minimum directional backgroundness, and objectness. The first step is to calculate a backgroundness score for each region by calculating the difference between the feature vector of image boundary and image regions. This backgroundness map is then used for calculating the minimum directional background difference. The image is segmented using Gabor filters, and then the objectness criterion is used to choose the segment containing the salient object. The normalized foreground saliency map is then used to refine the selected segment.Further enhancement of this intermediate output is done using morphological operations, and boundary correction is done using the method of lazy snapping. The algorithm is tested on eight publicly available datasets and is compared against five algorithms. The performance is evaluated by PR-curve, F-Measure curve, and Mean Absolute Error.

Extended Locality-Constrained Linear Self-Coding for Saliency Detection

In complex scenes, foreground saliency can hardly be detected completely, which may further result in the ambiguous cues of objects for other computer vision tasks. In this letter, an extended locality-constrained linear self-coding (eLLsC) scheme is proposed to assist to solve the saliency detection problem under the complex scenes. The locality of both spatial relation and feature distance is preserved in eLLsC, thus making the transformed code involved in the manifold ranking to prompt the generation of the saliency map with more complete foreground and clearer boundary. Experimental results on three saliency detection benchmarks demonstrate the effectiveness of the proposed hybrid method.

Ranking Saliency.

Most existing bottom-up algorithms measure the foreground saliency of a pixel or region based on its contrast within a local context or the entire image, whereas a few methods focus on segmenting out background regions and thereby salient objects. Instead of only considering the contrast between salient objects and their surrounding regions, we consider both foreground and background cues in this work. We rank the similarity of image elements with foreground or background cues via graph-based manifold ranking. The saliency of image elements is defined based on their relevances to the given seeds or queries. We represent an image as a multi-scale graph with fine superpixels and coarse regions as nodes. These nodes are ranked based on the similarity to background and foreground queries using affinity matrices. Saliency detection is carried out in a cascade scheme to extract background regions and foreground salient objects efficiently. Experimental results demonstrate the proposed method performs well against the state-of-the-art methods in terms of accuracy and speed. We also propose a new benchmark dataset containing 5,168 images for large-scale performance evaluation of saliency detection methods.

Image structure-based saliency detection

We consider image intrinsic structure in two aspects: foreground saliency computation and background saliency calculation. On the one hand, a segmented image is represented as a weighted undirected graph, and we calculate the foreground saliency by choosing some superpixels as ranking queries, where we apply a robust background measure to select those for accuracy. On the other hand, we compute the posterior probabilities to measure the background saliency and in terms of the probability, we construct a probability tree via multimerging on superpixels, and then apply the optimization strategy to the background saliency. We evaluate our proposed algorithm on two benchmark datasets and our algorithm yields the competitive results when compared with nine state-of-the-art algorithms in terms of five evaluation metrics.

Visual comfort prediction for stereoscopic image using stereoscopic visual saliency

Perceptually salient regions of stereoscopic images significantly affect visual comfort (VC). In this paper, we propose a new objective approach for predicting VC of stereoscopic images according to visual saliency. The proposed approach includes two stages. The first stage involves the extraction of foreground saliency and depth contrast from a disparity map to generate a depth saliency map, which in turn is combined with 2D saliency to obtain a stereoscopic visual saliency map. The second stage involves the extraction of saliency-weighted VC features, and feeding them into a prediction metric to produce VC scores of the stereoscopic images. We demonstrate the effectiveness of the proposed approach compared with the conventional prediction methods on the IVY Lab database, with performance gain ranging from 0.016 to 0.198 in terms of correlation coefficients.

Saliency Detection for Stereoscopic Images Based on Depth Confidence Analysis and Multiple Cues Fusion

Stereoscopic perception is an important part of human visual system that allows the brain to perceive depth. However, depth information has not been well explored in existing saliency detection models. In this letter, a novel saliency detection method for stereoscopic images is proposed. Firstly, we propose a measure to evaluate the reliability of depth map, and use it to reduce the influence of poor depth map on saliency detection. Then, the input image is represented as a graph, and the depth information is introduced into graph construction. After that, a new definition of compactness using color and depth cues is put forward to compute the compactness saliency map. In order to compensate the detection errors of compactness saliency when the salient regions have similar appearances with background, foreground saliency map is calculated based on depth-refined foreground seeds selection mechanism and multiple cues contrast. Finally, these two saliency maps are integrated into a final saliency map through weighted-sum method according to their importance. Experiments on two publicly available stereo datasets demonstrate that the proposed method performs better than other 10 state-of-the-art approaches.

A novel background-weighted histogram scheme based on foreground saliency for mean-shift tracking

Effective appearance models are one critical factor for robust object tracking. In this paper, we introduce foreground feature saliency concept into the background modelling, and put forward a novel foreground saliency-based background-weighted histogram scheme (FSBWH) for target representation and tracking, which exploits salient features from both foreground and background. We think that background and foreground salient features are both crucial for target representation and tracking. Experimental results show that the proposed FSBWH scheme can improve the robustness and performance of tracker significantly especially in complex occlusions and similar background scenes.

Salient object detection via spectral matting

A number of pro-superpixel based saliency models have recently been proposed, which segment the image into small perceptually homogeneous regions before saliency computation. Such approaches ignore important object properties, resulting in inappropriate object annotations and considerably different saliency assignment to the various regions of an object. Although previous techniques employ multi-scale saliency maps in an attempt to rectify this problem, it becomes difficult to retain the characteristics of proto-objects after the first stage of processing. We introduce matting components based saliency to address the problems of inappropriate object annotations and inappropriate saliency assignment to object regions. The matting components account for proto-object properties by employing object aware spectral segmentation. To complement the matting component based saliency, we also employ the smallest eigenvectors of a matting Laplacian matrix. Color spatial distribution features are employed to capture global relationships at the pixel-level and assist the process of matting components based saliency computation. A novel joint optimization framework is introduced to fuse the features and learn important associated parameters. The contributions of the proposed approach are two-fold. The first contribution is the introduction of proto-objects aware spectral segmentation to obtain an accurate foreground saliency. The second contribution is the joint optimization of important parameters in conjunction with learning feature importance. In contrast to superpixel based approaches, the proposed model is able to completely annotate salient objects and assign similar saliency to various regions of the salient object. Moreover, the proposed approach shows robust and efficient performance across five challenging benchmark datasets when compared with 10 recently proposed state-of-the-art saliency detection models.