Non-salient Regions Research Articles

Rethinking Self-Supervised Semantic Segmentation: Achieving End-to-End Segmentation.

The challenge of semantic segmentation with scarce pixel-level annotations has induced many self-supervised works, however most of which essentially train an image encoder or a segmentation head that produces finer dense representations, and when performing segmentation inference they need to resort to supervised linear classifiers or traditional clustering. Segmentation by dataset-level clustering not only deviates the real-time and end-to-end inference practice, but also escalates the problem from segmenting per image to clustering all pixels at once, which results in downgraded performance. To remedy this issue, we propose a novel self-supervised semantic segmentation training and inferring paradigm where inferring is performed in an end-to-end manner. Specifically, based on our observations in probing dense representation by image-level self-supervised ViT, i.e. semantic inconsistency between patches and poor semantic quality in non-salient regions, we propose prototype-image alignment and global-local alignment with attention map constraint to train a tailored Transformer Decoder with learnable prototypes and utilize adaptive prototypes for segmentation inference per image. Extensive experiments under fully unsupervised semantic segmentation settings demonstrate the superior performance and the generalizability of our proposed method. The code is available at: https://github.com/yliu1229/AlignSeg.

Content-Aware Rectified Activation for Zero-Shot Fine-Grained Image Retrieval.

Fine-grained image retrieval mainly focuses on learning salient features from the seen subcategories as discriminative embedding while neglecting the problems behind zero-shot settings. We argue that retrieving fine-grained objects from unseen subcategories may rely on more diverse clues, which are easily restrained by the salient features learnt from seen subcategories. To address this issue, we propose a novel Content-aware Rectified Activation model, which enables this model to suppress the activation on salient regions while preserving their discrimination, and spread activation to adjacent non-salient regions, thus mining more diverse discriminative features for retrieving unseen subcategories. Specifically, we construct a content-aware rectified prototype (CARP) by perceiving semantics of salient regions. CARP acts as a channel-wise non-destructive activation upper bound and can be selectively used to suppress salient regions for obtaining the rectified features. Moreover, two regularizations are proposed: 1) a semantic coherency constraint that imposes a restriction on semantic coherency of CARP and salient regions, aiming at propagating the discriminative ability of salient regions to CARP, 2) a feature-navigated constraint to further guide the model to adaptively balance the discrimination power of rectified features and the suppression power of salient features. Experimental results on fine-grained and product retrieval benchmarks demonstrate that our method consistently outperforms the state-of-the-art methods.

WBNet: Weakly-supervised salient object detection via scribble and pseudo-background priors

Weakly supervised salient object detection (WSOD) methods endeavor to boost sparse labels to get more salient cues in various ways. Among them, an effective approach is using pseudo labels from multiple unsupervised self-learning methods, but inaccurate and inconsistent pseudo labels could ultimately lead to detection performance degradation. To tackle this problem, we develop a new multi-source WSOD framework, WBNet, that can effectively utilize pseudo-background (non-salient region) labels combined with scribble labels to obtain more accurate salient features. We first design a comprehensive salient pseudo-mask generator from multiple self-learning features. Then, we pioneer the exploration of generating salient pseudo-labels via point-prompted and box-prompted Segment Anything Models (SAM). Then, WBNet leverages a pixel-level Feature Aggregation Module (FAM), a mask-level Transformer-decoder (TFD), and an auxiliary Boundary Prediction Module (EPM) with a hybrid loss function to handle complex saliency detection tasks. Comprehensively evaluated with state-of-the-art methods on five widely used datasets, the proposed method significantly improves saliency detection performance. The code and results are publicly available at https://github.com/yiwangtz/WBNet.

Video Generalized Semantic Segmentation via Non-Salient Feature Reasoning and Consistency

Video semantic segmentation is beneficial for dynamic scene processing in real-world environments, and achieves superior performance on independent and identically distributed data. However, it suffers from performance degradation in environments with various domain styles, which is known as the distribution shift problem. Although some previous studies on image generalized semantic segmentation considered the distribution shift problem, temporal-frame information could not be used to obtain more accurate prediction. Thus, in this study, we explore a new task, known as the video generalized semantic segmentation (VGSS) task, which establishes a connection between continuous frames and domain generalization. We propose a novel method named Non-Salient Feature Reasoning and Consistency (NSFRC) for this task. Specifically, we first define the class-wise non-salient feature, which describes the features of the class-wise non-salient region that carry more generalized information. We then propose a class-wise non-salient feature reasoning strategy to select and enhance generalized channels adaptively. This strategy adopts a new form to use domain-invariant features by treating the domain-invariant features as prior information to assist domain-invariant model learning. Finally, we propose a non-salient centroid alignment loss to alleviate the temporally inconsistent and negative transfer problems in the VGSS task. We also extend our video-based framework to the image generalized semantic segmentation (IGSS) task. Experiments demonstrate that our NSFRC framework yields significant improvements in both the VGSS and IGSS tasks. To explain the idea of this research in a clear and attractive way, we provide the visual abstract shown in Fig. 1.

LA-Net: layer attention network for 3D-to-2D retinal vessel segmentation in OCTA images

Objective. Retinal vessel segmentation from optical coherence tomography angiography (OCTA) volumes is significant in analyzing blood supply structures and the diagnosing ophthalmic diseases. However, accurate retinal vessel segmentation in 3D OCTA remains challenging due to the interference of choroidal blood flow signals and the variations in retinal vessel structure. Approach. This paper proposes a layer attention network (LA-Net) for 3D-to-2D retinal vessel segmentation. The network comprises a 3D projection path and a 2D segmentation path. The key component in the 3D path is the proposed multi-scale layer attention module, which effectively learns the layer features of OCT and OCTA to attend to the retinal vessel layer while suppressing the choroidal vessel layer. This module also efficiently captures 3D multi-scale information for improved semantic understanding during projection. In the 2D path, a reverse boundary attention module is introduced to explore and preserve boundary and shape features of retinal vessels by focusing on non-salient regions in deep features. Main results. Experimental results in two subsets of the OCTA-500 dataset showed that our method achieves advanced segmentation performance with Dice similarity coefficients of 93.04% and 89.74%, respectively. Significance. The proposed network provides reliable 3D-to-2D segmentation of retinal vessels, with potential for application in various segmentation tasks that involve projecting the input image. Implementation code: https://github.com/y8421036/LA-Net.

Joint learning of foreground, background and edge for salient object detection

Although significant progress has been made in saliency detection, predicting saliency remains challenging when the scene is complex, especially when salient and non-salient regions are similar or salient objects have intricate contours. Previous advanced methods rarely explored learning in the background of images. In fact, background and foreground of an image contain complementary information. In this work, we propose to decompose the saliency detection task into three subtasks: foreground awareness, background suppression, and edge refinement. More specifically, our decoder is comprised of three branches: a foreground awareness branch, a background suppression branch, and an edge refinement branch. Each branch aims to learn specific features for predicting its corresponding map. Meanwhile, we design a regional focus loss function with controllable modulating factors to supervise the learning of each branch at the training stage. Moreover, we build an attention guided feature fusion module to adaptively fuse multi-scale features and a global information capture module to locate salient objects. Experiments on five benchmark datasets demonstrate that our approach is superior to the state-of-the-art methods.

LR Aerial Photo Categorization by Cross-Resolution Perceptual Knowledge Propagation.

There are hundreds of high-and low-altitude earth observation satellites that asynchronously capture massive-scale aerial photographs every day. Generally, high-altitude satellites take low-resolution (LR) aerial pictures, each covering a considerably large area. In contrast, low-altitude satellites capture high-resolution (HR) aerial photos, each depicting a relatively small area. Accurately discovering the semantics of LR aerial photos is an indispensable technique in computer vision. Nevertheless, it is also a challenging task due to: 1) the difficulty to characterize human hierarchical visual perception and 2) the intolerable human resources to label sufficient training data. To handle these problems, a novel cross-resolution perceptual knowledge propagation (CPKP) framework is proposed, focusing on adapting the visual perceptual experiences deeply learned from HR aerial photos to categorize LR ones. Specifically, by mimicking the human vision system, a novel low-rank model is designed to decompose each LR aerial photo into multiple visually/semantically salient foreground regions coupled with the background nonsalient regions. This model can: 1) produce a gaze-shifting path (GSP) simulating human gaze behavior and 2) engineer the deep feature for each GSP. Afterward, a kernel-induced feature selection (FS) algorithm is formulated to obtain a succinct set of deep GSP features discriminative across LR and HR aerial photos. Based on the selected features, the labels from LR and HR aerial photos are collaboratively utilized to train a linear classifier for categorizing LR ones. It is worth emphasizing that, such a CPKP mechanism can effectively optimize the linear classifier training, as labels of HR aerial photos are acquired more conveniently in practice. Comprehensive visualization results and comparative study have validated the superiority of our approach.

Image Watermarking Framework using Histogram Equalization and Visual Saliency

This paper proposes a digital image watermarking strategy using histogram equalization and visual Saliency followed by LSB (Least Significant Bit) replacement for better imperceptibility with hiding capacity. With this technique, a saliency map determines lesser-observable parts of the original image and gradually implants with increasing amounts of information based on histogram equalization information. The output from saliency is the perceptible areas within an image, which is the most notable position from the perspective of vision; as a result, any changes made other than those areas will be less noticeable to viewers. Implementing the histogram method helps identify the areas where we can hide our secret information within that image. Using the LSB replacement technique, we adaptively insert our confidential data into the original image. Here, we use the saliency map to find out the non-salient region or less perceptible region to improve the imperceptibility, and the histogram equalization technique is used to maximize the hiding capacity within those less perceptible regions. So that we can improve the imperceptibility as well as the hiding capacity.

Gaze-guided CT image retargeting by multi-attribute binary hashing

Computed tomography (CT) imaging is pervasively utilized for detecting tumors and internal body injuries. CT image retargeting means to horizontally/vertically shrink the semantically non-salient regions (e.g, the normal organs) while preserving the salient ones (e.g., the diseased organs) inside a CT image, as exemplified in Fig. 1. In practice, retargeting can substantially facilitate CT image displaying, which can benefit the subsequent medical treatment. In this work, we propose a bio-inspired CT image retargeting pipeline by mimicking human gaze behavior. More specifically, for each CT image, we extract the gaze shifting path (GSP) to capture human gaze distribution during the visual perception toward each CT image. Afterward, a multi-attribute binary hashing (MABH) is formulated to exploit the semantics of these GSPs. Thereby, each graphlet can be converted into the binary hash codes. Finally, the hash codes corresponding to GSP from each CT image are quantized into a feature vector, which is leveraged to learn a Gaussian mixture model (GMM) that guides CT image shrinking. In the experiments, to evaluate how gaze allocation influencing CT image retargeting, a user study is designed to compare the GSPs produced by normal observers and Alzheimer’s patients respectively. Besides, a comparative study has verified the superiority of our method.

Dual-aware Domain Mining and Cross-aware Supervision for Weakly-supervised Semantic Segmentation

Weakly Supervised Semantic Segmentation with image-level annotation uses localization maps from the classifier to generate pseudo labels. However, such localization maps focus only on sparse salient object regions, it is difficult to generate high-quality segmentation labels, which deviates from the requirement of semantic segmentation. To address this issue, we propose a dual-aware domain mining and cross-aware supervision (DDMCAS) method for weakly-supervised semantic segmentation. Specifically, we propose a dual-aware domain mining (DDM) module consisting of graph-based global reasoning unit and salient-region extension controller, which produces dense localization maps by exploring object features in salient regions and adjacent non-salient regions simultaneously. In order to further bridge the gap between salient regions and adjacent non-salient regions to generate more refined localization maps, we propose a cross-aware supervision (CAS) strategy to recover missing parts of the target objects and enhance weak attention in adjacent non-salient regions, leading to pseudo labels of higher quality for training the segmentation network. Based on the generated pseudo-labels, extensive experiments on PASCAL VOC 2012 dataset demonstrate that our method outperforms state-of-the-art methods using image-level labels for weakly supervised semantic segmentation.

RA-DENet: Reverse Attention and Distractions Elimination Network for polyp segmentation

To address the problems of polyps of different shapes, sizes, and colors, low-contrast polyps, various noise distractions, and blurred edges on colonoscopy, we propose the Reverse Attention and Distraction Elimination Network, which includes Improved Reverse Attention, Distraction Elimination, and Feature Enhancement. First, we input the images in the polyp image set, and use the five levels polyp features and the global polyp feature extracted from the Res2Net-based backbone as the input of the Improved Reverse Attention to obtain augmented representations of salient and non-salient regions to capture the different shapes of polyp and distinguish low-contrast polyps from background. Then, the augmented representations of salient and non-salient areas are fed into the Distraction Elimination to obtain the refined polyp feature without false positive and false negative distractions for eliminating noises. Finally, the extracted low-level polyp feature is used as the input of the Feature Enhancement to obtain the edge feature for supplementing missing edge information of polyp. The polyp segmentation result is output by connecting the edge feature with the refined polyp feature. The proposed method is evaluated on five polyp datasets and compared with the current polyp segmentation models. Our model improves the mDice to 0.760 on the most challenge dataset (ETIS).

Progressive Dual-Attention Residual Network for Salient Object Detection

Due to the rapid development of deep learning, the performance of salient object detection has been constantly refreshed. Nevertheless, it is still challenging for existing methods to distinguish the location of salient objects and retain fine structural details. In this paper, a novel progressive dual-attention residual network (PDRNet) is proposed to exploit two complementary attention maps to guide residual learning, thus progressively refining prediction in a coarse-to-fine manner. We design a dual-attention residual module (DRM) to achieve residual refinement with the help of the dual attention (DA) scheme. Specifically, an attention map and its corresponding reverse attention map are used to make the network be aware of learning residual details from the perspective of the salient and non-salient regions, thus utilizing their complementarity to correct the mistakes of object parts and boundary details. Besides, a hierarchical feature screening module (HFSM) is designed to capture more powerful global contextual knowledge for locating salient objects. It establishes cross-scale skip connections among multi-scale features and utilizes the intra-channel dependency of these scales to enhance information interaction and feature representation. Extensive experiments have proved that our proposed PDRNet performs favorably against 18 state-of-the-art competitors on five benchmark datasets, demonstrating the effectiveness and superiority of our method.

Non-salient region erasure for time series augmentation

Non-salient region erasure for time series augmentation

Pan-Sharpening Based on Joint Visual Saliency Analysis and Parallel Bidirectional Network

In remote sensing (RS) images, the demands for spectral and spatial quality of different regions are different, which means the unified fusion strategy on the whole image is not suitable for pan-sharpening task. Saliency, derived from visual attention mechanism, provides an effective way to satisfy these demands. Inspired by this, we propose a novel pan-sharpening method based on joint visual saliency analysis and parallel bidirectional network (JSPBN). Firstly, considering the complex scenes and uneven distribution of targets in RS images, we develop a Bayesian optimization based joint visual saliency analysis (B-JVSA) method that integrates prior saliency based on global color contrast with likelihood saliency based on joint co-occurrence histogram, which can highlight common salient regions while suppressing individual ones and irrelevant background by exploring the correlation among multiple RS images. Secondly, we construct a parallel bidirectional feature pyramid (PBFP) network to obtain coarse fusion features, fully considering individual characteristics of panchromatic images and multispectral images. Finally, we design a saliency-aware layer (SAL) according to B-JVSA to further refine the fusion effect in salient regions and non-salient regions. With the help of SAL, diverse strategies for certain regions are learned through two independent residual dense networks and thereby generating accurate fusion results. Experimental results show that our proposal performs better than the competing methods in both spatial quality enhancement and spectral fidelity preservation.

Content-Aware Point Cloud Simplification of Natural Scenes

Laser scanning technology is becoming ubiquitous in studies involving 3D characterizations of natural scenes, e.g., for geomorphological or archaeological interpretations. Setting the point density in such scanning campaigns is usually dictated by the objects of interest within the site yet is applied to the entire scene. Such campaigns result in large data volumes, which are difficult to analyse and where the objects of interest may be hidden in the redundant data. To reduce these excessive volumes, existing simplification strategies maintain smoothness, preserve discontinuities in the point cloud, but disregard the need to preserve detail at the regions of interest (ROI). To address that, this paper proposes a new, context-aware, subsampling approach that retains the high resolution of objects of interest while reducing the data load of less important regions. To do so, we identify the ROI by means of visual saliency measures and reduce the data volume only at the non-salient regions. To facilitate progressive subsampling the reduction is based on a hierarchical data structure that is surficial in nature. This way, the retained representative points describe the underlying surface rather than an interpolation of it. We demonstrate our proposed model on datasets originating from different scanners that feature a variety of scenes. We compare our results to three common simplification approaches. Our results show a reduced point cloud that is similar to the original and allows analysis of ROI at the required point resolution, regardless of the level of simplification.

A Generative Adversarial Network with Dual Discriminators for Infrared and Visible Image Fusion Based on Saliency Detection

Infrared and visible image fusion needs to preserve both the salient target of the infrared image and the texture details of the visible image. Therefore, an infrared and visible image fusion method based on saliency detection is proposed. Firstly, the saliency map of the infrared image is obtained by saliency detection. Then, the specific loss function and network architecture are designed based on the saliency map to improve the performance of the fusion algorithm. Specifically, the saliency map is normalized to [0, 1], used as a weight map to constrain the loss function. At the same time, the saliency map is binarized to extract salient regions and nonsalient regions. And, a generative adversarial network with dual discriminators is obtained. The two discriminators are used to distinguish the salient regions and the nonsalient regions, respectively, to promote the generator to generate better fusion results. The experimental results show that the fusion results of our method are better than those of the existing methods in both subjective and objective aspects.

Abstract 14467: Machine Learning for Identifying Salient Regions on the Coronary Artery on CT for Radiomic Feature Extraction and Characterization of CAD

Introduction: Radiomics is a quantitative approach that allows for high throughput extraction of image features such as texture. It has been used for characterization of coronary artery disease (CAD). However, the reliability of radiomic features (RFs) varies with the image segmentation quality. We present a new strategy to establish a segmentation confidence map spatially across the coronary arteries in the dataset. We then evaluate the quality of RFs in salient (high confidence) and non-salient (low confidence) regions in the coronary artery. Purpose: To identify salient regions in the coronary artery, providing high segmentation certainty as well as a site for informative radiomic feature extraction, useful in CAD characterization. Methods: In our approach, we first applied a deep learning-based segmentation model (U-Net with ResNet backbone) to segment the coronary arteries (Fig. 1a). Next, we constructed a mean probability atlas from the segmentation probability maps, using fiducial registration (Fig. 1b). Finally, we identified salient and non-salient regions within the segmentation (Fig. 1c) by masking the mean probability atlas with the predicted segmentation and then thresholding them. Human expert-generated manual segmentation was used as ground truth. RFs—shape (26), first order (19) and second order (75)—were extracted from salient and non-salient regions for comparative correlation analysis with features from ground truth segmentation. A higher correlation implies that these features are equally informative as those from the manual segmentation. Results: The model achieved a dice score of 79.8% on the hold-out set. The RFs extracted only from the salient regions (Proximal portion of RCA and LAD) of the coronary artery display strong correlations with RFs extracted from manual segmentations (Fig 1d). Conclusion: We present a framework to identify regions for optimal extraction of RFs for improved CAD characterization.

A 3D Keypoints Voting Network for 6DoF Pose Estimation in Indoor Scene

This paper addresses the problem of instance-level 6DoF pose estimation from a single RGBD image in an indoor scene. Many recent works have shown that a two-stage network, which first detects the keypoints and then regresses the keypoints for 6d pose estimation, achieves remarkable performance. However, the previous methods concern little about channel-wise attention and the keypoints are not selected by comprehensive use of RGBD information, which limits the performance of the network. To enhance RGB feature representation ability, a modular Split-Attention block that enables attention across feature-map groups is proposed. In addition, by combining the Oriented FAST and Rotated BRIEF (ORB) keypoints and the Farthest Point Sample (FPS) algorithm, a simple but effective keypoint selection method named ORB-FPS is presented to avoid the keypoints appear on the non-salient regions. The proposed algorithm is tested on the Linemod and the YCB-Video dataset, the experimental results demonstrate that our method outperforms the current approaches, achieves ADD(S) accuracy of 94.5% on the Linemod dataset and 91.4% on the YCB-Video dataset.

EDNet: Encoder-Decoder Network with Low Rank and Margin Regularizations for Saliency Detection

Deep learning based methods have achieved great progress in the saliency detection domain. However, it remains to be challenging to segment objects in complicated scenarios consistently and accurately. To address the problem, we propose an encoder-decoder based framework in this paper, where the encoder extracts deep features for each super-pixel via fast R-CNN and attention mechanism, and a loss term is introduced to penalize the segmentation inaccuracy; the decoder makes saliency predictions by two kinds of bidirectional long short-term memory networks: the segmented BLSTM collects information horizontally and the skipping BLSTM fuse them along the vertical direction. For further improving the detection accuracy and robustness, low rank constraint is imposed on the background super-pixels, and the classification margin between salient and non-salient regions is expanded simultaneously. In experiments, we evaluate the networks on five benchmark datasets, and the results under different metrics demonstrate that our model without any post-processing has superiority over other state-of-the-art ones.

A salient object detection framework using linear quadratic regulator controller

In this paper, a novel salient object detection framework based on Linear Quadratic Regulator (LQR) controller is proposed. The major goal of this research is to take advantage of optimal control theory for improving the performance of detecting salient objects in images. In this regard, for the sake of detection of salient and non-salient regions, two LQR-based control systems are employed. In the proposed framework, for the initialization of the control systems, background and foreground estimations have been done with two different strategies. Doing so, we would ultimately have more effective distinction between those regions. After the initialization step, the control systems refine both estimations in parallel until reaching a steady state for each of them. Within the mentioned process, by using optimal control concept, specifically LQR controller (for the first time in the field), control signals which are in charge of determining saliency values, would be constantly optimized. At the end, the raw saliency map will be generated by combination of background and foreground optimized initial maps. Finally, the integrated saliency map will be refined by using angular embedding method. The experimental evaluations on three benchmark datasets shows that the proposed framework performs well and introduces comparable results with some deep learning based methods.