Salient Regions Research Articles

Interpretable unsupervised learning enables accurate clustering with high-throughput imaging flow cytometry

A primary challenge of high-throughput imaging flow cytometry (IFC) is to analyze the vast amount of imaging data, especially in applications where ground truth labels are unavailable or hard to obtain. We present an unsupervised deep embedding algorithm, the Deep Convolutional Autoencoder-based Clustering (DCAEC) model, to cluster label-free IFC images without any prior knowledge of input labels. The DCAEC model first encodes the input images into the latent representations and then clusters based on the latent representations. Using the DCAEC model, we achieve a balanced accuracy of 91.9% for human white blood cell (WBC) clustering and 97.9% for WBC/leukemia clustering using the 3D IFC images and 3D DCAEC model. Above all, although no human recognizable features can separate the clusters of cells with protein localization, we demonstrate the fused DCAEC model can achieve a cluster balanced accuracy of 85.3% from the label-free 2D transmission and 3D side scattering images. To reveal how the neural network recognizes features beyond human ability, we use the gradient-weighted class activation mapping method to discover the cluster-specific visual patterns automatically. Evaluation results show that the automatically identified salient image regions have strong cluster-specific visual patterns for different clusters, which we believe is a stride for the interpretable neural network for cell analysis with high-throughput IFCs.

Scale-wise discriminative region learning for medical image segmentation

Vision Transformer (ViT) has shown comparable capabilities to convolutional neural networks for medical image segmentation in recent years. However, most ViT-based models fail to effectively model long-range feature dependencies at multi-scales and ignore the crucial importance of the semantic richness of features at each scale for medical segmentation. To address this problem, we propose a novel Scale-wise Discriminative Region Learning Network (SDRL-Net) in this paper, which guides the model to focus on salient regions by differential modeling the global context relationships at each scale. In SDRL-Net, a scale-wise enhancement module is proposed to achieve more distinguishing feature representations in the encoder by concentrating spatially localized information and differentiated regional interactions simultaneously. Furthermore, we propose a multi-scale upsampling module that focuses on global multi-scale information through pyramid attention and then complements the local upsampling information to achieve better segmentation. Extensive experiments on three widely used public datasets demonstrate that our proposed SDRL-Net can perform excellently and outperform most state-of-the-art medical image segmentation methods. Code is available at https://github.com/MiniCoCo-be/SDRL-Net.

Robust infrared small target detection with multi-feature fusion

The performance of the infrared search and track system depends on the robust of the infrared small target detection. In this paper, a novel small target detection with multi-feature fusion is proposed. At first, the salient proposal region is obtained with phase spectrum Fourier transform (PFT). Then, geometric moment is utilized to design an adaptive steerable edges awareness factor, which effectively eliminates the strong edge clutters. Afterwards, we adopt a three-layers local contrast structure and the difference-ration form is introduced to enhance the target and suppress the complex backgrounds. Finally, the adaptive threshold is applied to extract the target in the saliency map. Experiments are conducted on four real IR image sequences, which demonstrate that the proposed method outperforms the comparing methods with various complex backgrounds.

An Adaptive Multi-Content Complementary Network for Salient Object Detection

Deep learning methods for salient object detection (SOD) have been studied actively and promisingly. However, the existing methods mainly focus on the decoding process and ignore the differences in the contributions of different encoder blocks. To address this problem for SOD, we propose an adaptive multi-content complementary network (PASNet) for salient object detection which aims to exploit the valuable contextual information in the encoder fully. Unlike existing CNN-based methods, we adopt the pyramidal visual transformer (PVTv2) as the backbone network to learn global and local representations with its self-attention mechanism. Then, we follow the coarse-to-fine strategy and introduce two novel modules, including an advanced semantic fusion module (ASFM) and a self-refinement module (SRM). Among these, the ASFM takes local branches and adjacent branches as inputs and collects semantic and location information of salient objects from high-level features to generate an initial coarse saliency map. The coarse saliency map serves as the location guidance for low-level features, and the SRM is applied to capture detailed information disguised in low-level features. We expand the location information with high-level semantics from top to bottom across the salient region, which is effectively fused with detailed information through feature modulation. The model effectively suppresses noises in the features and significantly improves their expressive capabilities. To verify the effectiveness of our PASNet, we conducted extensive experiments on five challenging datasets, and the results show that the proposed model is superior to some of the current state-of-the-art methods under different evaluation metrics.

Feature-Specific Salience Maps in Human Cortex.

Priority map theory is a leading framework for understanding how various aspects of stimulus displays and task demands guide visual attention. Per this theory, the visual system computes a priority map, which is a representation of visual space indexing the relative importance, or priority, of locations in the environment. Priority is computed based on both salience, defined based on image-computable properties; and relevance, defined by an individual's current goals, and is used to direct attention to the highest-priority locations for further processing. Computational theories suggest that priority maps identify salient locations based on individual feature dimensions (e.g., color, motion), which are integrated into an aggregate priority map. While widely accepted, a core assumption of this framework, the existence of independent feature dimension maps in visual cortex, remains untested. Here, we tested the hypothesis that retinotopic regions selective for specific feature dimensions (color or motion) in human cortex act as neural feature dimension maps, indexing salient locations based on their preferred feature. We used fMRI activation patterns to reconstruct spatial maps while male and female human participants viewed stimuli with salient regions defined by relative color or motion direction. Activation in reconstructed spatial maps was localized to the salient stimulus position in the display. Moreover, the strength of the stimulus representation was strongest in the ROI selective for the salience-defining feature. Together, these results suggest that feature-selective extrastriate visual regions highlight salient locations based on local feature contrast within their preferred feature dimensions, supporting their role as neural feature dimension maps.SIGNIFICANCE STATEMENT Identifying salient information is important for navigating the world. For example, it is critical to detect a quickly approaching car when crossing the street. Leading models of computer vision and visual search rely on compartmentalized salience computations based on individual features; however, there has been no direct empirical demonstration identifying neural regions as responsible for performing these dissociable operations. Here, we provide evidence of a critical double dissociation that neural activation patterns from color-selective regions prioritize the location of color-defined salience while minimally representing motion-defined salience, whereas motion-selective regions show the complementary result. These findings reveal that specialized cortical regions act as neural "feature dimension maps" that are used to index salient locations based on specific features to guide attention.

Saliency detection of textured 3D models based on multi-view information and texel descriptor

Saliency-driven mesh simplification methods have shown promising results in maintaining visual detail, but effective simplification requires accurate 3D saliency maps. The conventional mesh saliency detection method may not capture salient regions in 3D models with texture. To address this issue, we propose a novel saliency detection method that fuses saliency maps from multi-view projections of textured models. Specifically, we introduce a texel descriptor that combines local convexity and chromatic aberration to capture texel saliency at multiple scales. Furthermore, we created a novel dataset that reflects human eye fixation patterns on textured models, which serves as an objective evaluation metric. Our experimental results demonstrate that our saliency-driven method outperforms existing approaches on several evaluation metrics. Our method source code can be accessed at https://github.com/bkballoon/mvsm-fusion and the dataset can be accessed at 10.5281/zenodo.8131602.

Drogue detection for autonomous aerial refueling via hybrid pigeon-inspired optimized color opponent and saliency aggregation

Drogue detection is one of the challenging tasks in autonomous aerial refueling due to the requirement for accuracy and rapidity. Saliency detection based on image intrinsic cues can achieve fast detection, but with poor accuracy. Recent studies reveal that optimization-based methods provide accurate and quick solutions for saliency detection. This paper presents a hybrid pigeon-inspired optimization method, the optimized color opponent, that aims to adjust the weight of color opponent channels to detect the drogue region. It can optimize the weights in the selected aerial refueling scene offline, and the results are applied for drogue detection in the scene. A novel algorithm aggregated by the optimized color opponent and robust background detection is presented to provide better precision and robustness. Experimental results on benchmark datasets and aerial refueling images show that the proposed method successfully extracts the saliency region or drogue and exhibits superior performance against the other saliency detection methods with intrinsic cues. The algorithm designed in this paper is competent for the drogue detection task of autonomous aerial refueling.

Attention-guided Multi-modality Interaction Network for RGB-D Salient Object Detection

The past decade has witnessed great progress in RGB-D salient object detection (SOD). However, there are two bottlenecks that limit its further development. The first one is low-quality depth maps. Most existing methods directly use raw depth maps to perform detection, but low-quality depth images can bring negative impacts to the detection performance. Hence, it is not desirable to utilize depth maps indiscriminately. The other one is how to effectively predict salient maps with clear boundary and complete salient region. To address these problems, an Attention-Guided Multi-Modality Interaction Network (AMINet) is proposed. First, we propose a new quality enhancement strategy for unreliable depth images, named D epth E nhancement M odule ( DEM ). With respect to the second issue, we propose C ross- M odality A ttention M odule ( CMAM ) to rapidly locate salient region. The B oundary- A ware M odule ( BAM ) is designed to utilize high-level feature to guide the low-level feature generation in a top-down way to make up for the dilution of the boundary. To further improve the accuracy, we propose A trous R efined B lock ( ARB ) to adaptively compensate for the shortcoming of atrous convolution. By integrating these interactive modules, features from depth and RGB streams can be refined efficiently, which consequently boosts the detection performance. Experimental results demonstrate the proposed AMINet exceeds state-of-the-art (SOTA) methods on several public RGB-D datasets.

TMNet: Triple-modal interaction encoder and multi-scale fusion decoder network for V-D-T salient object detection

Salient object detection methods based on two-modal images have achieved remarkable success with the aid of image acquisition equipment. However, environmental factors often interfere with the Depth and Thermal maps, rendering them ineffective in providing object information. To address this weakness, we utilize the VDT dataset, which includes Visible, Depth, and Thermal images, and propose a triple-modal interaction encoder and multi-scale fusion decoder network (TMNet) to highlight the salient regions. The triple-modal interaction encoder comprises the separation context-aware feature module, channel-wise fusion module, and triple-modal refinement and fusion module, enabling us to fully explore and utilize the complementarity between Visible, Depth, and Thermal information. The multi-scale fusion decoder involves the semantic-aware localizing module and contour-aware refinement module to extract and fuse the location and boundary information, yielding a high-quality saliency map. Extensive experiments on the public VDT-2048 dataset demonstrate that our TMNet outperforms existing state-of-the-art methods in terms of all evaluation metrics.

Frequency-aware feature aggregation network with dual-task consistency for RGB-T salient object detection

RGB-Thermal salient object detection (SOD) aims to merge two spectral images to segment visually appealing objects. Current methods primarily extract salient object information in the pixel perspective. However, biological and psychological research indicates notable frequency sensitivity of the human visual system (HVS). The high-frequency (HF) and low-frequency (LF) information in images are processed by different neural channels, which has been overlooked in SOD. In this study, we argue that the objective of RGB-T SOD is not only to enhance feature representation in the pixel-aware but also to emulate human visual mechanisms. To our best knowledge, we explore RGB-T SOD from the frequency perspective for the first time. Specifically, we first present a frequency-aware multi-spectral feature aggregation module (FMFA) to exploit the separability and complementarity of frequency-aware features, generating and making full use of LF and HF cues. FMFA improves the feature representation of RGB-T from the frequency perspective and provides stronger frequency cues for boundary auxiliary tasks. Then, we develop an HF-guided signed distance map prediction module (HF-SDM) with dual-task consistency to effectively alleviate the coarse mask caused by blur boundary. HF-SDM employs the geometric relationship of objects which boosts the interaction between salient regions and boundaries. As a result, the model can gain sharper boundaries for salient objects. Finally, we propose a frequency-aware feature aggregation network (FFANet) incorporated with dual-task learning. Extensive experiments on RGB-T SOD datasets demonstrate that our proposed method outperforms other state-of-the-art methods. Ablation studies and visualizations further verify the effectiveness and interpretability of our method.

Comprehensive mining of information in Weakly Supervised Semantic Segmentation: Saliency semantics and edge semantics

In the studies of Weakly Supervised Semantic Segmentation (WSSS) with image-level labels, there is an issue of incomplete semantic information, which we summarize as insufficient saliency semantic mining and neglected edge semantics. We proposes a two-stage framework, Saliency Semantic Full Mining-Edge Semantic Mining (SSFM-ESM), which views WSSS from the perspective of comprehensive information mining. In the first stage, we rely on SSFM to address the insufficient saliency semantic mining. The network learns feature representations consistent with salient regions via the proposed pixel-level class-agnostic distance loss. Then, the full saliency semantic information is mined by explicitly receiving pixel-level feedback. The initial pseudo-label with complete saliency semantic information can be obtained after the first stage. In the second stage, we focus on the mining of edge semantic information through the proposed edge semantic mining module. Specifically, we guide the initial pseudo-label avoid learning about false semantic information and obtain high-confidence edge semantics. The self-correction ability of the segmentation network is also fully utilized to obtain more edge semantic information. Extensive experiments are conducted on the PASCAL VOC 2012 and MS COCO 2014 datasets to verify the feasibility and superiority of this approach.

Deep unsupervised part-whole relational visual saliency

Deep Supervised Salient Object Detection (SSOD) excessively relies on large-scale annotated pixel-level labels which consume intensive labour acquiring high quality labels. In such precondition, deep Unsupervised Salient Object Detection (USOD) draws public attention. Under the framework of the existing deep USOD methods, they mostly generate pseudo labels by fusing several hand-crafted detectors’ results. On top of that, a Fully Convolutional Network (FCN) will be trained to detect salient regions separately. While the existing USOD methods have achieved some progress, there are still challenges for them towards satisfactory performance on the complex scene, including (1) poor object wholeness owing to neglecting the hierarchy of those salient regions; (2) unsatisfactory pseudo labels causing by unprimitive fusion of hand-crafted results. To address these issues, in this paper, we introduce the property of part-whole relations endowed by a Belief Capsule Network (BCNet) for deep USOD, which is achieved by a multi-stream capsule routing strategy with a belief score for each stream within the CapsNets architecture. To train BCNet well, we generate high-quality pseudo labels from multiple hand-crafted detectors by developing a consistency-aware fusion strategy. Concretely, a weeding out criterion is first defined to filter out unreliable training samples based on the inter-method consistency among four hand-crafted saliency maps. In the following, a dynamic fusion mechanism is designed to generate high-quality pseudo labels from the remaining samples for BCNet training. Experiments on five public datasets illustrate the superiority of the proposed method. Codes have been released on: https://github.com/Mirlongue/Deep-Unsupervised-Part-Whole-Relational-Visual-Saliency.

Temporal-agnostic change region proposal for semantic change detection

Remote sensing imagery allows temporal and large-scale observation of the Earth, and advanced techniques such as deep learning have been developed to deal with the massive data. As a result, by combining remote sensing with deep learning, it is possible to realize change detection, which can reveal the change in land cover and land use. Recent work has focused on semantic change detection (SCD), which simultaneously locates where the changes took place while identifying what objects changed between the different periods. Deep learning based SCD methods are typically based on a multi-task structure, which combines the classification and binary change detection (BCD). In this structure, the change region proposals are derived from the transformation (e.g., concatenation, differencing) of multi-temporal images, but after being processed by a convolutional neural network, there can be an incomplete response in the spatio-temporal features. Because of the high similarity of the color, texture, and geometric information of the ground objects in temporal high-resolution remote sensing imagery, this reduces the distinguishability of features and makes it difficult to identify changed areas. Meanwhile, there is less interaction between the BCD and classification, resulting in semantic inconsistency in changed areas. In this paper, the temporal-agnostic change region proposal network (TCRPN) is proposed, which is based on exploring the saliency of the changed regions in single-temporal images to highlight the changed areas in the spatio-temporal features and improve the response to change. As the changed areas can be the salient regions in the single-temporal images, a single-image salient area extractor (SSAE) is proposed to extract the salient areas from the different temporal images, obtaining salient change region candidates. As the change-related salient regions are temporally agnostic, temporal saliency fusion is used to integrate the candidates extracted from the single-temporal images to obtain a more reliable salient area map, where the salient area map is used to enhance the spatio-temporal features. Meanwhile, a shortcut is proposed to further make the change representation more distinguishable and suppress incorrectly highlighted regions. Directly utilizing multi-temporal images for change region proposal can reduce the change information loss and increase the interaction between the classification and BCD tasks to obtain better spatio-temporal features. The TCRPN is simple and flexible, and can be plugged into any multi-task SCD network. We selected four advanced multi-task SCD methods as the baseline, and added the TCRPN in their encoder or decoder part to conduct comprehensive experiments on three large-scale SCD datasets. The experimental results confirmed the effectiveness of the proposed method.

Multi-feature fusion learning for Alzheimer's disease prediction using EEG signals in resting state.

Diagnosing Alzheimer's disease (AD) lesions via visual examination of Electroencephalography (EEG) signals poses a considerable challenge. This has prompted the exploration of deep learning techniques, such as Convolutional Neural Networks (CNNs) and Visual Transformers (ViTs), for AD prediction. However, the classification performance of CNN-based methods has often been deemed inadequate. This is primarily attributed to CNNs struggling with extracting meaningful lesion signals from the complex and noisy EEG data. In contrast, ViTs have demonstrated proficiency in capturing global signal patterns. In light of these observations, we propose a novel approach to enhance AD risk assessment. Our proposition involves a hybrid architecture, merging the strengths of CNNs and ViTs to compensate for their respective feature extraction limitations. Our proposed Dual-Branch Feature Fusion Network (DBN) leverages both CNN and ViT components to acquire texture features and global semantic information from EEG signals. These elements are pivotal in capturing dynamic electrical signal changes in the cerebral cortex. Additionally, we introduce Spatial Attention (SA) and Channel Attention (CA) blocks within the network architecture. These attention mechanisms bolster the model's capacity to discern abnormal EEG signal patterns from the amalgamated features. To make well-informed predictions, we employ a two-factor decision-making mechanism. Specifically, we conduct correlation analysis on predicted EEG signals from the same subject to establish consistency. This is then combined with results from the Clinical Neuropsychological Scale (MMSE) assessment to comprehensively evaluate the subject's susceptibility to AD. Our experimental validation on the publicly available OpenNeuro database underscores the efficacy of our approach. Notably, our proposed method attains an impressive 80.23% classification accuracy in distinguishing between AD, Frontotemporal dementia (FTD), and Normal Control (NC) subjects. This outcome outperforms prevailing state-of-the-art methodologies in EEG-based AD prediction. Furthermore, our methodology enables the visualization of salient regions within pathological images, providing invaluable insights for interpreting and analyzing AD predictions.

Fast autofocusing of recorded planes by salient feature region for coherent diffraction imaging.

In multi-distance coherent diffraction imaging, the task of distance calculation for multi-diffraction images is cumbersome. The information features are hard-to-extract and the region of interest extraction algorithms are difficult to be adopted. A universal salient feature region selection algorithm by using the area with the highest density of corners is proposed to extract the most representative feature region. In addition, equally spaced recording modes and mismatched diffraction distances will result in system noise and destroy image quality. The polydirectional maximum gradient is offered as a sharpness criterion to weigh a quantitative feature for the final pattern. A fast, sensitive, and high-accuracy autofocusing and sample reconstruction can be achieved using only a small number of images while ensuring that morphological properties and quantification of the reconstructions are not compromised. The proposed method is promising for biological and medical dynamic observations for computational imaging systems.

A shadow preservation framework for effective content-aware image retargeting process

<p>In the discipline of image retargeting, the structure of shadows in the image is required to be preserved since it gives structural information about the objects. Many traditional image retargeting techniques do not pay attention to preserve the shadow objects present in the image. In this paper a shadow-preservation framework is proposed in which the saliency map is derived using segmentation-based gPb-owt-ucm approach to emphasize the salient regions. To assess the quality of saliency map a fixation prediction analysis is conducted on each section of the image using the eye tracker technology. Second, to pay more attention on the shadow objects a shadow map is developed after executing image pre-processing, shadow detection, region growing, and region filling operations. To rectify the issue of misclassification of shadow pixels erosion and dilation operations are performed. To obtain an efficient importance map, a gradient map is prepared using canny operator which is further combined with the saliency map and shadow map. The obtained importance map is supplied to the seam diversion based image retargeting (SDIR) technique to resize the image in horizontal and vertical directions. To justify the efficiency of the proposed framework’s the obtained results are compared with the existing state-of-the-art. By preserving shadows, the proposed framework enhances the overall visual quality and maintains the integrity of objects in the retargeted images. The framework’s effectiveness is validated through quantitative evaluations and visual comparisons with existing methods. The results demonstrate its potential for improving content-aware image retargeting applications, paving the way for more realistic and visually appealing image resizing techniques in various domains.<em></em></p>

Exploring Focus and Depth-Induced Saliency Detection for Light Field.

An abundance of features in the light field has been demonstrated to be useful for saliency detection in complex scenes. However, bottom-up saliency detection models are limited in their ability to explore light field features. In this paper, we propose a light field saliency detection method that focuses on depth-induced saliency, which can more deeply explore the interactions between different cues. First, we localize a rough saliency region based on the compactness of color and depth. Then, the relationships among depth, focus, and salient objects are carefully investigated, and the focus cue of the focal stack is used to highlight the foreground objects. Meanwhile, the depth cue is utilized to refine the coarse salient objects. Furthermore, considering the consistency of color smoothing and depth space, an optimization model referred to as color and depth-induced cellular automata is improved to increase the accuracy of saliency maps. Finally, to avoid interference of redundant information, the mean absolute error is chosen as the indicator of the filter to obtain the best results. The experimental results on three public light field datasets show that the proposed method performs favorably against the state-of-the-art conventional light field saliency detection approaches and even light field saliency detection approaches based on deep learning.

Registration of Large Optical and SAR Images with Non-Flat Terrain by Investigating Reliable Sparse Correspondences

Optical and SAR image registration is the primary procedure to exploit the complementary information from the two different image modal types. Although extensive research has been conducted to narrow down the vast radiometric and geometric gaps so as to extract homogeneous characters for feature point matching, few works have considered the registration issue for non-flat terrains, which will bring in more difficulties for not only sparse feature point matching but also outlier removal and geometric relationship estimation. This article addresses these issues with a novel and effective optical-SAR image registration framework. Firstly, sparse feature points are detected based on the phase congruency moment map of the textureless SAR image (SAR-PC-Moment), which helps to identify salient local regions. Then a template matching process using very large local image patches is conducted, which increases the matching accuracy by a significant margin. Secondly, a mutual verification-based initial outlier removal method is proposed, which takes advantage of the different mechanisms of sparse and dense matching and requires no geometric consistency assumption within the inliers. These two procedures will produce a putative correspondence feature point (CP) set with a low outlier ratio and high reliability. In the third step, the putative CPs are used to segment the large input image of non-flat terrain into dozens of locally flat areas using a recursive random sample consensus (RANSAC) method, with each locally flat area co-registered using an affine transformation. As for the mountainous areas with sharp elevation variations, anchor CPs are first identified, and then optical flow-based pixelwise dense matching is conducted. In the experimental section, ablation studies using four precisely co-registered optical-SAR image pairs of flat terrain quantitatively verify the effectiveness of the proposed SAR-PC-Moment-based feature point detector, big template matching strategy, and mutual verification-based outlier removal method. Registration results on four 1 m-resolution non-flat image pairs prove that the proposed framework is able to produce robust and quite accurate registration results.

Optic disc detection based on fully convolutional network and weighted matrix recovery model.

Eye diseases often affect human health. Accurate detection of the optic disc contour is one of the important steps in diagnosing and treating eye diseases. However, the structure of fundus images is complex, and the optic disc region is often disturbed by blood vessels. Considering that the optic disc is usually a saliency region in fundus images, we propose a weakly-supervised optic disc detection method based on the fully convolution neural network (FCN) combined with the weighted low-rank matrix recovery model (WLRR). Firstly, we extract the low-level features of the fundus image and cluster the pixels using the Simple Linear Iterative Clustering (SLIC) algorithm to generate the feature matrix. Secondly, the top-down semantic prior information provided by FCN and bottom-up background prior information of the optic disc region are used to jointly construct the prior information weighting matrix, which more accurately guides the decomposition of the feature matrix into a sparse matrix representing the optic disc and a low-rank matrix representing the background. Experimental results on the DRISHTI-GS dataset and IDRiD dataset show that our method can segment the optic disc region accurately, and its performance is better than existing weakly-supervised optic disc segmentation methods. Graphical abstract of optic disc segmentation.

Retraction Note: Recurrent learning of context for salient region detection

Retraction Note: Recurrent learning of context for salient region detection