Salient Image Regions Research Articles

Automated Identification of Clinically Relevant Regions in Glaucoma OCT Reports Using Expert Eye Tracking Data and Deep Learning.

To propose a deep learning-based approach for predicting the most-fixated regions on optical coherence tomography (OCT) reports using eye tracking data of ophthalmologists, assisting them in finding medically salient image regions. We collected eye tracking data of ophthalmology residents, fellows, and faculty as they viewed OCT reports to detect glaucoma. We used a U-Net model as the deep learning backbone and quantized eye tracking coordinates by dividing the input report into an 11 × 11 grid. The model was trained to predict the grids on which fixations would land in unseen OCT reports. We investigated the contribution of different variables, including the viewer's level of expertise, model architecture, and number of eye gaze patterns included in training. Our approach predicted most-fixated regions in OCT reports with precision of 0.723, recall of 0.562, and f1-score of 0.609. We found that using a grid-based eye tracking structure enabled efficient training and using a U-Net backbone led to the best performance. Our approach has the potential to assist ophthalmologists in diagnosing glaucoma by predicting the most medically salient regions on OCT reports. Our study suggests the value of eye tracking in guiding deep learning algorithms toward informative regions when experts may not be accessible. By suggesting important OCT report regions for a glaucoma diagnosis, our model could aid in medical education and serve as a precursor for self-supervised deep learning approaches to expedite early detection of irreversible vision loss owing to glaucoma.

DCFusion: Difference correlation-driven fusion mechanism of infrared and visible images

In end-to-end image fusion models, the loss function significantly impacts performance. However, most loss functions treat salient and background regions in source images equally, failing to distinguish complementary areas in multimodal images. This limits the model’s ability to effectively integrate information from these regions. Therefore, we propose difference correlation-driven fusion mechanism of infrared and visible images, which called DCFusion. Specifically, the model utilizes a dual-branch interactive network that dynamically fuses cross-modal multi-scale complementary information through element-wise multiplication, effectively integrating region-specific information. We introduce a two-stage method for generating salient target masks that adaptively focus on high-contrast regions in infrared images by analyzing pixel contrasts in local areas. Furthermore, we utilize the salient target masks to create heterogeneous images and design the LSCD loss function to minimize the information gap between the heterogeneous images and the fused image, thereby enhancing the model’s interpretability. Experiments on the RoadScene and TNO datasets show that DCFusion surpasses with existing representativity fusion approaches, achieving state-of-the-art performance in both subjective visual and objective evaluations. Our code will be publicly available at https://github.com/MinLila/DCFusion.

Multiview adaptive attention pooling for image–text retrieval

Image–Text Retrieval (ITR) aims to bridge the heterogeneity gap between image and text and establish the retrieval ability between the two modalities. There is a large intraclass difference in visual language data; that is, the content of an image can be described from different views. Representing images and texts as single embedded features to measure similarity makes it difficult to capture the diversity and fine-grained information of modal features. Previous methods use cumbersome dilated convolution structures or stack multiple feature pooling operators to perform multiview learning of images, and then take the view with the highest similarity score to the text as the alignment result. This may lead to two problems: (1) The model becomes complex, and has poor scalability and limited feature learning ability; (2) The matching strategy with the highest similarity score may have the problem that text view features deliberately emphasize a certain area, resulting in sub-optimal matching. Therefore, we propose a multiview adaptive attention pooling (MVAAP) network, a simpler and more effective multiview global feature embedding method. Specifically, MVAAP learns the query for each view, extracts the salient image region of the view through an adaptive attention mechanism, and generates an optimal pooling strategy to aggregate it into the global feature of the view. Beyond that, we introduce multiview embedding of the text branch and consider the response between different views to improve the generalization ability of the model. Plenty of experiments on the two mainstream cross-modal datasets of MS-COCO and Flickr30K prove the accuracy and superiority of the method.

SalDA: DeepConvNet Greets Attention for Visual Saliency Prediction

Predicting salient regions in images requires the capture of contextual information in the scene. Conventional saliency models typically use the encoder-decoder architecture and multi-scale feature fusion for modeling contextual features, which, however, possess huge computational cost and model parameters. In this paper, we address the saliency prediction task by capturing long-range dependencies based on the self-attention mechanism. Self-attention has been widely used in image recognition or other classification tasks, but is still rarely being considered in regression based saliency prediction task. Inspired by non-local block, we propose a new SALiency prediction network in which DeepConvNet is integrated with the Attention mechanism, namely SalDA. Considering each feature map may capture different salient regions, our spatial attention module firstly adaptively aggregates the feature at each position by a weighted sum of the features at all positions within each independent channel. Meanwhile, in order to capture interdependence between channels, we also introduce a channel attention module to integrate different features among different channels. We combine these two attention modules into a multi-attention module to further improve the saliency map prediction for the network. We show the effectiveness of SalDA on the largest saliency prediction dataset SALICON. Compared to other state-of-the-art methods in this area, we can yield comparable saliency prediction performance, but with substantially less model parameters and shorter inference time.

HMDA: A Hybrid Model with Multi-scale Deformable Attention for Medical Image Segmentation.

Transformers have been applied to medical image segmentation tasks owing to their excellent longrange modeling capability, compensating for the failure of Convolutional Neural Networks (CNNs) to extract global features. However, the standardized self-attention modules in Transformers, characterized by a uniform and inflexible pattern of attention distribution, frequently lead to unnecessary computational redundancy with high-dimensional data, consequently impeding the model's capacity for precise concentration on salient image regions. Additionally, achieving effective explicit interaction between the spatially detailed features captured by CNNs and the long-range contextual features provided by Transformers remains challenging. In this architecture, we propose a Hybrid Transformer and CNN architecture with Multi-scale Deformable Attention(HMDA), designed to address the aforementioned issues effectively. Specifically, we introduce a Multi-scale Spatially Adaptive Deformable Attention (MSADA) mechanism, which attends to a small set of key sampling points around a reference within the multi-scale features, to achieve better performance. In addition, we propose the Cross Attention Bridge (CAB) module, which integrates multi-scale transformer and local features through channelwise cross attention enriching feature synthesis. HMDA is validated on multiple datasets, and the results demonstrate the effectiveness of our approach, which achieves competitive results compared to the previous methods.

MSEDNet: Multi-scale fusion and edge-supervised network for RGB-T salient object detection

RGB-T Salient object detection (SOD) is to accurately segment salient regions in both visible light images and thermal infrared images. However, most of existing methods for SOD neglects the critical complementarity between multiple modalities images, which is beneficial to further improve the detection accuracy. Therefore, this work introduces the MSEDNet RGB-T SOD method. We utilize an encoder to extract multi-level modalities features from both visible light images and thermal infrared images, which are subsequently categorized into high, medium, and low level. Additionally, we propose three separate feature fusion modules to comprehensively extract complementary information between different modalities during the fusion process. These modules are applied to specific feature levels: the Edge Dilation Sharpening module for low-level features, the Spatial and Channel-Aware module for mid-level features, and the Cross-Residual Fusion module for high-level features. Finally, we introduce an edge fusion loss function for supervised learning, which effectively extracts edge information from different modalities and suppresses background noise. Comparative demonstrate the superiority of the proposed MSEDNet over other state-of-the-art methods. The code and results can be found at the following link: https://github.com/Zhou-wy/MSEDNet.

Interpretable unsupervised learning enables accurate clustering with high-throughput imaging flow cytometry

A primary challenge of high-throughput imaging flow cytometry (IFC) is to analyze the vast amount of imaging data, especially in applications where ground truth labels are unavailable or hard to obtain. We present an unsupervised deep embedding algorithm, the Deep Convolutional Autoencoder-based Clustering (DCAEC) model, to cluster label-free IFC images without any prior knowledge of input labels. The DCAEC model first encodes the input images into the latent representations and then clusters based on the latent representations. Using the DCAEC model, we achieve a balanced accuracy of 91.9% for human white blood cell (WBC) clustering and 97.9% for WBC/leukemia clustering using the 3D IFC images and 3D DCAEC model. Above all, although no human recognizable features can separate the clusters of cells with protein localization, we demonstrate the fused DCAEC model can achieve a cluster balanced accuracy of 85.3% from the label-free 2D transmission and 3D side scattering images. To reveal how the neural network recognizes features beyond human ability, we use the gradient-weighted class activation mapping method to discover the cluster-specific visual patterns automatically. Evaluation results show that the automatically identified salient image regions have strong cluster-specific visual patterns for different clusters, which we believe is a stride for the interpretable neural network for cell analysis with high-throughput IFCs.

Temporal-agnostic change region proposal for semantic change detection

Remote sensing imagery allows temporal and large-scale observation of the Earth, and advanced techniques such as deep learning have been developed to deal with the massive data. As a result, by combining remote sensing with deep learning, it is possible to realize change detection, which can reveal the change in land cover and land use. Recent work has focused on semantic change detection (SCD), which simultaneously locates where the changes took place while identifying what objects changed between the different periods. Deep learning based SCD methods are typically based on a multi-task structure, which combines the classification and binary change detection (BCD). In this structure, the change region proposals are derived from the transformation (e.g., concatenation, differencing) of multi-temporal images, but after being processed by a convolutional neural network, there can be an incomplete response in the spatio-temporal features. Because of the high similarity of the color, texture, and geometric information of the ground objects in temporal high-resolution remote sensing imagery, this reduces the distinguishability of features and makes it difficult to identify changed areas. Meanwhile, there is less interaction between the BCD and classification, resulting in semantic inconsistency in changed areas. In this paper, the temporal-agnostic change region proposal network (TCRPN) is proposed, which is based on exploring the saliency of the changed regions in single-temporal images to highlight the changed areas in the spatio-temporal features and improve the response to change. As the changed areas can be the salient regions in the single-temporal images, a single-image salient area extractor (SSAE) is proposed to extract the salient areas from the different temporal images, obtaining salient change region candidates. As the change-related salient regions are temporally agnostic, temporal saliency fusion is used to integrate the candidates extracted from the single-temporal images to obtain a more reliable salient area map, where the salient area map is used to enhance the spatio-temporal features. Meanwhile, a shortcut is proposed to further make the change representation more distinguishable and suppress incorrectly highlighted regions. Directly utilizing multi-temporal images for change region proposal can reduce the change information loss and increase the interaction between the classification and BCD tasks to obtain better spatio-temporal features. The TCRPN is simple and flexible, and can be plugged into any multi-task SCD network. We selected four advanced multi-task SCD methods as the baseline, and added the TCRPN in their encoder or decoder part to conduct comprehensive experiments on three large-scale SCD datasets. The experimental results confirmed the effectiveness of the proposed method.

GuidedMixup: An Efficient Mixup Strategy Guided by Saliency Maps

Data augmentation is now an essential part of the image training process, as it effectively prevents overfitting and makes the model more robust against noisy datasets. Recent mixing augmentation strategies have advanced to generate the mixup mask that can enrich the saliency information, which is a supervisory signal. However, these methods incur a significant computational burden to optimize the mixup mask. From this motivation, we propose a novel saliency-aware mixup method, GuidedMixup, which aims to retain the salient regions in mixup images with low computational overhead. We develop an efficient pairing algorithm that pursues to minimize the conflict of salient regions of paired images and achieve rich saliency in mixup images. Moreover, GuidedMixup controls the mixup ratio for each pixel to better preserve the salient region by interpolating two paired images smoothly. The experiments on several datasets demonstrate that GuidedMixup provides a good trade-off between augmentation overhead and generalization performance on classification datasets. In addition, our method shows good performance in experiments with corrupted or reduced datasets.

Application of Minnan Folk Light and Shadow Animation in Built Environment in Object Detection Algorithm

To resolve the problems of deep convolutional neural network models with many parameters and high memory resource consumption, a lightweight network-based algorithm for building detection of Minnan folk light synthetic aperture radar (SAR) images is proposed. Firstly, based on the rotating target detection algorithm R-centernet, the Ghost ResNet network is constructed to reduce the number of model parameters by replacing the traditional convolution in the backbone network with Ghost convolution. Secondly, a channel attention module integrating width and height information is proposed to enhance the network’s ability to accurately locate salient regions in folk light images. Content-aware reassembly of features (CARAFE) up-sampling is used to replace the deconvolution module in the network to fully incorporate feature map information during up-sampling to improve target detection. Finally, the constructed dataset of rotated and annotated light and shadow SAR images is trained and tested using the improved R-centernet algorithm. The experimental results show that the improved algorithm improves the accuracy by 3.8%, the recall by 1.2% and the detection speed by 12 frames/second compared with the original R-centernet algorithm.

Salient-Centeredness and Saliency Size in Computational Aesthetics

We investigate the optimal aesthetic location and size of a single dominant salient region in a photographic image. Existing algorithms for photographic composition do not take full account of the spatial positioning or sizes of these salient regions. We present a set of experiments to assess aesthetic preferences, inspired by theories of centeredness, principal lines, and Rule-of-Thirds. Our experimental results show a clear preference for the salient region to be centered in the image and that there is a preferred size of non-salient border around this salient region. We thus propose a novel image cropping mechanism for images containing a single salient region to achieve the best aesthetic balance. Our results show that the Rule-of-Thirds guideline is not generally valid but also allow us to hypothesize in which situations it is useful and in which it is inappropriate.

Foreground-Back ground Segmentation using K-Means Clustering Algorithm and Support Vector Machine

Foreground-background image segmentation has been an important research problem. It is one of the main tasks in the field of computer vision whose purpose is detecting variations in image sequences. It provides candidate objects for further attentional selection, e.g., in video surveillance. In this paper, we introduce an automatic and efficient Foreground-background segmentation. The proposed method starts with the detection of visually salient image regions with a saliency map that uses Fourier transform and a Gaussian filter. Then, each point in the maps classifies as salient or non-salient using a binary threshold. Next, a hole filling operator is applied for filling holes in the achieved image, and the area-opening method is used for removing small objects from the image. For better separation of the foreground and background, dilation and erosion operators are also used. Erosion and dilation operators are applied for shrinking and expanding the achieved region. Afterward, the foreground and background samples are achieved. Because the number of these data is large, K-means clustering is used as a sampling technique to restrict computational efforts in the region of interest. K cluster centers for each region are set for training of Support Vector Machine (SVM). SVM, as a powerful binary classifier, is used to segment the interest area from the background. The proposed method is applied on a benchmark dataset consisting of 1000 images and experimental results demonstrate the supremacy of the proposed method to some other foreground-background segmentation methods in terms of ER, VI, GCE, and PRI.

Cycle Generative Adversarial Network Based on Gradient Normalization for Infrared Image Generation

Image generation technology is currently one of the popular directions in computer vision research, especially regarding infrared imaging, bearing critical applications in the military field. Existing algorithms for generating infrared images from visible images are usually weak in perceiving the salient regions of images and cannot effectively highlight the ability to generate texture details in infrared images, resulting in less texture details and poorer generated image quality. In this study, a cycle generative adversarial network method based on gradient normalization was proposed to address the current problems of poor infrared image generation, lack of texture detail and unstable models. First, to address the problem of limited feature extraction capability of the UNet generator network that makes the generated IR images blurred and of low quality, the use of the residual network with better feature extraction capability in the generator was employed to make the generated infrared images highly defined. Secondly, in order to solve issues concerning severe lack of detailed information in the generated infrared images, channel attention and spatial attention mechanisms were introduced into the ResNet with the attention mechanism used to weight the generated infrared image features in order to enhance feature perception of the prominent regions of the image, helping to generate image details. Finally, to tackle the problem where the current training models of adversarial generator networks are insufficiently stable, which leads to easy collapse of the model, a gradient normalization module was introduced in the discriminator network to stabilize the model and render it less prone to collapse during the training process. The experimental results on several datasets showed that the proposed method obtained satisfactory data in terms of objective evaluation metrics. Compared with the cycle generative adversarial network method, the proposed method in this work exhibited significant improvement in data validity on multiple datasets.

CoCALC: A Self-Supervised Visual Place Recognition Approach Combining Appearance and Geometric Information

Visual place recognition (VPR) is considered among the most complicated tasks in SLAM due to the multiple challenges of drastic variations in both appearance and viewpoint. To address this issue, this article presents a self-supervised and lightweight VPR approach (namely CoCALC) that fully utilizes the appearance and geometric information provided by images. The main thing that makes CoCALC ultra-lightweight (only 0.27 MB) is our use of Depthwise Separable Convolution (DSC), a simple but effective architecture that enables our model to generate a more robust image representation. The network trained specifically for VPR can efficiently extract deep convolutional features from salient image regions that have relatively higher entropy, thereby expanding its applications on resource-limited platforms without GPUs. To further eliminate the negative consequences of the high percent false matches, a novel band-matrix-based geometric check is employed to filter out the incorrect matching of image patches, and the impact of different bandwidths on the recall rate is discussed. Results on several benchmark datasets confirm that the proposed CoCALC can yield state-of-the-art performance and superior generalization with acceptable efficiency. All relevant codes are provided at https://github.com/LiKangyuLKY/CoCALC-VPR for further studies.

Thermal images-aware guided early fusion network for cross-illumination RGB-T salient object detection

RGB-T salient object detection (SOD) has been developed rapidly and achieved excellent results in recent years. However, some problems have not yet been solved. The current RGB-T datasets contain only a tiny amount of low-illumination data. The RGB-T SOD method trained based on these RGB-T datasets does not detect the salient objects in extremely low-illumination scenes very well. To improve the detection performance of low-illumination data, we can spend a lot of labor to label low-illumination data, but we tried a new idea to solve the problem by making full use of the properties of Thermal (T) images. Therefore, we propose a T-aware guided early fusion network for cross-illumination salient object detection. Specifically, in the training and testing stage, we use normal illumination data to train our network and then use low and extremely low-illumination data to verify the effectiveness of our method. In the early fusion stage, we propose a T-aware guided module (T-aware) for enhancing salient regions of RGB images at different illumination levels. Secondly, in the decoding stage, we use T images to guide the cross-modal fusion of RGB and T images. In addition, we propose a cross-modal fusion localization-remote correction module (CFL-RCM), which is used to deeply screen and correct redundant information generated by illumination variations. Comparative experiments on the VDT-2048 dataset validate the superior performance of our method on the cross-illumination RGB-T saliency detection. We also obtained favorable results on generalizability experiments with VT5000, VT1000, and VT821 datasets.

Feature learning network with transformer for multi-label image classification

The purpose of multi-label image classification task is to accurately assign a set of labels to the objects in images. Although promising results have been achieved, most of the existing methods cannot effectively learn multi-scale features, so it is difficult to identify small-scale objects from images. Besides, current attention-based methods tend to learn the most salient feature regions in images, but fail to excavate various potential useful features concealed by the most salient feature, thus limiting the further improvement of model performance. To address above issues, we propose a novel Feature Learning network based on Transformer to learn salient features and excavate potential useful features (FL-Tran). Specifically, in order to solve the problem that current methods are difficult to identify small-scale objects, we first present a novel multi-scale fusion module (MSFM) to align high-level features and low-level features to learn multi-scale features. Additionally, a spatial attention module (SAM) utilizing transformer encoder is introduced to capture salient object features in images to enhance the model performance. Furthermore, we devise a feature enhancement and suppression module (FESM) with the aim of excavating potential useful features concealed by the most salient features. By suppressing the most salient features obtained in current SAM layer, and then forcing subsequent SAM layer to excavate potential salient features in feature maps, FL-Tran model can learn various useful features more comprehensively. Extensive experiments on MS-COCO 2014, PASCAL VOC 2007, and NUS-WIDE datasets demonstrate that our proposed FL-Tran model outperforms current state-of-the-art methods.

A multi-band image synchronous fusion method based on saliency

A new saliency-based multi-band image synchronous fusion method is proposed. First, combined window filtering is used to decompose the image into base and detail layers to better protect the edges of the image. Second, gamma correction is used for infrared image salient region extraction. To make it more applicable to the base layer fusion, we propose a method to calculate the parameters adaptively according to the image content. Consequently, the fusion result can better retain the thermal radiation information of the target. We then consider both the image quality and the amount of information contained in the image and construct an optimization model for synchronous fusion and noise reduction of detail layers to reduce the influence of noisy images on the fusion results. Finally, the fused base and detail layers are combined to obtain the final image. Our proposed method is evaluated and compared with representative methods, both qualitatively and quantitatively, applied to the TNO dataset. The results demonstrate that the proposed method aligns with human visual observation in highlighting salient targets and retaining valid detail information. The quality metrics are better than the comparison methods on the whole.

Deeply Activated Salient Region for Instance Search

The performance of instance search relies heavily on the ability to locate and describe a wide variety of object instances in a video/image collection. Due to the lack of a proper mechanism for locating instances and deriving feature representation, instance search is generally only effective when the instances are from known object categories. In this article, a simple but effective instance-level feature representation approach is presented. Different from the existing approaches, the issues of class-agnostic instance localization and distinctive feature representation are considered. The former is achieved by detecting salient instance regions from an image by a layer-wise back-propagation process. The back-propagation starts from the last convolution layer of a pre-trained CNNs that is originally used for classification. The back-propagation proceeds layer by layer until it reaches the input layer. This allows the salient instance regions in the input image from both known and unknown categories to be activated. Each activated salient region covers the full or, more usually, a major range of an instance. The distinctive feature representation is produced by average-pooling on the feature map of a certain layer with the detected instance region. Experiments show that this kind of feature representation demonstrates considerably better performance than most of the existing approaches.

Double-Attention YOLO: Vision Transformer Model Based on Image Processing Technology in Complex Environment of Transmission Line Connection Fittings and Rust Detection

Transmission line fittings have been exposed to complex environments for a long time. Due to the interference of haze and other environmental factors, it is often difficult for the camera to obtain high quality on-site images, and the traditional image processing technology and convolution neural networks find it difficult to effectively deal with the dense detection task of small targets with occlusion interference. Therefore, an image processing method based on an improved dark channel defogging algorithm, the fusion channel spatial attention mechanism, Vision Transformer, and the GhostNet model compression method is proposed in this paper. Based on the global receptive field of the saliency region capture and enhancement model, a small target detection network Double-attention YOLO for complex environments is constructed. The experimental results show that embedding a multi-head self-attention component into a convolutional neural network can help the model to better interpret the multi-scale global semantic information of images. In this way, the model learns more easily the distinguishable features in the image representation. Embedding an attention mechanism module can make the neural network pay more attention to the salient region of image. Dual attention fusion can balance the global and local characteristics of the model, to improve the performance of model detection.

Direct Gene Expression Profile Prediction for Uveal Melanoma from Digital Cytopathology Images via Deep Learning and Salient Image Region Identification

To demonstrate that deep learning (DL) methods can produce robust prediction of gene expression profile (GEP) in uveal melanoma (UM) based on digital cytopathology images. Evaluation of a diagnostic test or technology. Deidentified smeared cytology slides stained with hematoxylin and eosin obtained from a fine needle aspirated from UM. Digital whole-slide images were generated by fine-needle aspiration biopsies of UM tumors that underwent GEP testing. A multistage DL system was developed with automatic region-of-interest (ROI) extraction from digital cytopathology images, an attention-based neural network, ROI feature aggregation, and slide-level data augmentation. The ability of our DL system in predicting GEP on a slide (patient) level. Data were partitioned at the patient level (73% training; 27% testing). In total, our study included 89 whole-slide images from 82 patients and 121 388 unique ROIs. The testing set included 24 slides from 24 patients (12 class 1 tumors; 12 class 2 tumors; 1 slide per patient). Our DL system for GEP prediction achieved an area under the receiver operating characteristic curve of 0.944, an accuracy of 91.7%, a sensitivity of 91.7%, and a specificity of 91.7% on a slide-level analysis. The incorporation of slide-level feature aggregation and data augmentation produced a more predictive DL model (P= 0.0031). Our current work established a complete pipeline for GEP prediction in UM tumors: from automatic ROI extraction from digital cytopathology whole-slide images to slide-level predictions. Our DL system demonstrated robust performance and, if validated prospectively, could serve as an image-based alternative to GEP testing.