Saliency Map Research Articles

XRelevanceCAM: towards explainable tissue characterization with improved localisation of pathological structures in probe-based confocal laser endomicroscopy

PurposeProbe-based confocal laser endomicroscopy (pCLE) enables intraoperative tissue characterization with improved resection rates of brain tumours. Although a plethora of deep learning models have been developed for automating tissue characterization, their lack of transparency is a concern. To tackle this issue, techniques like Class Activation Map (CAM) and its variations highlight image regions related to model decisions. However, they often fall short of providing human-interpretable visual explanations for surgical decision support, primarily due to the shattered gradient problem or insufficient theoretical underpinning.MethodsIn this paper, we introduce XRelevanceCAM, an explanation method rooted in a better backpropagation approach, incorporating sensitivity and conservation axioms. This enhanced method offers greater theoretical foundation and effectively mitigates the shattered gradient issue when compared to other CAM variants.ResultsQualitative and quantitative evaluations are based on ex vivo pCLE data of brain tumours. XRelevanceCAM effectively highlights clinically relevant areas that characterize the tissue type. Specifically, it yields a remarkable 56% improvement over our closest baseline, RelevanceCAM, in the network’s shallowest layer as measured by the mean Intersection over Union (mIoU) metric based on ground-truth annotations (from 18 to 28.07%). Furthermore, a 6% improvement in mIoU is observed when generating the final saliency map from all network layers.ConclusionWe introduce a new CAM variation, XRelevanceCAM, for precise identification of clinically important structures in pCLE data. This can aid introperative decision support in brain tumour resection surgery, as validated in our performance study.

A visual attention-based algorithm for brain tumor detection using an on-center saliency map and a superpixel-based framework

Brain tumors are life-threatening and are typically identified by experts using imaging modalities like Magnetic Resonance Imaging (MRI), Computed Tomography (CT), and Positron Emission Tomography (PET). However, any error due to human intervention in brain anomaly detection can have devastating consequences. This study proposes a tumor detection algorithm for brain MRI images. Previous research into tumor detection has drawbacks, paving the way for further investigations. A visual attention-based technique for tumor detection is proposed to overcome these drawbacks. Brain tumors have a wide range of intensity, varying from inner matter-alike intensity to skull-alike intensity, making them difficult to threshold. Thus, a unique approach to threshold using entropy has been utilized. An on-center saliency map accurately captures the biological visual attention-focused tumorous region from the original image. Later, a superpixel-based framework has been proposed and used to capture the true structure of the tumor. Finally, it was experimentally shown that the proposed algorithm outperforms the existing algorithms for brain tumor detection.

SimPSI: A Simple Strategy to Preserve Spectral Information in Time Series Data Augmentation

Data augmentation is a crucial component in training neural networks to overcome the limitation imposed by data size, and several techniques have been studied for time series. Although these techniques are effective in certain tasks, they have yet to be generalized to time series benchmarks. We find that current data augmentation techniques ruin the core information contained within the frequency domain. To address this issue, we propose a simple strategy to preserve spectral information (SimPSI) in time series data augmentation. SimPSI preserves the spectral information by mixing the original and augmented input spectrum weighted by a preservation map, which indicates the importance score of each frequency. Specifically, our experimental contributions are to build three distinct preservation maps: magnitude spectrum, saliency map, and spectrum-preservative map. We apply SimPSI to various time series data augmentations and evaluate its effectiveness across a wide range of time series benchmarks. Our experimental results support that SimPSI considerably enhances the performance of time series data augmentations by preserving core spectral information. The source code used in the paper is available at https://github.com/Hyun-Ryu/simpsi.

Model-agnostic progressive saliency map generation for object detector

With the widespread adoption of object detection models across various industries, the interpretability of these detectors has become an important research topic. The interpretability of a detector helps humans understand which areas significantly contribute to the model's decision. Furthermore, the interpretability enhances the credibility of detectors and helps identify their strengths and weaknesses. Due to the ability to provide intuitive explanations of models, the saliency map has been widely employed in the field of interpreting deep models. Model-agnostic interpretability methods are more general approaches as they treat the model as a black box without considering its internal complexity structure. However, existing model-agnostic interpretability methods often introduce “noise” into saliency maps by applying random masking and fixed masking granularity. This noise reduces the quality and interpretability of the generated saliency maps. To address this challenge and obtain more interpretable saliency maps for object detection models, this paper proposes a model-agnostic progressive saliency map generation method based on a hierarchical framework called MAPSM. In MAPSM, an adaptive masking partition mechanism is introduced to adapt the masking granularity to different object sizes. Additionally, MAPSM employs a saliency-driven mask generation strategy to effectively reduce the “noise”. Utilizing a hierarchical framework, MAPSM progressively discovers and refines the saliency areas of objects, resulting in more interpretable saliency maps. To evaluate the quality of the saliency maps generated by MAPSM, we compare it with other methods in multiple metrics. Experimental results demonstrate that our method produces saliency maps with better quality and interpretability.

Auxiliary Tasks Enhanced Dual-Affinity Learning for Weakly Supervised Semantic Segmentation.

Most existing weakly supervised semantic segmentation (WSSS) methods rely on class activation mapping (CAM) to extract coarse class-specific localization maps using image-level labels. Prior works have commonly used an off-line heuristic thresholding process that combines the CAM maps with off-the-shelf saliency maps produced by a general pretrained saliency model to produce more accurate pseudo-segmentation labels. We propose AuxSegNet + , a weakly supervised auxiliary learning framework to explore the rich information from these saliency maps and the significant intertask correlation between saliency detection and semantic segmentation. In the proposed AuxSegNet + , saliency detection and multilabel image classification are used as auxiliary tasks to improve the primary task of semantic segmentation with only image-level ground-truth labels. We also propose a cross-task affinity learning mechanism to learn pixel-level affinities from the saliency and segmentation feature maps. In particular, we propose a cross-task dual-affinity learning module to learn both pairwise and unary affinities, which are used to enhance the task-specific features and predictions by aggregating both query-dependent and query-independent global context for both saliency detection and semantic segmentation. The learned cross-task pairwise affinity can also be used to refine and propagate CAM maps to provide better pseudo labels for both tasks. Iterative improvement of segmentation performance is enabled by cross-task affinity learning and pseudo-label updating. Extensive experiments demonstrate the effectiveness of the proposed approach with new state-of-the-art WSSS results on the challenging PASCAL VOC and MS COCO benchmarks.

Solution of wide and micro background bias in contrastive action representation learning

In recent years, contrastive learning has made great progress in the field of computer vision, which shows great potential in action representation learning. Current contrastive learning methods usually employ contrastive loss function to learn video motion representation, which attend to capturing the similar background appearance but ignore similar motion information. This problem is called background bias, which restricts the model from exploring motion patterns. The background bias can be divided into wide background bias and micro background bias. The wide background bias refers to the statistically significant background bias, while the micro background bias refers to the background bias directly interacting with the moving object. To tackle these problems, this paper first proposes a semi-negative pair merging foreground–background, which mainly decouples the dynamic factor with obvious motion and the static factor with stable invariants in the video frame sequence. Then the dynamic factor of the original video is fused with other static factors to obtain a random background image where the foreground is more similar than the background to solve the wide background bias. Secondly, a pixel-level motion aware representation decomposition module calculates the pixel-level intensity variations from the feature space of adjacent frames, and these variations are further accumulated to obtain the salient map that focuses on the boundaries of moving objects, so the model attends to the motion pattern rather than the background to solve the micro background bias. Furthermore, a new dual loss function is proposed based on joint wide and micro background bias to better capture both static and dynamic features. The heat map shows that the proposed method effectively can solve background bias and achieve better performance in public datasets compared with other existing methods.

Multivariate Upstream Kuroshio Transport (UKT) Prediction and Targeted Observation Sensitive Area Identification of UKT Seasonal Reduction

Variation and seasonal reduction in the Upstream Kuroshio Transport (UKT) have important impacts on surrounding climate and oceanic circulation systems. Therefore, reliable UKT prediction is crucial. In this paper, we propose an intelligent UKT prediction model, KuroshioNet, which is firstly pre-trained with simulation data generated by the Regional Ocean Modeling System (ROMS) and then fine-tuned with reanalysis data of the Simple Ocean Data Assimilation (SODA). Operating at a five-day time resolution and a 0.5°spatial resolution, KuroshioNet has the capability to predict multivariate fields associated with upstream Kuroshio, including 3D variables like velocity, temperature, as well as salinity and 2D variables like sea surface height. Subsequently, the UKT is computed from the predicted fields. We evaluate and analyze the experimental results, which show that KuroshioNet has a lead time of 55 days for UKT prediction. In order to enhance the physical interpretability of KuroshioNet, we conduct an ablation experiment to evaluate the impact of each predictor on prediction skill. It demonstrates that selecting zonal velocity, meridional velocity, temperature, salinity, and SSH contributes to UKT prediction by KuroshioNet. Besides, by analyzing model performance and visualizing what the convolutional kernels learn, we find that KuroshioNet, which has learned from ROMS data, is capable of obtaining better initial performance and acquiring more active kernels to better learn the features in SODA data. Furthermore, we identify the targeted observation sensitive area of UKT seasonal reduction by KuroshioNet with the saliency map method, which is situated to the east of upstream kuroshio. The sensitive area is consistent with the result identified by numerical models and yields 38.1% improvement on prediction demonstrated by observing system simulation experiments.

Semantic-Consistency-guided Learning on Deep Features for Unsupervised Salient Object Detection

Unsupervised salient object detection is an important task in many real-world scenarios where pixel-wise label information is of scarce availability. Despite its significance, this problem remains rarely explored, with a few works that consider unsupervised salient object detection methods based on the fused graph from the sum fusion of multiple deep feature similarity matrices. However, these methods ignore the interrelation of the low-level feature similarity matrices and the high-level semantic similarity matrice, which degrades the quality of the fused graph. In this article, we propose a semantic-consistency-guided multi-graph fusion learning algorithm for unsupervised saliency detection, where the consistency and inconsistency between multiple low-level feature similarity matrices and the high-level semantic similarity matrice are explored to promote the robustness and quality of the fused graph. In the first stage, a semantic-consistency-guided multi-graph fusion learning method is proposed to exploit consistency and inconsistency of multiple low-level deep features and the high-level semantic feature. The semantic-consistency-guided similarity matrices are computed for preliminary saliency ranking. In the following saliency refinement stage, the semantic-enhanced similarity matrices are built by the cross diffusion to fuse the multiple low-level deep features and the high semantic deep feature. Based on the semantic-enhanced similarity matrices, the refinement saliency maps are calculated in a semantic-enhanced cellular automata manner. Furthermore, the final ensemble stage of the large margin semi-supervised classification views the preliminary ranking results and refinement results as features, adopts the large margin graphs for saliency ensemble. Extensive evaluations over four benchmark datasets show that the proposed unsupervised method performs favorably against the state-of-the-art approaches and is competitive with some supervised deep learning-based methods.

PATNet: Patch-to-pixel attention-aware transformer network for RGB-D and RGB-T salient object detection

Multimodal salient object detection (SOD) combines different modal images to generate the most visually appealing saliency map. When fusing multimodal and multiscale features, maintaining the integrity and fine granularity of the target is critical for improving the performance of multimodal SOD. The fine-grained information differences between the modalities and the size of the features in the transformer prevent most existing studies from guaranteeing both granularities. Therefore, we propose a patch-to-pixel attention-aware transformer network (PATNet) to overcome these problems, whereby the integrity and fine-grained details of the saliency map are preserved by employing a decision-transformation strategy to map global patches onto local pixels. Specifically, PATNet consists of the shared attention fusion module (SAFM), adjacent modeling fusion module (AMFM), and fine-grained mapping module (FMM). SAFM enhances the consistency between multimodal features through a shared attention matrix and an identical convolutional feed-forward network. Meanwhile, AMFM enhances low-resolution features by modeling neighboring features to avoid the aliasing effect of upsampling. In the output stage, FMM is responsible for mapping the feature maps represented by patches onto pixels and restoring the salient object details. Numerous experimental results demonstrate that PATNet outperforms 24 state-of-the-art methods on six RGB-D and three RGB-T datasets. The source code is publicly available at https://github.com/LitterMa-820/PATNet.

Video saliency-recognition by applying custom spatio temporal fusion technique

<span lang="EN-US">Video saliency detection is a major growing field with quite few contributions to it. The general method available today is to conduct frame wise saliency detection and this leads to several complications, including an incoherent pixel-based saliency map, making it not so useful. This paper provides a novel solution to saliency detection and mapping with its custom spatio-temporal fusion method that uses frame wise overall motion colour saliency along with pixel-based consistent spatio-temporal diffusion for its temporal uniformity. In the proposed method section, it has been discussed how the video is fragmented into groups of frames and each frame undergoes diffusion and integration in a temporary fashion for the colour saliency mapping to be computed. Then the inter group frame are used to format the pixel-based saliency fusion, after which the features, that is, fusion of pixel saliency and colour information, guide the diffusion of the spatio temporal saliency. With this, the result has been tested with 5 publicly available global saliency evaluation metrics and it comes to conclusion that the proposed algorithm performs better than several state-of-the-art saliency detection methods with increase in accuracy with a good value margin. All the results display the robustness, reliability, versatility and accuracy.</span>

DMT-EV: An Explainable Deep Network for Dimension Reduction.

Dimension reduction (DR) is commonly utilized to capture the intrinsic structure and transform high-dimensional data into low-dimensional space while retaining meaningful properties of the original data. It is used in various applications, such as image recognition, single-cell sequencing analysis, and biomarker discovery. However, contemporary parametric-free and parametric DR techniques suffer from several significant shortcomings, such as the inability to preserve global and local features and the poor generalisation performance. On the other hand, regarding explainability, it is crucial to comprehend the embedding process, especially the contribution of each part to the embedding process, while understanding how each feature affects the embedding results that identify critical components and help diagnose the embedding process. To address these problems, we have developed a deep neural network method called DMT-EV, which provides not only excellent performance in structural maintainability but also explainability to the DR therein. DMT-EV starts with data augmentation and a manifold-based loss function to improve embedding performance. The explanation is based on saliency maps and aims to examine the trained DMT-EV parameters and contributions of components during the embedding process. The proposed techniques are integrated with a visual interface to help the user to adjust DMT-EV to achieve better DR performance and explainability. The interactive visual interface makes it easier to illustrate the data features, compare different DR techniques, and investigate DR. An in-depth experimental comparison shows that DMT-EV consistently outperforms the state-of-the-art methods in both performance measures and explainability.

An Interpretable Machine Learning Model for Heart Arrhythmia Classification

Machine learning (ML) has been a very effective tool for arrhythmia detection and classification using electrocardiograms (ECGs). However, in order for patients and healthcare professionals to trust the ML models, the models have to be interpretable to show how they arrived at a certain conclusion. We present an ML model that uses the Deep k-Nearest Neighbors framework in order to produce example-based explanations and uncertainty estimations. These examples are ECGs similar to the input derived from conducting a nearest neighbor search on encoded samples (which are values of a layer of the neural network after passing dataset samples through it). We introduce a new technique of using these example-based explanations in conjunction with saliency maps, and also use a neighbor-based uncertainty estimation technique. We show that the saliency maps provide good explanations, but the neighbor examples are needed to assess the credibility of those saliency maps. Our uncertainty estimations increase the accuracy of the model from 86% to 93% (when measured with coverage of 76%). Overall, our novel methods prove to be a promising solution in the field of interpretable ML for arrhythmia classification.

Comparison of clinical geneticist and computer visual attention in assessing genetic conditions.

Artificial intelligence (AI) for facial diagnostics is increasingly used in the genetics clinic to evaluate patients with potential genetic conditions. Current approaches focus on one type of AI called Deep Learning (DL). While DL- based facial diagnostic platforms have a high accuracy rate for many conditions, less is understood about how this technology assesses and classifies (categorizes) images, and how this compares to humans. To compare human and computer attention, we performed eye-tracking analyses of geneticist clinicians (n = 22) and non-clinicians (n = 22) who viewed images of people with 10 different genetic conditions, as well as images of unaffected individuals. We calculated the Intersection-over-Union (IoU) and Kullback-Leibler divergence (KL) to compare the visual attentions of the two participant groups, and then the clinician group against the saliency maps of our deep learning classifier. We found that human visual attention differs greatly from DL model's saliency results. Averaging over all the test images, IoU and KL metric for the successful (accurate) clinician visual attentions versus the saliency maps were 0.15 and 11.15, respectively. Individuals also tend to have a specific pattern of image inspection, and clinicians demonstrate different visual attention patterns than non-clinicians (IoU and KL of clinicians versus non-clinicians were 0.47 and 2.73, respectively). This study shows that humans (at different levels of expertise) and a computer vision model examine images differently. Understanding these differences can improve the design and use of AI tools, and lead to more meaningful interactions between clinicians and AI technologies.

Multi‐exposure embeddings for graph learning: Towards high dynamic range image saliency prediction

AbstractIdentifying saliency in high dynamic range (HDR) images is a fundamentally important issue in HDR imaging, and plays critical roles towards comprehensive scene understanding. Most of existing studies leverage hand‐crafted features for HDR image saliency prediction, lacking the capabilities of fully exploiting the characteristics of HDR image (i.e. wider luminance range and richer colour gamut). Here, systematical studies are carried out on HDR image saliency prediction by proposing a new framework to single out the contributions from multi‐exposure images. Specifically, inspired by the mechanism of HDR imaging, the method first utilizes graph neural networks to model the relations among multi‐exposure images and the tone‐mapped image obtained from an HDR image, enabling more discriminative saliency‐related feature representations. Subsequently, the saliency features driven by global semantic knowledge are aggregated from the tone‐mapped image through enhancing global context‐aware semantic information. Finally, a fusion module is designed to integrate saliency‐oriented feature representations originated from multi‐exposure images and the tone‐mapped image, producing the saliency maps of HDR images. Moreover, a new challenging HDR eye fixation database (HDR‐EYEFix) is created, expecting to further contribute the research on HDR image saliency prediction. Experiment results show that the method obtains superior performance compared to the state‐of‐the‐art methods.

Exploiting macro- and micro-structural brain changes for improved Parkinson’s disease classification from MRI data

Parkinson’s disease (PD) is the second most common neurodegenerative disease. Accurate PD diagnosis is crucial for effective treatment and prognosis but can be challenging, especially at early disease stages. This study aimed to develop and evaluate an explainable deep learning model for PD classification from multimodal neuroimaging data. The model was trained using one of the largest collections of T1-weighted and diffusion-tensor magnetic resonance imaging (MRI) datasets. A total of 1264 datasets from eight different studies were collected, including 611 PD patients and 653 healthy controls (HC). These datasets were pre-processed and non-linearly registered to the MNI PD25 atlas. Six imaging maps describing the macro- and micro-structural integrity of brain tissues complemented with age and sex parameters were used to train a convolutional neural network (CNN) to classify PD/HC subjects. Explainability of the model’s decision-making was achieved using SmoothGrad saliency maps, highlighting important brain regions. The CNN was trained using a 75%/10%/15% train/validation/test split stratified by diagnosis, sex, age, and study, achieving a ROC-AUC of 0.89, accuracy of 80.8%, specificity of 82.4%, and sensitivity of 79.1% on the test set. Saliency maps revealed that diffusion tensor imaging data, especially fractional anisotropy, was more important for the classification than T1-weighted data, highlighting subcortical regions such as the brainstem, thalamus, amygdala, hippocampus, and cortical areas. The proposed model, trained on a large multimodal MRI database, can classify PD patients and HC subjects with high accuracy and clinically reasonable explanations, suggesting that micro-structural brain changes play an essential role in the disease course.

Audio-visual saliency prediction with multisensory perception and integration

Audio-visual saliency prediction (AVSP) is a task that aims to model human attention patterns in the perception of auditory and visual scenes. Given the challenges associated with perceiving and combining multi-modal saliency features from videos, this paper presents a multi-sensory framework for AVSP. This framework is designed to extract audio, motion and image saliency features and integrate them effectively, which can then serve as a general architecture for the AVSP task. To obtain multi-sensory information, we develop a three-stream encoder that extracts audio, motion and image saliency features. In particular, we utilize a pre-trained encoder with knowledge related to image saliency to extract saliency features for each frame. The image saliency features are then incorporated with motion features using a spatial attention module. For motion features, 3D convolutional neural networks (CNNs) like S3D are commonly used in AVSP models. However, these networks are unable to effectively capture the global motion relationship in videos. To tackle this problem, we incorporate Transformer- and MLP-based motion encoders into the AVSP models. To learn joint audio-visual representations, an audio-visual fusion block is exploited to enhance the correlation between audio and visual motion features under the supervision of a cosine similarity loss in a self-supervised manner. Finally, a multi-stage decoder integrates audio, motion and image saliency features to generate the final saliency map. We evaluate our methods on six audio-visual eye-tracking datasets. Experimental results demonstrate that our method achieves compelling performance compared to the state-of-the-art methods. The source code is available at https://github.com/oraclefina/MSPI.

Adversarial Attacks with Defense Mechanisms on Convolutional Neural Networks and Recurrent Neural Networks for Malware Classification

In the field of behavioral detection, deep learning has been extensively utilized. For example, deep learning models have been utilized to detect and classify malware. Deep learning, however, has vulnerabilities that can be exploited with crafted inputs, resulting in malicious files being misclassified. Cyber-Physical Systems (CPS) may be compromised by malicious files, which can have catastrophic consequences. This paper presents a method for classifying Windows portable executables (PEs) using Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs). To generate malware executable adversarial examples of PE, we conduct two white-box attacks, Jacobian-based Saliency Map Attack (JSMA) and Carlini and Wagner attack (C&W). An adversarial payload was injected into the DOS header, and a section was added to the file to preserve the PE functionality. The attacks successfully evaded the CNN model with a 91% evasion rate, whereas the RNN model evaded attacks at an 84.6% rate. Two defense mechanisms based on distillation and training techniques are examined in this study for overcoming adversarial example challenges. Distillation and training against JSMA resulted in the highest reductions in the evasion rates of 48.1% and 41.49%, respectively. Distillation and training against C&W resulted in the highest decrease in evasion rates, at 48.1% and 49.9%, respectively.

EMNet: Edge-guided multi-level network for salient object detection in low-light images

Salient object detection (SOD) has achieved remarkable performance in well-lit scenes. However, when generalized to low-light scenes, the performance of SOD shows significant decrease owing to more challenging conditions such as weak brightness, low contrast, and poor signal-to-noise ratio. To address this issue, we propose a novel edge-guided and multi-level network (EMNet) for SOD in low light images, which learns robust multiscale region features by optimizing the boundaries of salient objects and employing a multi-stage cascaded strategy. To be more specific, the proposed Edge Feature Highlight (EFH) module can establish mapping relationships at different scales and fuse the outputs obtained from pairs of different branches for extracting accurate boundary information. Secondly, Multi-layer Feature Fusion (MFF) module is proposed for combining multi-scale deep features with salient edge cues, using stepwise fusion for effective integration of deep features. Finally, we employ a coarse-to-fine way for iterative prediction to generate high-quality saliency maps. We conducted comprehensive experiments on LLI dataset, and the results demonstrate that the proposed method achieves the best performance in terms of five evaluation metrics, where an average improvement of max Fβ, ωFβ, Em, Sm, and MAE outperforms state-of-the-art method by 7.20%, 12.06%, 4.62%, 5.00%, and 36.34%, respectively.

Separate first, then segment: An integrity segmentation network for salient object detection

Current methods aggregate multi-level features or introduce auxiliary information to get more refined saliency maps. However, little attention is paid to how to obtain complete salient objects in cluttered background. To address this problem, we propose an integrity segmentation network (ISNet) with a novel detection paradigm that first separates the targets completely and then segment them finely. Specifically, the ISNet architecture consists of a target separation (TS) branch and an object segmentation (OS) branch, trained using a hierarchical difference-aware (HDA) loss. The TS branch equipped with a fractal structure is utilized to produce saliency features with expanded boundary (SF w/ EB), which can enlarge the difference of border details to separate the target from background completely. Compared with the edge and skeleton information, the SF w/ EB contains a more complete structure, which can supplement the defect of salient objects. The OS branch is leveraged to generate complementary features, which gradually integrates the SF w/ EB and aggregated features to segment complete saliency maps. Moreover, we propose the HDA loss to further improve the structural integrity of prediction, which hierarchically assigns weight to pixels according to their differences. Hard pixels will be given more attention to discriminate the similar parts between foreground and background. Comprehensive experimental results on five datasets show that the proposed ISNet outperforms the state-of-the-art methods both quantitatively and qualitatively. Concretely, compared with three typical models, the average gain percentage reaches 2.6% in terms of Fβ, Sm and MAE on two large complex datasets. The improvements demonstrate that the proposed ISNet are beneficial for improving the integrity of prediction. Besides, the ISNet is efficient and runs at a real-time speed of 39.5 FPS when processing an image with size of 320 × 320. Furthermore, the proposed model has better generalization, which can also be applied to other vision tasks to handle complex scenes. Codes are available at https://github.com/lesonly/ISNet.

Category-Aware Saliency Enhance Learning Based on CLIP for Weakly Supervised Salient Object Detection

Weakly supervised salient object detection (SOD) using image-level category labels has been proposed to reduce the annotation cost of pixel-level labels. However, existing methods mostly train a classification network to generate a class activation map, which suffers from coarse localization and difficult pseudo-label updating. To address these issues, we propose a novel Category-aware Saliency Enhance Learning (CSEL) method based on contrastive vision-language pre-training (CLIP), which can perform image-text classification and pseudo-label updating simultaneously. Our proposed method transforms image-text classification into pixel-text matching and generates a category-aware saliency map, which is evaluated by the classification accuracy. Moreover, CSEL assesses the quality of the category-aware saliency map and the pseudo saliency map, and uses the quality confidence scores as weights to update the pseudo labels. The two maps mutually enhance each other to guide the pseudo saliency map in the correct direction. Our SOD network can be trained jointly under the supervision of the updated pseudo saliency maps. We test our model on various well-known RGB-D and RGB SOD datasets. Our model achieves an S-measure of 87.6%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document} on the RGB-D NLPR dataset and 84.3%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document} on the RGB ECSSD dataset. Additionally, we obtain satisfactory performance on the weakly supervised E-measure, F-measure, and mean absolute error metrics for other datasets. These results demonstrate the effectiveness of our model.