High Inter-class Similarity Research Articles

Scene Classification Based on Heterogeneous Features of Multi-Source Data

Remote sensing scene classification is quite important in earth observation and other fields. Previous research has found that most of the existing models are based on deep learning models. However, the classification accuracy of the deep learning model is difficult to break through due to the challenges of difficulty distinguishing the socio-economic attributes of scenes, high interclass similarity, and large intraclass differences. To tackle the challenges, we propose a novel scene classification model that integrates heterogeneous features of multi-source data. Firstly, a multi-granularity feature learning module is designed, which can conduct uniform grid sampling of images to learn multi-granularity features. In this module, in addition to the features of our previous research, we also supplemented the socio-economic semantic features of the scene, and attention-based pooling is introduced to achieve different levels of representation of images. Then, to reduce the dimension of the feature, we adopt the feature-level fusion method. Next, the maxout-based module is designed to fuse the features of different granularity and extract the most distinguishing second-order latent ontology essence features. The weighted adaptive fusion method is used to fuse all the features. Finally, the Lie Group Fisher algorithm is used for scene classification. Extensive experimentation and evaluations show that our proposed model can find better solutions to the above challenges.

A survey of fine-grained visual categorization based on deep learning

Deep learning has achieved excellent results in various tasks in the field of computer vision, especially in fine-grained visual categorization. It aims to distinguish the subordinate categories of the label-level categories. Due to high intra-class variances and high inter-class similarity, the fine-grained visual categorization is extremely challenging. This paper first briefly introduces and analyzes the related public datasets. After that, some of the latest methods are reviewed. Based on the feature types, the feature processing methods, and the overall structure used in the model, we divide them into three types of methods: methods based on general convolutional neural network (CNN) and strong supervision of parts, methods based on single feature processing, and methods based on multiple feature processing. Most methods of the first type have a relatively simple structure, which is the result of the initial research. The methods of the other two types include models that have special structures and training processes, which are helpful to obtain discriminative features. We conduct a specific analysis on several methods with high accuracy on public datasets. In addition, we support that the focus of the future research is to solve the demand of existing methods for the large amount of the data and the computing power. In terms of technology, the extraction of the subtle feature information with the burgeoning vision transformer (ViT) network is also an important research direction.

Open Set Domain Adaptation via Instance Affinity Metric and Fine-Grained Alignment for Remote Sensing Scene Classification

In the practical application of remote sensing scene classification (RSSC), domain adaptation is introduced to handle the situation where the distribution of training data (source) and test data (target) is different. Compared to general domain adaptation, open set domain adaptation (OSDA) is suitable for more realistic situations where there are additional unknown classes in the target domain. The key to solving this problem is to separate unknown samples from target data to avoid negative transfer caused by mismatching unknown/known samples. In this letter, we propose a novel and effective method, named Instance Affinity metric-based Fine-grained Adaptation Network (IAFAN), for OSDA in RSSC. Concretely, an Unknown Sample Separation (USS) mechanism based on the instance affinity-aware matrix is pioneeringly proposed to endow the model with the ability to distinguish unknown samples. In addition, considering the high inter-class similarity and rich intra-class diversity of remote sensing images, we introduce the Sample Discriminability Enhancement (SDE) loss to further increase the inter-class distance and narrow the intra-class difference, thereby alleviating the negative transfer caused by sample misclassification and mismatching. In the feature confusion stage, we specially design Mask-mmd for OSDA as an adaptation metric to conduct semantic fine-grained cross-domain alignment of known samples while keeping unknown samples out of alignment, which avoids negative transfer during the adaptation process. Finally, we evaluate IAFAN on transfer tasks between different public remote sensing datasets, and the results verify that our method significantly outperforms previous methods in RSSC.

Automated optical inspection of FAST’s reflector surface using drones and computer vision

The Five-hundred-meter Aperture Spherical radio Telescope (FAST) is the world ’ s largest single-dish radio telescope. Its large reflecting surface achieves unprecedented sensitivity but is prone to damage, such as dents and holes, caused by naturally-occurring falling objects. Hence, the timely and accurate detection of surface defects is crucial for FAST’s stable operation. Conventional manual inspection involves human inspectors climbing up and examining the large surface visually, a time-consuming and potentially unreliable process. To accelerate the inspection process and increase its accuracy, this work makes the first step towards automating the inspection of FAST by integrating deep-learning techniques with drone technology. First, a drone flies over the surface along a predetermined route. Since surface defects significantly vary in scale and show high inter-class similarity, directly applying existing deep detectors to detect defects on the drone imagery is highly prone to missing and misidentifying defects. As a remedy, we introduce cross-fusion, a dedicated plug-in operation for deep detectors that enables the adaptive fusion of multi-level features in a point-wise selective fashion, depending on local defect patterns. Consequently, strong semantics and fine-grained details are dynamically fused at different positions to support the accurate detection of defects of various scales and types. Our AI-powered drone-based automated inspection is time-efficient, reliable, and has good accessibility, which guarantees the long-term and stable operation of FAST.

Multiform Ensemble Self-Supervised Learning for Few-Shot Remote Sensing Scene Classification

Self-supervised learning is an effective way to solve model collapse for few-shot remote sensing scene classification (FSRSSC). However, most self-supervised contrastive learning auxiliary tasks perform poorly on the high interclass similarity problem in FSRSSC. Furthermore, it is time-consuming and computationally expensive to obtain the best combination among numerous self-supervised auxiliary tasks. In practical applications, we may encounter difficulties in remote sensing data acquisition and labeling, while most FSRSSC studies only focus on the former. To alleviate the above problems, we propose a multiform ensemble self-supervised learning (MES2L) framework for FSRSSC in this article. Based on the transfer learning-based few-shot scheme, we design a novel global–local contrastive learning auxiliary task to solve the low interclass separability problem. The self-attention mechanism is designed in the local contrast features to investigate the intrinsic associations between different remote sensing scene objectives. We also present a multiform ensemble enhancement (MEE) training method. Ensemble enhancement involves the concatenation of features extracted from different backbones trained by a combination of multiform self-supervised auxiliary tasks. MEE can not only be regarded as a more straightforward alternative to knowledge distillation but also can achieve an effective compromise between expensive computational cost and classification accuracy. In addition, we provide two scene classification schemes of inductive and transductive settings, corresponding to solving the difficulties of remote sensing data acquisition and labeling. The proposed network achieves state-of-the-art results on three benchmark FSRSSC datasets. The potential of the MES2L framework is also demonstrated in combination with classical metalearning-based and metric learning-based few-shot algorithms.

Extracting feature fusion and co-saliency clusters using transfer learning techniques for improving remote sensing scene classification

To attribute the semantics to land cover, scene classification of very high-resolution (VHR) imagery comprises many possible applications in diverse domains. Conventional remote sensing image classification techniques have not addressed real application requirements. Deep convolutional neural networks (CNNs) have recently demonstrated competitive performance owing to their strong feature extraction abilities. These approaches primarily rely on semantic information to increase classification performance. The network's difficulty in correctly classifying scene images with comparable structures and high interclass similarity and getting features from different domains for the same scene is significant to achieve high classification accuracy. Thus, this study proposes a VHR remote sensing image classification model based on the discriminant correlation analysis (DCA) fusion of transfer deep CNNs and co-saliency features. First, the global feature was extracted from the original VHR image based on a pre-trained EfficientNet-V2L CNN. Second, the co-saliency feature was extracted from the co-saliency image to distinguish between similar classes. Third, DCA deep feature fusion was used with global and co-saliency features to save time and obtain robust features representing the scene. DCA reduces the length of the feature vector and focuses on the differences between VHR images in different classes. Finally, a multilayer perceptron (MLP) is used to classify the image. The effectiveness of the proposed method was verified using five benchmark remote sensing datasets: the 30-class aerial image dataset (AID), 21-class UC Merced, 38-class PatternNet, 19-class WHU-RS19, and 13-class KSA. The results demonstrate that the proposed model significantly improves performance compared with other CNN-based scene classification models. In addition, ablation studies were conducted to prove the effectiveness of the co-saliency feature and DCA fusion algorithm.

A Dual Self-Attention mechanism for vehicle re-Identification

Vehicle re-identification has attracted tremendous attention from computer vision communities for its extensive applications in intelligent transportation and public security, while the high inter-class similarity and the large intra-class difference between vehicles bring out great challenges for re-identification (re-ID). To tackle these challenges, we learn from the self-attention mechanism in Natural Language Processing and propose a dual self-attention module to learn different regional dependencies: static self-attention for selectively refining semantic features and dynamic self-attention (called cross-region attention) for enhancing the spatial awareness of local feature. The static self-attention refines attended pixels within the entire image and salient regions, while the cross-region attention creatively captures the position-related regional dependencies for pixels within the windscreen area. These attention modules capture long-range dependencies and relative position information between different pixels or regions for vehicle feature learning globally and locally, realizing an efficient vehicle feature embedding by concatenating these augmented features for vehicle re-ID. Extensive experiments demonstrate the effectiveness and promising performance of our approach against the state-of-the-arts.

Prior Attention Network for Multi-Lesion Segmentation in Medical Images.

The accurate segmentation of multiple types of lesions from adjacent tissues in medical images is significant in clinical practice. Convolutional neural networks (CNNs) based on the coarse-to-fine strategy have been widely used in this field. However, multi-lesion segmentation remains to be challenging due to the uncertainty in size, contrast, and high interclass similarity of tissues. In addition, the commonly adopted cascaded strategy is rather demanding in terms of hardware, which limits the potential of clinical deployment. To address the problems above, we propose a novel Prior Attention Network (PANet) that follows the coarse-to-fine strategy to perform multi-lesion segmentation in medical images. The proposed network achieves the two steps of segmentation in a single network by inserting a lesion-related spatial attention mechanism in the network. Further, we also propose the intermediate supervision strategy for generating lesion-related attention to acquire the regions of interest (ROIs), which accelerates the convergence and obviously improves the segmentation performance. We have investigated the proposed segmentation framework in two applications: 2D segmentation of multiple lung infections in lung CT slices and 3D segmentation of multiple lesions in brain MRIs. Experimental results show that in both 2D and 3D segmentation tasks our proposed network achieves better performance with less computational cost compared with cascaded networks. The proposed network can be regarded as a universal solution to multi-lesion segmentation in both 2D and 3D tasks. The source code is available at https://github.com/hsiangyuzhao/PANet.

MSLF-Net: A Multi-Scale and Multi-Level Feature Fusion Net for Diabetic Retinopathy Segmentation.

Diabetic Retinopathy (DR) is a diabetic complication that predisposes patients to visual impairments that could lead to blindness. Lesion segmentation using deep learning algorithms is an effective measure to screen and prevent early DR. However, there are several types of DR with varying sizes and high inter-class similarity, making segmentation difficult. In this paper, we propose a supervised segmentation method (MSLF-Net) based on multi-scale-multi-level feature fusion to achieve accurate end-to-end DR lesion segmentation. MSLF-Net builds a Multi-Scale Feature Extraction (MSFE) module to extract multi-scale information and provide more comprehensive features for segmentation. This paper further introduces the Multi-Level Feature Fusion (MLFF) module to improve feature fusion using a cross-layer structure. This structure only fuses low- and high-level features of the same class based on category supervision, avoiding feature contamination. Moreover, this paper produces additional masked images for the dataset and performs image enhancement operations to ensure that the proposed method is trainable and functional on small datasets. The extensive experiments are conducted on public datasets IDRID and e_ophtha. The results showed that our proposed feature enhancement method can perform feature fusion more effectively. Therefore, In the end-to-end DR segmentation neural network model, MSLF Net is superior to other similar models in segmentation, and can effectively improve the DR lesion segmentation performance.

A Retail Object Classification Method Using Multiple Cameras for Vision-Based Unmanned Kiosks

Several unmanned retail stores have been introduced with the development of sensors, wireless communication, and computer vision technologies. A vision-based kiosk that is only equipped with a vision sensor has significant advantages such as compactness and low implementation cost. Using convolutional neural network (CNN)-based object detectors, the kiosk recognizes an object when a customer picks up a product. In retail object recognition, the key challenge is the limited number of detections and high interclass similarity. In this study, these challenges are addressed by utilizing the “view-specific” feature of an object; specifically, an object class is divided into multiple “view-based” subclasses, and the object detectors are trained using these data. Further, the “view-aware feature” is defined by aggregating subclass detection results from multiple cameras. A superclass classifier predicts a superclass by utilizing an informative subclass detection result that distinguishes the target object from other similar-looking objects. To verify the effectiveness of the proposed approach, a prototype of the vision-based unmanned kiosk system is implemented. Experimental results indicate that the proposed method outperforms the conventional method, even on a state-of-the-art detection network. The dataset used in this study has been subsequently provided in the IEEE DataPort for reproducibility.

Deep attentive convolutional neural network for automatic grading of imbalanced diabetic retinopathy in retinal fundus images.

Automated fine-grained diabetic retinopathy (DR) grading was of great significance for assisting ophthalmologists in monitoring DR and designing tailored treatments for patients. Nevertheless, it is a challenging task as a result of high intra-class variations, high inter-class similarities, small lesions, and imbalanced data distributions. The pivotal factor for the success in fine-grained DR grading is to discern more subtle associated lesion features, such as microaneurysms (MA), Hemorrhages (HM), soft exudates (SE), and hard exudates (HE). In this paper, we constructed a simple yet effective deep attentive convolutional neural network (DACNN) for DR grading and lesion discovery with only image-wise supervision. Designed as a top-down architecture, our model incorporated stochastic atrous spatial pyramid pooling (sASPP), global attention mechanism (GAM), category attention mechanism (CAM), and learnable connected module (LCM) to better extract lesion-related features and maximize the DR grading performance. To be concrete, we devised sASPP combining randomness with atrous spatial pyramid pooling (ASPP) to accommodate the various scales of the lesions and struggle against the co-adaptation of multiple atrous convolutions. Then, GAM was introduced to extract class-agnostic global attention feature details, whilst CAM was explored for seeking class-specific distinctive region-level lesion feature information and regarding each DR severity grade in an equal way, which tackled the problem of imbalance DR data distributions. Further, the LCM was designed to automatically and adaptively search the optimal connections among layers for better extracting detailed small lesion feature representations. The proposed approach obtained high accuracy of 88.0% and kappa score of 88.6% for multi-class DR grading task on the EyePACS dataset, respectively, while 98.5% AUC, 93.8% accuracy, 87.9% kappa, 90.7% recall, 94.6% precision, and 92.6% F1-score for referral and non-referral classification on the Messidor dataset. Extensive experimental results on three challenging benchmarks demonstrated that the proposed approach achieved competitive performance in DR grading and lesion discovery using retinal fundus images compared with existing cutting-edge methods, and had good generalization capacity for unseen DR datasets. These promising results highlighted its potential as an efficient and reliable tool to assist ophthalmologists in large-scale DR screening.

Deep Residual Vector Encoding for Vein Recognition

Vein recognition has been drawing more attention recently because it is highly secure and reliable for practical biometric applications. However, underlying issues such as uneven illumination, low contrast, and sparse patterns with high inter-class similarities make the traditional vein recognition systems based on hand-engineered features unreliable. Recent successes of convolutional neural networks (CNNs) for large-scale image recognition tasks motivate us to replace the traditional hand-engineered features with the superior CNN to design a robust and discriminative vein recognition system. To address the difficulty of direct training or fine-tuning of a CNN with existing small-scale vein databases, a new knowledge transfer approach is formulated using pre-trained CNN models together with a training dataset (e.g., ImageNet) as a robust descriptor generation machine. With the generated deep residual descriptors, a very discriminative model, namely deep residual vector encoding (DRVE), is proposed by a hierarchical design of dictionary learning, coding, and classifier training procedures. Rigorous experiments are conducted with a high-quality hand-dorsa vein database, and superior recognition results compared with state-of-the-art models fully demonstrate the effectiveness of the proposed models. An additional experiment with the PolyU multispectral palmprint database is designed to illustrate the generalization ability.

Feature enhancement network for CNV typing in optical coherence tomography images

Objective. Choroidal neovascularization (CNV) is a characteristic feature of wet age-related macular degeneration, which is one of the main causes of blindness in the elderly. Automatic classification of CNV in optical coherence tomography images plays an auxiliary role in the clinical treatment of CNV. Approach. This study proposes a feature enhancement network (FE-net) to discriminate between different CNV types with high inter-class similarity. The FE-net consists of two branches: discriminative FE and diverse FE. In the discriminative FE branch, a novel class-specific feature extraction module is introduced to learn class-specific features, and the discriminative loss is introduced to make the learned features more discriminative. In the diverse FE branch, the attention region selection is used to mine the multi-attention features from feature maps in the same class, and the diverse loss is introduced to guarantee that the attention features are different, which can improve the diversity of the learned features. Main results. Experiments were conducted on our CNV dataset, with significant accuracy of 92.33%, 87.45%, 90.10%, and 91.25% on ACC, AUC, SEN, and SPE, respectively. Significance. These results demonstrate that the proposed method can effectively learn the discriminative and diverse features to discriminate subtle differences between different types of CNV. And accurate classification of CNV plays an auxiliary role in clinical treatmen.

Mask-Aware Semi-Supervised Object Detection in Floor Plans

Research has been growing on object detection using semi-supervised methods in past few years. We examine the intersection of these two areas for floor-plan objects to promote the research objective of detecting more accurate objects with less labeled data. The floor-plan objects include different furniture items with multiple types of the same class, and this high inter-class similarity impacts the performance of prior methods. In this paper, we present Mask R-CNN-based semi-supervised approach that provides pixel-to-pixel alignment to generate individual annotation masks for each class to mine the inter-class similarity. The semi-supervised approach has a student–teacher network that pulls information from the teacher network and feeds it to the student network. The teacher network uses unlabeled data to form pseudo-boxes, and the student network uses both label data with the pseudo boxes and labeled data as the ground truth for training. It learns representations of furniture items by combining labeled and label data. On the Mask R-CNN detector with ResNet-101 backbone network, the proposed approach achieves a mAP of 98.8%, 99.7%, and 99.8% with only 1%, 5% and 10% labeled data, respectively. Our experiment affirms the efficiency of the proposed approach, as it outperforms the previous semi-supervised approaches using only 1% of the labels.

Transformer based on channel-spatial attention for accurate classification of scenes in remote sensing image

Recently, the scenes in large high-resolution remote sensing (HRRS) datasets have been classified using convolutional neural network (CNN)-based methods. Such methods are well-suited for spatial feature extraction and can classify images with relatively high accuracy. However, CNNs do not adequately learn the long-distance dependencies between images and features in image processing, despite this being necessary for HRRS image processing as the semantic content of the scenes in these images is closely related to their spatial relationship. CNNs also have limitations in solving problems related to large intra-class differences and high inter-class similarity. To overcome these challenges, in this study we combine the channel-spatial attention (CSA) mechanism with the Vision Transformer method to propose an effective HRRS image scene classification framework using Channel-Spatial Attention Transformers (CSAT). The proposed model extracts the channel and spatial features of HRRS images using CSA and the Multi-head Self-Attention (MSA) mechanism in the transformer module. First, the HRRS image is mapped into a series of multiple planar 2D patch vectors after passing to the CSA. Second, the ordered vector is obtained via the linear transformation of each vector, and the position and learnable embedding vectors are added to the sequence vector to capture the inter-feature dependencies at a distance from the generated image. Next, we use MSA to extract image features and the residual network structure to complete the encoder construction to solve the gradient disappearance problem and avoid overfitting. Finally, a multi-layer perceptron is used to classify the scenes in the HRRS images. The CSAT network is evaluated using three public remote sensing scene image datasets: UC-Merced, AID, and NWPU-RESISC45. The experimental results show that the proposed CSAT network outperforms a selection of state-of-the-art methods in terms of scene classification.

Multiple Thermal Sensor Array Fusion Toward Enabling Privacy-Preserving Human Monitoring Applications

Human-centric applications of a single thermal sensor array (TSA) have performed extremely well in many areas. However, most of these works have not yet reached the real applicability stage of the Internet of Things (IoT) applications. The main limitation of deploying such systems on a large scale is the challenge of fusing multiple TSAs to cover a wide inspection area, e.g., smart homes, hospitals, and many other domestic environments. On the other hand, objects that appear in the low-resolution thermal images acquired from TSA have low intraclass variations and high interclass similarities, making the identification of the overlapping regions through matching a comparable template image in multiple images very difficult. This article proposes a motion-based approach to fuse multiple TSAs and learn the domestic environment layout to enable further human-centred IoT applications to run in the cloud. Besides, a privacy improvement on utilizing these sensors in IoT applications is proposed. The proposed approach is evaluated with comprehensive experiments on different sensor placements and domestic environment conditions. This article shows an average performance of 92.5% accuracy using various machine learning techniques and use case scenarios.

AI-CenterNet CXR: An artificial intelligence (AI) enabled system for localization and classification of chest X-ray disease.

Machine learning techniques have lately attracted a lot of attention for their potential to execute expert-level clinical tasks, notably in the area of medical image analysis. Chest radiography is one of the most often utilized diagnostic imaging modalities in medical practice, and it necessitates timely coverage regarding the presence of probable abnormalities and disease diagnoses in the images. Computer-aided solutions for the identification of chest illness using chest radiography are being developed in medical imaging research. However, accurate localization and categorization of specific disorders in chest X-ray images is still a challenging problem due to the complex nature of radiographs, presence of different distortions, high inter-class similarities, and intra-class variations in abnormalities. In this work, we have presented an Artificial Intelligence (AI)-enabled fully automated approach using an end-to-end deep learning technique to improve the accuracy of thoracic illness diagnosis. We proposed AI-CenterNet CXR, a customized CenterNet model with an improved feature extraction network for the recognition of multi-label chest diseases. The enhanced backbone computes deep key points that improve the abnormality localization accuracy and, thus, overall disease classification performance. Moreover, the proposed architecture is lightweight and computationally efficient in comparison to the original CenterNet model. We have performed extensive experimentation to validate the effectiveness of the proposed technique using the National Institutes of Health (NIH) Chest X-ray dataset. Our method achieved an overall Area Under the Curve (AUC) of 0.888 and an average IOU of 0.801 to detect and classify the eight types of chest abnormalities. Both the qualitative and quantitative findings reveal that the suggested approach outperforms the existing methods, indicating the efficacy of our approach.

Skin lesion classification based on the VGG‐16 fusion residual structure

AbstractThe analysis of skin lesion images is challenging due to the high interclass similarity and intraclass variance. Therefore, improving the ability to automatically classify based on skin lesion images is necessary to help physicians classify skin lesions. We propose a network model based on the Visual Geometry Group Network (VGG‐16) fusion residual structure for the multiclass classification of skin lesions. based on the VGG‐16 network, we simplify and improve the network structure by adding a preprocessing layer (CBRM layer) and fusing the residual structure. We also use a hair removal algorithm and perform six data augmentation operations on a small number of skin lesion images to balance the total number of the seven skin lesions in the dataset. The model was evaluated on the ISIC2018 dataset. Experiments have shown that our network model achieves good classification performance, with a test accuracy rate of 88.14% and a macroaverage of 98%.

Rethinking the Two-Stage Framework for Grounded Situation Recognition

Grounded Situation Recognition (GSR), i.e., recognizing the salient activity (or verb) category in an image (e.g.,buying) and detecting all corresponding semantic roles (e.g.,agent and goods), is an essential step towards “human-like” event understanding. Since each verb is associated with a specific set of semantic roles, all existing GSR methods resort to a two-stage framework: predicting the verb in the first stage and detecting the semantic roles in the second stage. However, there are obvious drawbacks in both stages: 1) The widely-used cross-entropy (XE) loss for object recognition is insufficient in verb classification due to the large intra-class variation and high inter-class similarity among daily activities. 2) All semantic roles are detected in an autoregressive manner, which fails to model the complex semantic relations between different roles. To this end, we propose a novel SituFormerfor GSR which consists of a Coarse-to-Fine Verb Model (CFVM) and a Transformer-based Noun Model (TNM). CFVM is a two-step verb prediction model: a coarse-grained model trained with XE loss first proposes a set of verb candidates, and then a fine-grained model trained with triplet loss re-ranks these candidates with enhanced verb features (not only separable but also discriminative). TNM is a transformer-based semantic role detection model, which detects all roles parallelly. Owing to the global relation modeling ability and flexibility of the transformer decoder, TNM can fully explore the statistical dependency of the roles. Extensive validations on the challenging SWiG benchmark show that SituFormer achieves a new state-of-the-art performance with significant gains under various metrics. Code is available at https://github.com/kellyiss/SituFormer.

Precision medical image hash retrieval by interpretability and feature fusion

Background and Objective: To address the problem of low accuracy of medical image retrieval due to high inter-class similarity and easy omission of lesions, a precision medical image hash retrieval method combining interpretability and feature fusion is proposed, taking chest X-ray images as an example.Methods: Firstly, the DenseNet-121 network is pre-trained on a large dataset of medical images without manual annotation using the comparison to learn (C2L) method to obtain a backbone network model containing more medical representations with training weights. Then, a global network is constructed by using global image learning to acquire an interpretable saliency map as attention mechanisms, which can generate a mask crop to get a local discriminant region. Thirdly, the local discriminant regions are used as local network inputs to obtain local features, and the global features are used with the local features by dimension in the pooling layer. Finally, a hash layer is added between the fully connected layer and the classification layer of the backbone network, defining classification loss, quantization loss and bit-balanced loss functions to generate high-quality hash codes. The final retrieval result is output by calculating the similarity metric of the hash codes.Results: Experiments on the Chest X-ray8 dataset demonstrate that our proposed interpretable saliency map can effectively locate focal regions, the fusion of features can avoid information omission, and the combination of three loss functions can generate more accurate hash codes. Compared with the current advanced medical image retrieval methods, this method can effectively improve the accuracy of medical image retrieval.Conclusions: The proposed hash retrieval approach combining interpretability and feature fusion can effectively improve the accuracy of medical image retrieval which can be potentially applied in computer-aided-diagnosis systems.