Instance-level Features Research Articles

Hybrid multiple instance learning network for weakly supervised medical image classification and localization

Weakly supervised medical image analysis is of great significance for computer-aided diagnosis due to the difficulty in obtaining accurately labeled medical data. In this paper, we proposed a new Multi-instance Learning (MIL) framework called HybridMIL integrating CNN Convolutional Neural Networks (CNN) and Broad Learning Systems (BLS). Our HybridMIL can overcome several challenging issues over existing MIL methods based on either CNN or BLS alone: (i) Multiple levels (i.e., different resolutions) of feature information can be simultaneously extracted through a newly proposed instance-level feature enhancement (IFE) module; (ii) Global-level semantic information contained in the deep layers can be better represented under the global-level semantic enhancement (GSE) module; (iii) Hybrid feature fusion (HFF) module is newly designed to effectively fuse and align the multi-level outputs of IFE and global-level semantic information of GSE for subsequent classification and localization tasks. The proposed HybridMIL is evaluated on various public medical and MIL benchmark datasets. The results indicate that HybridMIL surpasses other recent MIL models in terms of classification and localization performance by up to 8.5% and 9.0%, respectively. Lastly, we demonstrate the highly competitive performance of HybridMIL in general MIL problems, going beyond weakly supervised medical image analysis.

Triplet Contrastive Representation Learning for Unsupervised Vehicle Re-identification

Part feature learning plays a crucial role in achieving fine-grained semantic understanding in unsupervised vehicle re-identification. However, existing approaches directly model part and global features, which can easily lead to severe gradient vanishing issues due to their unequal feature information and unreliable pseudo-labels. To address this problem, in this paper, we propose a triplet contrastive representation learning (TCRL) framework, which leverages cluster features to bridge the part features and global features for unsupervised vehicle re-identification. Specifically, TCRL devises three memory banks to store the instance/cluster features and proposes a proxy contrastive loss (PCL) to make contrastive learning between adjacent memory banks, thus presenting the associations between the part and global features as a transition of the part-cluster and cluster-global associations. Since the cluster memory bank copes with all the vehicle features, it can summarize them into a discriminative feature representation. To deeply exploit the instance/cluster information, TCRL proposes two additional loss functions. For the instance-level feature, a hybrid contrastive loss (HCL) re-defines the sample correlations by approaching the positive instance features and pushing all negative instance features away. For the cluster-level feature, a weighted regularization cluster contrastive loss (WRCCL) refines the pseudo labels by penalizing the mislabeled images according to the instance similarity. Extensive experiments show that TCRL outperforms many state-of-the-art unsupervised vehicle re-identification approaches.

Instance-aware image dehazing

The existing deep learning-based image dehazing algorithms commonly employ an encoder–decoder structure to learn a direct mapping from hazy images to haze-free images. However, these state-of-the-art methods often fail to consider the varying contents of hazy images across different scenes, resulting in unsatisfactory dehazing outcomes. To address this issue, this paper attempts to integrate a novel instance-aware subnet into the classic encoder–decoder structure in order to achieve a clear separation between figure and background, conducting the selective incorporation of instance features into the dehazing network. Specifically, we introduce a novel architecture called the hybrid residual attention network, which is capable of separately extracting full-image features and instance-level features. This architecture incorporates attention mechanisms and a multi-scale dilated convolution structure, enabling adaptive perception of haze density in different scenes. Additionally, we introduce a global feature fusion subnet that employs a pixel attention structure to fuse features from the entire image and multiple individual instances, thus being aware of instance features. Compared to existing methods, our approach offers a major advantage in accurately estimating the haze density of individual instances, reducing color distortion, and mitigating noise amplification in the output images. Experimental results demonstrate that our method outperforms existing methods across different evaluation metrics and testing benchmarks. Therefore, we believe that our method will serve as a valuable addition to the current collection of artificial intelligence models and will benefit engineering applications in video surveillance and high-level computer vision tasks.

Holistic-Guided Disentangled Learning With Cross-Video Semantics Mining for Concurrent First-Person and Third-Person Activity Recognition.

The popularity of wearable devices has increased the demands for the research on first-person activity recognition. However, most of the current first-person activity datasets are built based on the assumption that only the human-object interaction (HOI) activities, performed by the camera-wearer, are captured in the field of view. Since humans live in complicated scenarios, in addition to the first-person activities, it is likely that third-person activities performed by other people also appear. Analyzing and recognizing these two types of activities simultaneously occurring in a scene is important for the camera-wearer to understand the surrounding environments. To facilitate the research on concurrent first- and third-person activity recognition (CFT-AR), we first created a new activity dataset, namely PolyU concurrent first- and third-person (CFT) Daily, which exhibits distinct properties and challenges, compared with previous activity datasets. Since temporal asynchronism and appearance gap usually exist between the first- and third-person activities, it is crucial to learn robust representations from all the activity-related spatio-temporal positions. Thus, we explore both holistic scene-level and local instance-level (person-level) features to provide comprehensive and discriminative patterns for recognizing both first- and third-person activities. On the one hand, the holistic scene-level features are extracted by a 3-D convolutional neural network, which is trained to mine shared and sample-unique semantics between video pairs, via two well-designed attention-based modules and a self-knowledge distillation (SKD) strategy. On the other hand, we further leverage the extracted holistic features to guide the learning of instance-level features in a disentangled fashion, which aims to discover both spatially conspicuous patterns and temporally varied, yet critical, cues. Experimental results on the PolyU CFT Daily dataset validate that our method achieves the state-of-the-art performance.

Weakly Supervised Object Detection for Remote Sensing Images via Progressive Image-Level and Instance-Level Feature Refinement

Weakly supervised object detection (WSOD) aims to predict a set of bounding boxes and corresponding category labels for instances with only image-level supervisions. Compared with fully supervised object detection, WSOD in remote sensing images (RSIs) is much more challenging due to the vast foreground-related context regions. In this paper, we propose a progressive image-level and instance-level feature refinement network to address the problems of missing detection and part domination for WSOD in RSIs. Firstly, we propose a multi-label attention mining loss (MAML)-guided image-level feature refinement branch to effectively allocate the computational resources towards the most informative part of images. With the supervision of MAML, all latent instances in images are emphasized. However, image-level feature refinement further expands responsive gaps between the informative part and other sub-optimal informative ones, which results in exacerbating the problem of part domination. In order to alleviate the above-mentioned limitation, we further construct an instance-level feature refinement branch to re-balance the contributions of different adjacent candidate bounding boxes according to the detection task. An instance selection loss (ISL) is proposed to progressively boost the representation of salient regions by exploring supervision from the network itself. Finally, we integrate the image-level and instance-level feature refinement branches into a complete network and the proposed MAML and ISL functions are merged with class classification and box regression to optimize the whole WSOD network in an end-to-end training fashion. We conduct experiments on two popular WSOD datasets, NWPU VHR-10.v2 and DIOR. All the experimental results demonstrate that our method achieves a competitive performance compared with other state-of-the-art approaches.

Box2Poly: Memory-Efficient Polygon Prediction of Arbitrarily Shaped and Rotated Text

Recently, Transformer-based text detection techniques have sought to predict polygons by encoding the coordinates of individual boundary vertices using distinct query features. However, this approach incurs a significant memory overhead and struggles to effectively capture the intricate relationships between vertices belonging to the same instance. Consequently, irregular text layouts often lead to the prediction of outlined vertices, diminishing the quality of results. To address these challenges, we present an innovative approach rooted in Sparse R-CNN: a cascade decoding pipeline for polygon prediction. Our method ensures precision by iteratively refining polygon predictions, considering both the scale and location of preceding results. Leveraging this stabilized regression pipeline, even employing just a single feature vector to guide polygon instance regression yields promising detection results. Simultaneously, the leverage of instance-level feature proposal substantially enhances memory efficiency ( > 50% less vs. the SOTA method DPText-DETR) and reduces inference speed (> 40% less vs. DPText-DETR) with comparable performance on benchmarks. The code is available at https://github.com/Albertchen98/Box2Poly.git.

Discriminative Action Snippet Propagation Network for Weakly Supervised Temporal Action Localization

Weakly supervised temporal action localization (WTAL) aims to classify and localize actions in untrimmed videos with only video-level labels. Recent studies have attempted to obtain more accurate temporal boundaries by exploiting latent action instances in ambiguous snippets or propagating representative action features. However, empirically handcrafted ambiguous snippet extraction and the imprecise alignment of representative snippet propagation lead to challenges in modeling the completeness of actions for these methods. In this article, we propose a Discriminative Action Snippet Propagation Network (DASP-Net) to accurately discover ambiguous snippets in videos and propagate discriminative instance-level features throughout the video for improving action completeness. Specifically, we introduce a novel discriminative feature propagation module for capturing the global contextual attention and propagating the action concept across the whole video by perceiving the discriminative action snippets with instance information from the same video. Simultaneously, we incorporate denoised pseudo-labels as supervision, where we correct the controversial prediction based on the feature space distribution during training, thereby alleviating false detection caused by noise background features. Furthermore, we design an ambiguous feature mining module, which maximizes the feature affinity information of action and background in ambiguous snippets to generate more accurate latent action and background snippets and learns more precise action instance boundaries through contrastive learning of action and background snippets. Extensive experiments show that DASP-Net achieves state-of-the-art results on THUMOS14 and ActivityNet1.2 datasets.

Domain Adaptation of Anchor-Free object detection for urban traffic

Modern detectors are mostly trained under single and limited conditions. However, object detection faces various complex and open situations in autonomous driving, especially in urban street scenes with dense objects and complex backgrounds. Due to the shift in data distribution, modern detectors cannot perform well in actual urban environments. Using domain adaptation to improve detection performance is one of the key methods to extend object detection from limited situations to open situations. To this end, this article proposes a Domain Adaptation of Anchor-Free object detection (DAAF) for urban traffic. DAAF is a cross-domain object detection method that performs feature alignment including two aspects. On the one hand, we designed a fully convolutional adversarial training method for global feature alignment at the image level. Meanwhile, images can generally be decomposed into structural information and texture information. In urban street scenes, the structural information of images is generally similar. The main difference between the source domain and the target domain is texture information. Therefore, during global feature alignment, this paper proposes a method called texture information limitation (TIL). On the other hand, in order to solve the problem of variable aspect ratios of objects in urban street scenes, this article uses an anchor-free detector as the baseline detector. Since the anchor-free object detector can obtain neither explicit nor implicit instance-level features, we adopt Pixel-Level Adaptation (PLA) to align local features instead of instance-level alignment for local features. The size of the object has the greatest impact on the final detection effect, and the object scale in urban scenes is relatively rich. Guided by the differentiation of attention mechanisms, a multi-level adversarial network is designed to perform feature alignment of the output space at different feature levels called Scale Information Limitation (SIL). We conducted cross-domain detection experiments by using various urban streetscape autonomous driving object detection datasets, including adverse weather conditions, synthetic data to real data, and cross-camera adaptation. The experimental results indicate that the method proposed in this article is effective.

Causality-Enhanced Multiple Instance Learning with Graph Convolutional Networks for Parkinsonian Freezing-of-Gait Assessment.

Freezing of gait (FoG) is a common disabling symptom of Parkinson's disease (PD). It is clinically characterized by sudden and transient walking interruptions for specific human body parts, and it presents the localization in time and space. Due to the difficulty in extracting global fine-grained features from lengthy videos, developing an automated five-point FoG scoring system is quite challenging. Therefore, we propose a novel video-based automated five-classification FoG assessment method with a causality-enhanced multiple-instance-learning graph convolutional network (GCN). This method involves developing a temporal segmentation GCN to segment each video into three motion stages for stage-level feature modeling, followed by a multiple-instance-learning framework to divide each stage into short clips for instance-level feature extraction. Subsequently, an uncertainty-driven multiple-instance-learning GCN is developed to capture spatial and temporal fine-grained features through GCN scheme and uncertainty learning, respectively, for acquiring global representations. Finally, a causality-enhanced graph generation strategy is proposed to exploit causal inference for mining and enhancing human structures causally related to clinical assessment, thereby extracting spatial causal features. Extensive experimental results demonstrate the excellent performance of the proposed method on five-classification FoG assessment with an accuracy of 62.72% and an acceptable accuracy of 91.32%, which is confirmed by independent testing. Additionally, it enables temporal and spatial localization of FoG events to a certain extent, facilitating reasonable clinical interpretations. In conclusion, our method provides a valuable tool for automated FoG assessment in PD, and the proposed causality-related component exhibits promising potential for extension to other general and medical fine-grained action recognition tasks.

Contrasting augmented features for domain adaptation with limited target domain data

Domain adaptation aims to alleviate distribution gaps between source and target domains. However, when the available target domain data are scarce for training, learning generalizable representations for domain adaptation is challenging. We propose a novel approach, dubbed Contrasting Augmented Features (CAF), to tackle the challenge of insufficient target domain data for domain adaptation, by generating and contrasting augmented features. We introduce a semantic feature generator to generate augmented features by replacing the instance-level feature statistics of one domain with another domain. With the augmented features, we further design the reweighted instance contrastive loss and category contrastive loss to improve feature discrimination and align feature distributions of source and target domains. CAF can be applied to few-shot domain adaptation and unsupervised domain adaptation with limited unlabeled target domain data. Despite its simplicity, extensive experiments show promising results for both applications. In addition, experiments demonstrate that CAF is more robust to the number of target domain data and also effective in vanilla unsupervised domain adaptation setting with full target domain data.

Rice seed vigor detection based on near-infrared hyperspectral imaging and deep transfer learning.

Vigor is one of the important factors that affects rice yield and quality. Rapid and accurate detection of rice seed vigor is of great importance for rice production. In this study, near-infrared hyperspectral imaging technique and transfer learning were combined to detect rice seed vigor. Four varieties of artificial-aged rice seeds (Yongyou12, Yongyou1540, Suxiangjing100, and Longjingyou1212) were studied. Different convolutional neural network (CNN) models were built to detect the vigor of the rice seeds. Two transfer strategies, fine-tuning and MixStyle, were used to transfer knowledge among different rice varieties for vigor detection. The experimental results showed that the convolutional neural network model of Yongyou12 classified the vigor of Yongyou1540, Suxiangjing100, and Longjingyou1212 through MixStyle transfer knowledge, and the accuracy reached 90.00%, 80.33%, and 85.00% in validation sets, respectively, which was better or close to the initial modeling performances of each variety. MixStyle statistics are based on probabilistic mixed instance-level features of cross-source domain training samples. When training instances, new domains can be synthesized, which increases the domain diversity of the source domain, thereby improving the generalization ability of the trained model. This study would help rapid and accurate detection of a large varieties of crop seeds.

A hybrid few-shot multiple-instance learning model predicting the aggressiveness of lymphoma in PET/CT images

Background and objectivePatients with aggressive non-Hodgkin lymphoma (NHL) undergo distinct therapy strategies compared with indolent NHL patients. However, it is challenging to estimate NHL aggressiveness based on visual inspection of positron emission tomography (PET) or computed tomography (CT) images. Since diffuse large B-cell lymphoma (DLBCL) and Follicular lymphoma (FL) are the most typical and dominant aggressive and indolent NHL, respectively, this study aims to develop an artificial-intelligence-enabled model to distinguish DLBCL from FL in PET/CT images as the first step to tackle this challenge. MethodsWe propose a hybrid few-shot multiple-instance learning model to predict the aggressiveness of the NHL. First, rotation-based self-supervision learning (SSL) has been employed to train the encoder on a large-scale, publicly available CT image dataset. Second, hybrid instance-level features are obtained for each NHL lesion by combining deep features with the radiomics features from both PET and CT modalities. Third, instance-level features are transformed into bag-level (or patient-level) representations. Finally, bag-level representations are fed into a distance-based classifier through few-shot learning to predict NHL aggressiveness. ResultsOur model achieves an accuracy of 0.751 ± 0.008, a sensitivity of 0.787 ± 0.012, a specificity of 0.715 ± 0.013, an F1-score of 0.753 ± 0.009, and an area under the curve (AUC) of 0.795 ± 0.009 at the bag level. It outperforms the typical counterparts that use the radiomic features, random forest for feature selection, and support vector machines (SVMs) as classifiers. The three counterparts yield accuracies of 0.714 ± 0.023, 0.705 ± 0.008, and 0.698 ± 0.008, respectively. Moreover, settings of the SSL training dataset (Deep lesion) and task (rotation), hybrid CT and radiomic PET features, the pool-layer strategy of maximum, and distance-based classifier generate the best model. ConclusionsA hybrid few-shot multiple-instance learning model can predict lymphoma aggressiveness in PET/CT images and could be a potential tool for determining therapy strategies. Hybrid features and the combination of SSL, few-shot learning, and weakly supervised learning are the two powerful pillars of the model, and these can be expanded to other medical applications with limited samples and incomplete annotations.

Hierarchical Contrast for Unsupervised Skeleton-Based Action Representation Learning

This paper targets unsupervised skeleton-based action representation learning and proposes a new Hierarchical Contrast (HiCo) framework. Different from the existing contrastive-based solutions that typically represent an input skeleton sequence into instance-level features and perform contrast holistically, our proposed HiCo represents the input into multiple-level features and performs contrast in a hierarchical manner. Specifically, given a human skeleton sequence, we represent it into multiple feature vectors of different granularities from both temporal and spatial domains via sequence-to-sequence (S2S) encoders and unified downsampling modules. Besides, the hierarchical contrast is conducted in terms of four levels: instance level, domain level, clip level, and part level. Moreover, HiCo is orthogonal to the S2S encoder, which allows us to flexibly embrace state-of-the-art S2S encoders. Extensive experiments on four datasets, i.e., NTU-60, NTU-120, PKU-I and PKU-II, show that HiCo achieves a new state-of-the-art for unsupervised skeleton-based action representation learning in two downstream tasks including action recognition and retrieval, and its learned action representation is of good transferability. Besides, we also show that our framework is effective for semi-supervised skeleton-based action recognition. Our code is available at https://github.com/HuiGuanLab/HiCo.

Pseudo-supervised image clustering based on meta-features

Stable semantics is a prerequisite for achieving excellent image clustering. However, most current methods suffer from inaccurate class semantic estimation, which limits the clustering performance. For the sake of addressing the issue, we propose a pseudo-supervised clustering framework based on meta-features. First, the framework mines meta-semantic features (i.e., meta-features) of image categories based on instance-level features, which not only preserves instance-level information but also ensures the semantic robustness of meta-features. Ulteriorly, we propagate pseudo-labels to its global neighbor samples with meta-features as the center, which effectively avoids the accumulation of errors caused by the misclassification of samples at the cluster boundary. Finally, we exploit the cross-entropy loss with label smoothing to optimize the pseudo-label optimization network. This optimization method not only achieves a direct mapping from features to stable semantic labels, but also effectively avoids suboptimal solutions caused by multi-level optimization. Extensive experiments demonstrate that our method significantly outperforms twenty-one competing clustering methods on six challenging datasets.

Cross-scene target detection based on feature adaptation and uncertainty-aware pseudo-label learning for high resolution SAR images

The characteristics of synthetic aperture radar (SAR) images are easily affected by factors such as sensor parameters, imaging scenes, etc., which may lead to data distributional discrepancies between the ideal training phase and the practical application phase, resulting in generalization performance degradation of deep learning based detectors in practice. In this paper, an accurate and fast domain adaptive detection method is proposed to address the performance degradation problem of cross-scene detection without requiring additional manual annotations. The cross-scene detection performance is gradually improved by confidence-induced unsupervised feature adaptation (CUFA) and uncertainty-aware adaptive pseudo-label learning (UAPL). In CUFA, image-level and instance-level feature adaptation are progressively implemented to bridge the distribution discrepancies between the source and the target domain in the latent feature space. Then, UAPL is performed to adaptively introduce accurate and stable supervisions of target domain according to the learning status of the detector, avoiding error accumulations and further improving the cross-scene target detection performance. The proposed cross-scene detection method is validated on airborne and spaceborne SAR image datasets of different imaging scenes. The experimental results show that by combining both CUFA and UAPL, compared with the second-best comparison method, the proposed method significantly improves the cross-scene target detection performance in SAR images (at least + 6.5% AP50 and F1 in miniSAR and FARAD cross-scene tasks, and at least + 7.5% AP50 and F1 in Gaofen-3 and Sentinel-1 cross-scene tasks), which facilitates application of the deep learning based detectors in the realistic scenarios.

A deep multi-instance neural network for dyeing-free inspection of yarn dyeing uniformity

Dyeing evenness is a decisive factor influencing the commercial values of polyester yarn. Thus, the inspection of yarn dyeing uniformity plays a vital role in the textile community. Due to the inefficiency of dyeing process, the traditional dyeing-based inspection methods are both laborious and time-consuming. To improve efficiency, this study attempts to develop a fast dyeing-free method for dyeing uniformity inspection based on imaging spectrometer and multi-instance learning (MIL). The relevant properties of yarn samples are first recorded in the hyperspectral images (HSIs). As the available labels are ambiguous, classifying the collected HSIs has become a MIL problem. Meanwhile, the correlation between the spectral pixels and the sample labels might be sophisticated. It might be difficult for the existing MIL methods to learn such data. In this paper, a deep Fisher score-based multi-instance neural network (DFSNet) is also proposed for classifying the captured HSIs. The DFSNet is able to learn a sophisticated correlation between deep instance features and bag representation. Specifically, a Fisher score-based MIL pooling layer is first developed to convert the instance-level features into bag-level features. The DFSNet is then developed with a ladder structure and the Fisher score-based MIL pooling layer. The proposed dyeing-free method and DFSNet are evaluated using the actual polyester samples. Moreover, the proposed DFSNet and the spectral range of HSI are further analyzed. The experiment results have indicated that the proposed method could achieve satisfactory performance, providing a potential solution to fast dyeing uniformity inspection.

DA-RDD: Toward Domain Adaptive Road Damage Detection Across Different Countries

Recent advances on road damage detection relies on a large amount of labeled data, whilst collecting pavement image is labor-intensive and time-consuming. Unsupervised Domain Adaptation (UDA) provides a promising solution to adapt a source domain to the target domain, however, cross-domain crack detection is still an open problem. In this paper, we propose domain adaptive road damage detection termed as DA-RDD, by incorporating image-level with instance-level feature alignment for domain-invariant representation learning in an adversarial manner. Specifically, importance weighting is introduced to evaluate the intermediate samples for image-level alignment between domains, and we aggregate RoI-wise feature with multi-scale contextual information to recover the crack details for progressive domain alignment at instance level. Additionally, a large-scale road damage dataset (based on Road Damage Dataset 2020 (RDD2020)) named as RDD2021 is constructed with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$100k$ </tex-math></inline-formula> synthetic labeled distress images. Extensive experimental results on damage detection across different countries demonstrate the universality and superiority of DA-RDD, and empirical studies on RDD2021 further claim its effectiveness and advancement. To our best knowledge, it is the first time to investigate domain adaptative pavement crack detection, and we expect the contributions in this work would facilitate the development of generalized road damage detection in the future.

SSDA-YOLO: Semi-supervised domain adaptive YOLO for cross-domain object detection

Domain adaptive object detection (DAOD) aims to alleviate transfer performance degradation caused by the cross-domain discrepancy. However, most existing DAOD methods are dominated by outdated and computationally intensive two-stage Faster R-CNN, which is not the first choice for industrial applications. In this paper, we propose a novel semi-supervised domain adaptive YOLO (SSDA-YOLO) based method to improve cross-domain detection performance by integrating the compact one-stage stronger detector YOLOv5 with domain adaptation. Specifically, we adapt the knowledge distillation framework with the Mean Teacher model to assist the student model in obtaining instance-level features of the unlabeled target domain. We also utilize the scene style transfer to cross-generate pseudo images in different domains for remedying image-level differences. In addition, an intuitive consistency loss is proposed to further align cross-domain predictions. We evaluate SSDA-YOLO on public benchmarks including PascalVOC, Clipart1k, Cityscapes, and Foggy Cityscapes. Moreover, to verify its generalization, we conduct experiments on yawning detection datasets collected from various real classrooms. The results show considerable improvements of our method in these DAOD tasks, which reveals both the effectiveness of proposed adaptive modules and the urgency of applying more advanced detectors in DAOD. Our code is available on https://github.com/hnuzhy/SSDA-YOLO.

Diversity feature constraint based on heterogeneous data for unsupervised person re-identification

Person re-identification (ReID) based on heterogeneous data aims to search for the same pedestrian from different modalities. The existing unsupervised heterogeneous ReID method overly relies on pseudo labels and ignores the inter-image feature relationship. In the paper, we propose a novel Diversity Feature Constraint (DFC) method to simultaneously consider the clustering-level and instance-level feature relationship for unsupervised heterogeneous ReID. On the one hand, we employ the clustering algorithm to produce pseudo labels for heterogeneous images. Then, the clustering-level constraint is designed to optimize the model. On the other hand, considering that the clustering algorithm may generate some noise, we propose the complementary intra-modality instance-level constraint to correlate any two intra-modality images. Meanwhile, for eliminating the inter-modality discrepancy, the inter-modality instance-level constraint is developed to decrease the large inter-modality gap. We construct the potential feature relationship between heterogeneous images to constrain the feature distribution. By experiments, we prove that over-reliance on pseudo labels generates limited performance. Exploring inter-image potential relationships is an important way to solve the unsupervised problem. Extensive results demonstrate that DFC achieves superior performance that outperforms other methods by a large margin, improving 15.23% and 9.37% at rank-1 and mAP indexes compared with the clustering method on SYSU-MM01.

GraphSKT: Graph-guided Structured Knowledge Transfer for Domain Adaptive Lesion Detection.

Adversarial-based adaptation has dominated the area of domain adaptive detection over the past few years. Despite their general efficacy for various tasks, the learned representations may not capture the intrinsic topological structures of the whole images and thus are vulnerable to distributional shifts especially in real-world applications, such as geometric distortions across imaging devices in medical images. In this case, forcefully matching data distributions across domains cannot ensure precise knowledge transfer and are prone to result in the negative transfer. In this paper, we explore the problem of domain adaptive lesion detection from the perspective of relational reasoning, and propose a Graph-Structured Knowledge Transfer (GraphSKT) framework to perform hierarchical reasoning by modeling both the intra- and inter-domain topological structures. To be specific, we utilize cross-domain correspondence to mine meaningful foreground regions for representing graph nodes and explicitly endow each node with contextual information. Then, the intra- and inter-domain graphs are built on the top of instance-level features to achieve a high-level understanding of the lesion and whole medical image, and transfer the structured knowledge from source to target domains. The contextual and semantic information is propagated through graph nodes methodically, enhancing the expressive power of learned features for the lesion detection tasks. Extensive experiments on two types of challenging datasets demonstrate that the proposed GraphSKT significantly outperforms the state-of-the-art approaches for detection of polyps in colonoscopy images and of mass in mammographic images.