Vision Mamba Empowered by Dynamic Domain Generalization for Cross-Modality Medical Segmentation.

The performance of convolutional neural networks (CNNs) often drop when they encounter a domain shift. Recently, unsupervised domain adaptation (UDA) and domain generalization (DG) techniques have been proposed to solve this problem. However, access to source domain data is required for UDA and DG approaches, which may not always be available in practice due to data privacy. In this paper, we propose a novel test-time adaptation framework for volumetric medical image segmentation without any source domain data for adaptation and target domain data for offline training. Specifically, our proposed framework only needs pre-trained CNNs in the source domain, and the target image itself. Our method aligns the target image on both image and latent feature levels to source domain during the test-time. There are three parts in our proposed framework: (1) multi-task segmentation network (Seg), (2) autoencorders (AEs) and (3) translation network (T). Seg and AEs are pre-trained with source domain data. At test-time, the weights of these pre-trained CNNs (decoders of Seg and AEs) are fixed, and T is trained to align the target image to source domain at image-level by the autoencoders which optimize the similarity between input and reconstructed output. The encoder of Seg is also updated to increase the domain generalizability of the model towards the source domain at the feature level with self-supervised tasks. We evaluate our method on healthy controls, adult Huntington’s disease (HD) patients and pediatric Aicardi Goutières Syndrome (AGS) patients, with different scanners and MRI protocols. The results indicate that our proposed method improves the performance of CNNs in the presence of domain shift at test-time.KeywordsSelf-supervisedTest-time trainingTest-time adaptationSegmentation

Medical image segmentation tasks hitherto have achieved excellent progresses with large-scale datasets, which empowers us to train potent deep convolutional neural networks (DCNNs). However, labeling such large-scale datasets is laborious and error-prone, which leads the noisy (or incorrect) labels to be an ubiquitous problem in the real-world scenarios. In addition, data collected from different sites usually exhibit significant data distribution shift (or domain shift). As a result, noisy label and domain shift become two common problems in medical imaging application scenarios, especially in medical image segmentation, which degrade the performance of deep learning models significantly. In this paper, we identify a novel problem hidden in medical image segmentation, which is unsupervised domain adaptation on noisy labeled data, and propose a novel algorithm named "Self-Cleansing Unsupervised Domain Adaptation" (S-CDUA) to address such issue. S-CUDA sets up a realistic scenario to solve the above problems simultaneously where training data (i.e., source domain) not only shows domain shift w.r.t. unsupervised test data (i.e., target domain) but also contains noisy labels. The key idea of S-CUDA is to learn noise-excluding and domain invariant knowledge from noisy supervised data, which will be applied on the highly corrupted data for label cleansing and further data-recycling, as well as on the test data with domain shift for supervised propagation. To this end, we propose a novel framework leveraging noisy-label learning and domain adaptation techniques to cleanse the noisy labels and learn from trustable clean samples, thus enabling robust adaptation and prediction on the target domain. Specifically, we train two peer adversarial networks to identify high-confidence clean data and exchange them in companions to eliminate the error accumulation problem and narrow the domain gap simultaneously. In the meantime, the high-confidence noisy data are detected and cleansed in order to reuse the contaminated training data. Therefore, our proposed method can not only cleanse the noisy labels in the training set but also take full advantage of the existing noisy data to update the parameters of the network. For evaluation, we conduct experiments on two popular datasets (REFUGE and Drishti-GS) for optic disc (OD) and optic cup (OC) segmentation, and on another public multi-vendor dataset for spinal cord gray matter (SCGM) segmentation. Experimental results show that our proposed method can cleanse noisy labels efficiently and obtain a model with better generalization performance at the same time, which outperforms previous state-of-the-art methods by large margin. Our code can be found at https://github.com/zzdxjtu/S-cuda.

<p>Unsupervised medical domain adaptation (DA) is an important field of study in medical image analysis, as domain shift is a very common issue for medical imaging. Unsupervised domain adaptation for the purpose of image segmentation on an unseen target domain has shown to be effective for brain MR scan problems. To improve the performance of unsupervised medical DA for segmentation, a Structure Preserving Cycle-GAN (SP Cycle-GAN) implementation was introduced. The SP Cycle-GAN was used to adapt STARE-domain retinal scans to the target DRIVE dataset domain, for the purpose of blood vessel segmentation via an Attention U-Net model. Pseudo-label generation on unlabelled source domain images was investigated to see if additional generated data could improve segmentation performance on the target domain. The implemented SP Cycle-GAN was shown to be effective for preserving structures in the source domain in translated target domain images. The SP Cycle-GAN however was not suited to the STARE and DRIVE datasets and did not outperform a simple baseline method of a trained segmentation model on the original source domain STARE data, which achieved a mean Dice Score (DSC) of 0.786 ± 0.015 on the DRIVE test dataset. The use of pseudo-labelled data to incorporate unlabelled source domain data for unsupervised DA was shown to have potential for improving performance, with a DSC of 0.718 ± 0.060 using pseudo-labelled data vs. 0.709 ± 0.063 when not using pseudo-labelled data. While the SP Cycle-GAN preserved blood vessel structure effectively, this resulted in another domain shift caused by the difference in overall shape of the scans. The introduced SP Cycle-GAN method shows promise for other DA datasets for medical image segmentation, in which small structures within an overall larger scan structure must be preserved.</p>

<p>Unsupervised medical domain adaptation (DA) is an important field of study in medical image analysis, as domain shift is a very common issue for medical imaging. Unsupervised domain adaptation for the purpose of image segmentation on an unseen target domain has shown to be effective for brain MR scan problems. To improve the performance of unsupervised medical DA for segmentation, a Structure Preserving Cycle-GAN (SP Cycle-GAN) implementation was introduced. The SP Cycle-GAN was used to adapt STARE-domain retinal scans to the target DRIVE dataset domain, for the purpose of blood vessel segmentation via an Attention U-Net model. Pseudo-label generation on unlabelled source domain images was investigated to see if additional generated data could improve segmentation performance on the target domain. The implemented SP Cycle-GAN was shown to be effective for preserving structures in the source domain in translated target domain images. The SP Cycle-GAN however was not suited to the STARE and DRIVE datasets and did not outperform a simple baseline method of a trained segmentation model on the original source domain STARE data, which achieved a mean Dice Score (DSC) of 0.786 ± 0.015 on the DRIVE test dataset. The use of pseudo-labelled data to incorporate unlabelled source domain data for unsupervised DA was shown to have potential for improving performance, with a DSC of 0.718 ± 0.060 using pseudo-labelled data vs. 0.709 ± 0.063 when not using pseudo-labelled data. While the SP Cycle-GAN preserved blood vessel structure effectively, this resulted in another domain shift caused by the difference in overall shape of the scans. The introduced SP Cycle-GAN method shows promise for other DA datasets for medical image segmentation, in which small structures within an overall larger scan structure must be preserved.</p>

Unsupervised Domain Adaptation of Auto-Segmentation on Multi-Source MRIs

Domain shift happens in cross-domain scenarios commonly because of the wide gaps between different domains: when applying a deep learning model well-trained in one domain to another target domain, the model usually performs poorly. To tackle this problem, unsupervised domain adaptation (UDA) techniques are proposed to bridge the gap between different domains, for the purpose of improving model performance without annotation in the target domain. Particularly, UDA has a great value for multimodal medical image analysis, where annotation difficulty is a practical concern. However, most existing UDA methods can only achieve satisfactory improvements in one adaptation direction (e.g., MRI to CT), but often perform poorly in the other (CT to MRI), limiting their practical usage. In this paper, we propose a bidirectional UDA (BiUDA) framework based on disentangled representation learning for equally competent two-way UDA performances. This framework employs a unified domain-aware pattern encoder which not only can adaptively encode images in different domains through a domain controller, but also improve model efficiency by eliminating redundant parameters. Furthermore, to avoid distortion of contents and patterns of input images during the adaptation process, a content-pattern consistency loss is introduced. Additionally, for better UDA segmentation performance, a label consistency strategy is proposed to provide extra supervision by recomposing target-domain-styled images and corresponding source-domain annotations. Comparison experiments and ablation studies conducted on two public datasets demonstrate the superiority of our BiUDA framework to current state-of-the-art UDA methods and the effectiveness of its novel designs. By successfully addressing two-way adaptations, our BiUDA framework offers a flexible solution of UDA techniques to the real-world scenario.KeywordsDomain adaptationMulti-modalitySegmentation

EDRL: Entropy-guided disentangled representation learning for unsupervised domain adaptation in semantic segmentation

With the rapid development of deep learning technology, semantic segmentation methods have been widely used in remote sensing data. A pretrained semantic segmentation model usually cannot perform well when the testing images (target domain) have an obvious difference from the training data set (source domain), while a large enough labeled data set is almost impossible to be acquired for each scenario. Unsupervised domain adaptation (DA) techniques aim to transfer knowledge learned from the source domain to a totally unlabeled target domain. By reducing the domain shift, DA methods have shown the ability to improve the classification accuracy for the target domain. Hence, in this letter, we propose an unsupervised adversarial DA network that converts deep features into 2-D feature curves and reduces the discrepancy between curves from the source domain and curves from the target domain based on a conditional generative adversarial networks (cGANs) model. Our proposed DA network is able to improve the semantic labeling accuracy when we apply a pretrained semantic segmentation model to the target domain. To test the effectiveness of the proposed method, experiments are conducted on the International Society for Photogrammetry and Remote Sensing (ISPRS) 2-D Semantic Labeling data set. Results show that our proposed network is able to stably improve overall accuracy not only when the source and target domains are from the same city but with different building styles but also when the source and target domains are from different cities and acquired by different sensors. By comparing with a few state-of-the-art DA methods, we demonstrate that our proposed method achieves the best cross-domain semantic segmentation performance.

Medical image deep learning segmentation has shown great potential in becoming an ubiquitous part of the clinical analysis pipeline. However, these methods cannot guarantee high quality predictions when the source and target domains are dissimilar due to different acquisition protocols, or biases in patient cohorts. Recently, unsupervised domain adaptation techniques have shown great potential in alleviating this problem by minimizing the shift between the source and target distributions. In this work, we aim to predict tissue segmentation maps on an unseen dataset, which has both different acquisition parameters and population bias when compared to our training data. We achieve this by investigating two unsupervised domain adaptation (UDA) techniques with the objective of finding the best solution for our problem. We compare the two methods with a baseline fully-supervised segmentation network in terms of cortical thickness measures.

Agricultural land extraction is an essential technical means to promote sustainable agricultural development and modernization research. Existing supervised algorithms rely on many finely annotated remote-sensing images, which is both time-consuming and expensive. One way to reduce the annotation cost approach is to migrate models trained on existing annotated data (source domain) to unannotated data (target domain). However, model generalization capability is often unsatisfactory due to the limit of the domain gap. In this work, we use an unsupervised adversarial domain adaptation method to train a neural network to close the gap between the source and target domains for unsupervised agricultural land extraction. The overall approach consists of two phases: inter-domain and intra-domain adaptation. In the inter-domain adaptation, we use a generative adversarial network (GAN) to reduce the inter-domain gap between the source domain (labeled dataset) and the target domain (unlabeled dataset). The transformer with robust long-range dependency modeling acts as the backbone of the generator. In addition, the multi-scale feature fusion (MSFF) module is designed in the generator to accommodate remote sensing datasets with different spatial resolutions. Further, we use an entropy-based approach to divide the target domain. The target domain is divided into two subdomains, easy split images and hard split images. By training against each other between the two subdomains, we reduce the intra-domain gap. Experiments results on the “DeepGlobe → LoveDA”, “GID → LoveDA” and “DeepGlobe → GID” unsupervised agricultural land extraction tasks demonstrate the effectiveness of our method and its superiority to other unsupervised domain adaptation techniques.

Due to the intrinsic domain shift among different modalities, it is nontrivial to directly apply a well-trained model into other cross-modality medical images. Unsupervised domain adaptation (UDA) has the potential to reduce such domain shift. However, existing UDA methods try to align domains in either image level or in feature level, failing to consider the unified relationship between cross-modality images and their corresponding features. In this paper, we propose a novel UDA framework for domain adaptive medical image segmentation. The proposed framework synergizes both pixel space and entropy space for domain alignment. Specifically, in the pixel space, we introduce the attention mechanism into CycleGAN, and enhance the semantic and geometric consistency of the target organs during the image style transformation. In entropy space, we utilize entropy minimization principle to force consistent image segmentation between well-annotated source domain and non-annotated target domain. The aligned ensemble of two representation spaces enables a well-trained segmentation model to effectively transfer from source domain to target domain. The experimental results demonstrate the effectiveness of the proposed method. For the task of multi-organs segmentation from cross-modality medical images, our proposed framework achieves state-of-the-art performance, with some specific metric even superior to those of supervised methods. The code is available at https://github.com/lichen14/AttENT.

Unsupervised domain adaptation (UDA) techniques have the potential to enhance the transferability of neural network models in unknown scenarios and reduce the labelling costs associated with unlabelled datasets. Popular solutions to this challenging UDA task are adversarial training and self-training. However, current adversarial-based UDA methods emphasize only global or local feature alignment, which is insufficient for tackling the domain shift. In addition, self-training-based methods inevitably produce many wrong pseudo labels on the target domain due to bias towards the source domain. To tackle the above problems, this paper proposes a hybrid training framework that integrates global-local adversarial training and self-training strategies to effectively tackle global-local domain shift. First, the adversarial approach measures the discrepancies between domains from domain and category-level perspectives. The adversarial network incorporates discriminators at the local-category and global-domain levels, thereby facilitating global-local feature alignment through multi-level adversarial training. Second, the self-training strategy is integrated to acquire domain-specific knowledge, effectively mitigating negative migration. By combining these two domain adaptation strategies, we present a more efficient approach for mitigating the domain gap. Finally, a self-labelling mechanism is introduced to directly explore the inherent distribution of pixels, allowing for the rectification of pseudo labels generated during the self-training stage. Compared to state-of-the-art UDA methods, the proposed method gains 3.2 % , 1.21 % , 5.86 % , 6.16 % mIoU improvements on Rural → Urban, Urban → Rural, Potsdam → Vaihingen, Vaihingen → Potsdam, respectively.

In Active Domain Adaptation (ADA), one uses Active Learning (AL) to select a subset of images from the target domain, which are then annotated and used for supervised domain adaptation (DA). Given the large performance gap between supervised and unsupervised DA techniques, ADA allows for an excellent trade-off between annotation cost and performance. Prior art makes use of measures of uncertainty or disagreement of models to identify ‘regions' to be annotated by the human oracle. However, these regions frequently comprise of pixels at object boundaries which are hard and tedious to annotate. Hence, even if the fraction of image pixels annotated reduces, the overall annotation time and the resulting cost still remain high. In this work, we propose an ADA strategy, which given a frame, identifies a set of classes that are hardest for the model to predict accurately, thereby recommending semantically meaningful regions to be annotated in a selected frame. We show that these set of ‘hard' classes are context-dependent and typically vary across frames, and when annotated help the model generalize better. We propose two ADA techniques: the Anchor-based and Augmentation-based approaches to select complementary and diverse regions in the context of the current training set. Our approach achieves 66.6 mIoU on GTA5 →Cityscapes dataset with an annotation budget of 4.7% in comparison to 64.9 mIoU by MADA [22] using 5% of annotations. Our technique can also be used as a decorator for any existing frame-based AL technique, e.g., we report 1.5% performance improvement for CDAL [1] on Cityscapes using our approach.

CyCMIS: Cycle-consistent Cross-domain Medical Image Segmentation via diverse image augmentation.

Medical image segmentation methods normally perform poorly when there is a domain shift between training and testing data. Unsupervised Domain Adaptation (UDA) addresses the domain shift problem by training the model using both labeled data from the source domain and unlabeled data from the target domain. Source-Free UDA (SFUDA) was recently proposed for UDA without requiring the source data during the adaptation, due to data privacy or data transmission issues, which normally adapts the pre-trained deep model in the testing stage. However, in real clinical scenarios of medical image segmentation, the trained model is normally frozen in the testing stage. In this paper, we propose Fourier Visual Prompting (FVP) for SFUDA of medical image segmentation. Inspired by prompting learning in natural language processing, FVP steers the frozen pre-trained model to perform well in the target domain by adding a visual prompt to the input target data. In FVP, the visual prompt is parameterized using only a small amount of low-frequency learnable parameters in the input frequency space, and is learned by minimizing the segmentation loss between the predicted segmentation of the prompted target image and reliable pseudo segmentation label of the target image under the frozen model. To our knowledge, FVP is the first work to apply visual prompts to SFUDA for medical image segmentation. The proposed FVP is validated using three public datasets, and experiments demonstrate that FVP yields better segmentation results, compared with various existing methods.