Domain-invariant Feature Representations Research Articles

A domain generalization network for imbalanced machinery fault diagnosis

Traditional models for Imbalanced Fault Diagnosis (IFD) face challenges in practical applications due to domain shifts caused by varying working conditions and machinery. Domain Generalization (DG) models provide an advantage over traditional approaches by learning class-discriminative and domain-invariant feature representations, allowing them to generalize to unseen target data. However, the scarcity of fault samples relative to healthy ones limits their application in real-world industrial scenarios. In this paper, we propose a Domain Mixed-Enhanced Domain Generalization Network (DEMDGN) that enhances IFD performance by utilizing mixup-based data augmentation and domain-based discrepancy metrics to align feature distributions across multiple heterogeneous source domains. By creating domain-invariant features, DEMDGN allows robust fault diagnosis under varying conditions. Extensive experiments on one marine machinery dataset and two bearing datasets demonstrate that the proposed method effectively addresses class imbalance and domain shift problems, achieving superior diagnostic performance.

A two-layer regression network for robust and accurate domain adaptation

Naturally, classification models suffer from severe performance degradation when they are tested on datasets different from the ones used in training. Unsupervised domain adaptation helps to improve the generalizability of a pre-trained model by transferring knowledge from the labeled source domain (i.e., training dataset) to the unlabeled target domain (i.e., test dataset). The typical way of domain adaptation is to align the data distributions in embedding spaces between source and target domains. However, most existing works only enhance cross-domain consistency in the latent space disregarding the impact of source samples on the target classification task. Besides, discovering a subspace shared by two domains first and then training a transfer classifier separately hardly ensures that the obtained target features are suited for the classification model. To address these issues, in this work, we propose a two-layer regression network for domain adaptation (TRN-DA). TRN-DA learns class-wise domain invariant feature representations by jointly optimizing three tasks: supervised classification of source domain, unsupervised reconstruction of target domain, and alignment of cross-domain distributions. Furthermore, we introduce the concept of weight (importance) for source instances so that the resulting classifier can adapt well to the target domain. We formulate the domain adaptation problem as a unified optimization problem and solve it in an iterative way. Experimental results on different cross-domain image classification tasks demonstrate the encouraging performance of our TRN-DA compared to several recent state-of-the-art methods.

EEG-DG: A Multi-Source Domain Generalization Framework for Motor Imagery EEG Classification.

Motor imagery EEG classification plays a crucial role in non-invasive Brain-Computer Interface (BCI) research. However, the performance of classification is affected by the non-stationarity and individual variations of EEG signals. Simply pooling EEG data with different statistical distributions to train a classification model can severely degrade the generalization performance. To address this issue, the existing methods primarily focus on domain adaptation, which requires access to the test data during training. This is unrealistic and impractical in many EEG application scenarios. In this paper, we propose a novel multi-source domain generalization framework called EEG-DG, which leverages multiple source domains with different statistical distributions to build generalizable models on unseen target EEG data. We optimize both the marginal and conditional distributions to ensure the stability of the joint distribution across source domains and extend it to a multi-source domain generalization framework to achieve domain-invariant feature representation, thereby alleviating calibration efforts. Systematic experiments conducted on a simulative dataset, BCI competition IV 2a, 2b, and OpenBMI datasets, demonstrate the superiority and competitive performance of our proposed framework over other state-of-the-art methods. Specifically, EEG-DG achieves average classification accuracies of 81.79% and 87.12% on datasets IV-2a and IV-2b, respectively, and 78.37% and 76.94% for inter-session and inter-subject evaluations on dataset OpenBMI, which even outperforms some domain adaptation methods. Our code is available at https://github.com/zxchit2022/EEG-DG for evaluation.

Marginal Subspace Learning With Group Low-Rank for Unsupervised Domain Adaptation.

Unsupervised domain adaptation is intended to construct a reliable model for the unlabeled target samples using the well-labeled but differently distributed source samples. To tackle the domain shift issue, learning domain-invariant feature representations across domains is important, and most of the existing methods have concentrated on this goal. However, these methods rarely take into consideration the group discriminability of the feature representation, which is detrimental to the final recognition. Therefore, this article proposes a novel unsupervised domain adaptation method, named marginal subspace learning with group low-rank (MSL-GLR), to extract both domain-invariant and discriminative feature representations. Specifically, MSL-GLR uses the retargeting strategy to relax the regression matrix, such that the regression values would be forced to satisfy a margin maximization criterion for the requirement of correct classification. Moreover, MSL-GLR imposes a class-induced low-rank constraint, which enables the samples of each class to be located in their respective subspace. In this way, the distance between samples from the same class can be decreased and the discriminant ability of the projection is greatly improved. Furthermore, with the help of alternating direction method of multipliers (ADMM), an efficient algorithm is presented to solve the resulting optimization problem. Finally, the effectiveness of the proposed MSL-GLR is demonstrated by comprehensive evaluations on multiple domain adaptation benchmark datasets.

TO-UGDA: target-oriented unsupervised graph domain adaptation

Graph domain adaptation (GDA) aims to address the challenge of limited label data in the target graph domain. Existing methods such as UDAGCN, GRADE, DEAL, and COCO for different-level (node-level, graph-level) adaptation tasks exhibit variations in domain feature extraction, and most of them solely rely on representation alignment to transfer label information from a labeled source domain to an unlabeled target domain. However, this approach can be influenced by irrelevant information and usually ignores the conditional shift of the downstream predictor. To effectively address this issue, we introduce a target-oriented unsupervised graph domain adaptive framework for graph adaptation called TO-UGDA. Particularly, domain-invariant feature representations are extracted using graph information bottleneck. The discrepancy between two domains is minimized using an adversarial alignment strategy to obtain a unified feature distribution. Additionally, the meta pseudo-label is introduced to enhance downstream adaptation and improve the model’s generalizability. Through extensive experimentation on real-world graph datasets, it is proved that the proposed framework achieves excellent performance across various node-level and graph-level adaptation tasks.

DI-V2X: Learning Domain-Invariant Representation for Vehicle-Infrastructure Collaborative 3D Object Detection

Vehicle-to-Everything (V2X) collaborative perception has recently gained significant attention due to its capability to enhance scene understanding by integrating information from various agents, e.g., vehicles, and infrastructure. However, current works often treat the information from each agent equally, ignoring the inherent domain gap caused by the utilization of different LiDAR sensors of each agent, thus leading to suboptimal performance. In this paper, we propose DI-V2X, that aims to learn Domain-Invariant representations through a new distillation framework to mitigate the domain discrepancy in the context of V2X 3D object detection. DI-V2X comprises three essential components: a domain-mixing instance augmentation (DMA) module, a progressive domain-invariant distillation (PDD) module, and a domain-adaptive fusion (DAF) module. Specifically, DMA builds a domain-mixing 3D instance bank for the teacher and student models during training, resulting in aligned data representation. Next, PDD encourages the student models from different domains to gradually learn a domain-invariant feature representation towards the teacher, where the overlapping regions between agents are employed as guidance to facilitate the distillation process. Furthermore, DAF closes the domain gap between the students by incorporating calibration-aware domain-adaptive attention. Extensive experiments on the challenging DAIR-V2X and V2XSet benchmark datasets demonstrate DI-V2X achieves remarkable performance, outperforming all the previous V2X models. Code is available at https://github.com/Serenos/DI-V2X.

Generalizable Sleep Staging via Multi-Level Domain Alignment

Automatic sleep staging is essential for sleep assessment and disorder diagnosis. Most existing methods depend on one specific dataset and are limited to be generalized to other unseen datasets, for which the training data and testing data are from the same dataset. In this paper, we introduce domain generalization into automatic sleep staging and propose the task of generalizable sleep staging which aims to improve the model generalization ability to unseen datasets. Inspired by existing domain generalization methods, we adopt the feature alignment idea and propose a framework called SleepDG to solve it. Considering both of local salient features and sequential features are important for sleep staging, we propose a Multi-level Feature Alignment combining epoch-level and sequence-level feature alignment to learn domain-invariant feature representations. Specifically, we design an Epoch-level Feature Alignment to align the feature distribution of each single sleep epoch among different domains, and a Sequence-level Feature Alignment to minimize the discrepancy of sequential features among different domains. SleepDG is validated on five public datasets, achieving the state-of-the-art performance.

Learning to Generalize over Subpartitions for Heterogeneity-Aware Domain Adaptive Nuclei Segmentation

Annotation scarcity and cross-modality/stain data distribution shifts are two major obstacles hindering the application of deep learning models for nuclei analysis, which holds a broad spectrum of potential applications in digital pathology. Recently, unsupervised domain adaptation (UDA) methods have been proposed to mitigate the distributional gap between different imaging modalities for unsupervised nuclei segmentation in histopathology images. However, existing UDA methods are built upon the assumption that data distributions within each domain should be uniform. Based on the over-simplified supposition, they propose to align the histopathology target domain with the source domain integrally, neglecting severe intra-domain discrepancy over subpartitions incurred by mixed cancer types and sampling organs. In this paper, for the first time, we propose to explicitly consider the heterogeneity within the histopathology domain and introduce open compound domain adaptation (OCDA) to resolve the crux. In specific, a two-stage disentanglement framework is proposed to acquire domain-invariant feature representations at both image and instance levels. The holistic design addresses the limitations of existing OCDA approaches which struggle to capture instance-wise variations. Two regularization strategies are specifically devised herein to leverage the rich subpartition-specific characteristics in histopathology images and facilitate subdomain decomposition. Moreover, we propose a dual-branch nucleus shape and structure preserving module to prevent nucleus over-generation and deformation in the synthesized images. Experimental results on both cross-modality and cross-stain scenarios over a broad range of diverse datasets demonstrate the superiority of our method compared with state-of-the-art UDA and OCDA methods.Graphical abstract

Correlation enhanced distribution adaptation for prediction of fall risk

With technological advancements in diagnostic imaging, smart sensing, and wearables, a multitude of heterogeneous sources or modalities are available to proactively monitor the health of the elderly. Due to the increasing risks of falls among older adults, an early diagnosis tool is crucial to prevent future falls. However, during the early stage of diagnosis, there is often limited or no labeled data (expert-confirmed diagnostic information) available in the target domain (new cohort) to determine the proper treatment for older adults. Instead, there are multiple related but non-identical domain data with labels from the existing cohort or different institutions. Integrating different data sources with labeled and unlabeled samples to predict a patient's condition poses a significant challenge. Traditional machine learning models assume that data for new patients follow a similar distribution. If the data does not satisfy this assumption, the trained models do not achieve the expected accuracy, leading to potential misdiagnosing risks. To address this issue, we utilize domain adaptation (DA) techniques, which employ labeled data from one or more related source domains. These DA techniques promise to tackle discrepancies in multiple data sources and achieve a robust diagnosis for new patients. In our research, we have developed an unsupervised DA model to align two domains by creating a domain-invariant feature representation. Subsequently, we have built a robust fall-risk prediction model based on these new feature representations. The results from simulation studies and real-world applications demonstrate that our proposed approach outperforms existing models.

Multi-domain Adversarial Variational Bayesian Inference for Domain Generalization

Domain generalization aims to learn common knowledge from multiple observed source domains and transfer it to unseen target domains, e.g. the object recognition in varieties of visual environments. Traditional domain generalization methods aim to learn the feature representation of the raw data with its distribution invariant across domains. This relies on the assumption that the two posterior distributions (the distributions of the label given the feature distribution and given the raw data) are stable in different domains. However, this does not always hold in many practical situations. In this paper, we relax the above assumption by permitting the posterior distribution of the label given the raw data changes in difference domains, and thus focuses on a more realistic learning problem that infers the conditional domain-invariant feature representation. Specifically, a multi-domain adversarial variational Bayesian inference approach is proposed to minimize the inter-domain discrepancy of the conditional distributions of the feature given the label. Besides, it is imposed by the constraints from the adversarial learning and feedback mechanism to enhance the condition invariant feature representation. The extensive experiments on two datasets demonstrate the effectiveness of our approach, as well as the state-of-the-art performance comparing with thirteen methods.

MSMTSeg: Multi-Stained Multi-Tissue Segmentation of Kidney Histology Images via Generative Self-Supervised Meta-Learning Framework.

Accurately diagnosing chronic kidney disease requires pathologists to assess the structure of multiple tissues under different stains, a process that is timeconsuming and labor-intensive. Current AI-based methods for automatic structure assessment, like segmentation, often demand extensive manual annotation and focus on single stain domain. To address these challenges, we introduce MSMTSeg, a generative self-supervised meta-learning framework for multi-stained multi-tissue segmentation in renal biopsy whole slide images (WSIs). MSMTSeg incorporates multiple stain transform models for style translation of inter-stain domains, a self-supervision module for obtaining pre-trained models with the domain-specific feature representation, and a meta-learning strategy that leverages generated virtual data and pre-trained models to learn the domain-invariant feature representation across multiple stains, thereby enhancing segmentation performance. Experimental results demonstrate that MSMTSeg achieves superior and robust performance, with mDSC of 0.836 and mIoU of 0.718 for multiple tissues under different stains, using only one annotated training sample for each stain. Our ablation study confirms the effectiveness of each component, positioning MSMTSeg ahead of classic advanced segmentation networks, recent few-shot segmentation methods, and unsupervised domain adaptation methods. In conclusion, our proposed few-shot cross-domain technology offers a feasible and cost-effective solution for multi-stained renal histology segmentation, providing convenient assistance to pathologists in clinical practice. The source code and conditionally accessible data are available at https://github.com/SnowRain510/MSMTSeg.

Selective transfer subspace learning for small-footprint end-to-end cross-domain keyword spotting

In small-footprint end-to-end keyword spotting, it is often expensive and time-consuming to acquire sufficient labels in various speech scenarios. To overcome this problem, transfer learning leverages the rich knowledge of the auxiliary domain to annotate the unlabeled target data. However, most existing transfer learning methods typically learn a domain-invariant feature representation while ignoring the negative transfer problem. In this paper, we propose a new and general cross-domain keyword spotting framework called selective transfer subspace learning (STSL) that avoid negative transfer and dramatically improve the accuracy for cross-domain keyword spotting by actively selecting appropriate source samples. Specifically, STSL first aligns geometrical relationship and weighted distribution discrepancy to learn a domain-invariant projection subspace. Then, it actively selects appropriate source samples that are similar to the target domain for transfer learning to avoid negative transfer. Finally, we formulate a minimization problem that alternately optimizes the projection subspace and source active selection, giving an effective optimization. Experimental results on 10 groups of cross-domain keyword spotting tasks show that our STSL outperforms some state-of-the-art transfer learning methods and no transfer learning methods.

Regularized Hypothesis-Induced Wasserstein Divergence for unsupervised domain adaptation

Unsupervised domain adaptation (UDA) aims to align the data distributions of source and target domains to learn domain-invariant feature representations. The performance of such representations heavily relies on the measurement of the discrepancy between the distributions. Recent UDA methods utilize hypothesis-based measurements, also known as classifier-induced divergences, to quantify the differences between the two domain distributions, for instance, HΔH-divergence. Nevertheless, classifier-induced divergences face several challenges, including the fake covariate alignment problem and the inconsistency problem, rendering them inadequate for discerning dissimilarities between different categories of output by the classifier. This paper proposes a novel method called Regularized Hypothesis-Induced Wasserstein Divergence-Based Representation Learning for Unsupervised Domain Adaptation (RHWD) to address these issues. Specifically, RHWD introduces a new hypothesis-induced Wasserstein divergence (HWD) to effectively quantify the differences between the two distributions, aiming to overcome the fake covariate alignment problem. Furthermore, RHWD includes a consistency regularization term to mitigate the inconsistency problem. We train RHWD adversarially in three iterative steps. Additionally, we provide empirical evaluations on standard benchmarks, demonstrating the effectiveness of our proposed RHWD.

Adaptive Refining-Aggregation-Separation Framework for Unsupervised Domain Adaptation Semantic Segmentation

Unsupervised domain adaptation has attracted widespread attention as a promising method to solve the labeling difficulties of semantic segmentation tasks. It trains a segmentation network for unlabeled real target images using easily available labeled virtual source images. To improve performance, clustering is used to obtain domain-invariant feature representations. However, most clustering-based methods indiscriminately cluster all features mapped by category from both domains, causing the centroid shift and affecting the generation of discriminative features. We propose a novel clustering-based method that uses an adaptive refining-aggregation-separation framework, which learns the discriminative features by designing different adaptive schemes for different domains and features. The clustering does not require any tunable thresholds. To estimate more accurate domain-invariant centroids, we design different ways to guide the adaptive refinement of different domain features. A critic is proposed to directly evaluate the confidence of target features to solve the absence of target labels. We introduce a domain-balanced aggregation loss and two adaptive separation losses for distance and similarity respectively, which can discriminate clustering features by combining the refinement strategy to improve segmentation performance. Experimental results on GTA5→Cityscapes and SYNTHIA→Cityscapes benchmarks show that our method outperforms existing state-of-the-art methods.

Reinforced Adaptation Network for Partial Domain Adaptation

Domain adaptation enables generalized learning in new environments by transferring knowledge from label-rich source domains to label-scarce target domains. As a more realistic extension, partial domain adaptation (PDA) relaxes the assumption of fully shared label space, and instead deals with the scenario where the target label space is a subset of the source label space. In this paper, we propose a Reinforced Adaptation Network (RAN) to address the challenging PDA problem. Specifically, a deep reinforcement learning model is proposed to learn source data selection policies. Meanwhile, a domain adaptation model is presented to simultaneously determine rewards and learn domain-invariant feature representations. By combining reinforcement learning and domain adaptation techniques, the proposed network alleviates negative transfer by automatically filtering out less relevant source data and promotes positive transfer by minimizing the distribution discrepancy across domains. Experiments on three benchmark datasets demonstrate that RAN consistently outperforms seventeen existing state-of-the-art methods by a large margin.

Multi-view Feature Learning for the Over-penalty in Adversarial Domain Adaptation

Abstract Domain adaptation aims to transfer knowledge from the labeled source domain to an unlabeled target domain that follows a similar but different distribution. Recently, adversarial-based methods have achieved remarkable success due to the excellent performance of domain-invariant feature presentation learning. However, the adversarial methods learn the transferability at the expense of the discriminability in feature representation, leading to low generalization to the target domain. To this end, we propose a Multi-view Feature Learning method for the Overpenalty in Adversarial Domain Adaptation. Specifically, multi-view representation learning is proposed to enrich the discriminative information contained in domain-invariant feature representation, which will counter the over-penalty for discriminability in adversarial training. Besides, the class distribution in the intra-domain is proposed to replace that in the inter-domain to capture more discriminative information in the learning of transferrable features. Extensive experiments show that our method can improve the discriminability while maintaining transferability and exceeds the most advanced methods in the domain adaptation benchmark datasets.

Unsupervised domain adaptation for the detection of cardiomegaly in cross-domain chest X-ray images.

In recent years, several deep learning approaches have been successfully applied in the field of medical image analysis. More specifically, different deep neural network architectures have been proposed and assessed for the detection of various pathologies based on chest X-ray images. While the performed assessments have shown very promising results, most of them consist in training and evaluating the performance of the proposed approaches on a single data set. However, the generalization of such models is quite limited in a cross-domain setting, since a significant performance degradation can be observed when these models are evaluated on data sets stemming from different medical centers or recorded under different protocols. The performance degradation is mostly caused by the domain shift between the training set and the evaluation set. To alleviate this problem, different unsupervised domain adaptation approaches are proposed and evaluated in the current work, for the detection of cardiomegaly based on chest X-ray images, in a cross-domain setting. The proposed approaches generate domain invariant feature representations by adapting the parameters of a model optimized on a large set of labeled samples, to a set of unlabeled images stemming from a different data set. The performed evaluation points to the effectiveness of the proposed approaches, since the adapted models outperform optimized models which are directly applied to the evaluation sets without any form of domain adaptation.

Deep Domain-Adversarial Anomaly Detection With One-Class Transfer Learning

Despite the big success of transfer learning techniques in anomaly detection, it is still challenging to achieve good transition of detection rules merely based on the preferred data in the anomaly detection with one-class classification, especially for the data with a large distribution difference. To address this challenge, a novel deep one-class transfer learning algorithm with domain-adversarial training is proposed in this paper. First, by integrating a hypersphere adaptation constraint into domain-adversarial neural network, a new hypersphere adversarial training mechanism is designed. Second, an alternative optimization method is derived to seek the optimal network parameters while pushing the hyperspheres built in the source domain and target domain to be as identical as possible. Through transferring one-class detection rule in the adaptive extraction of domain-invariant feature representation, the end-to-end anomaly detection with one-class classification is then enhanced. Furthermore, a theoretical analysis about the model reliability, as well as the strategy of avoiding invalid and negative transfer, is provided. Experiments are conducted on two typical anomaly detection problems, i.e., image recognition detection and online early fault detection of rolling bearings. The results demonstrate that the proposed algorithm outperforms the state-of-the-art methods in terms of detection accuracy and robustness.

Unsupervised model adaptation for multivariate calibration by domain adaptation-regularization based kernel partial least square

In chemometrics, calibration model adaptation is desired when training- and test-samples come from different distributions. Domain-invariant feature representation is currently a successful strategy to realize model adaptation and has received wide attention. The paper presents a nonlinear unsupervised model adaptation method termed as domain adaption regularization-based kernel partial least squares regression (DarKPLS). DarKPLS aims to minimize the source and target distributions in a low-dimensional latent space projected from the reproducing kernel Hilbert space (RKHS) generated with the labeled source data and unlabeled target data. Specially, the distributional means and variances between source and target latent variables are aligned in the RKHS. By extending existing domain invariant partial least square regression (di-PLS) with the projected maximum mean discrepancy (PMMD) to reduce the mean discrepancy in the RKHS further, DarKPLS could realize fine-grained domain alignment that further improves the adaptation performance. DarKPLS is applied to the γ-polyglutamic acid fermentation dataset, tobacco dataset and corn dataset, and it demonstrates improved prediction results in comparison with No adaptation partial least squares (PLS), null augmented regression (NAR), extended linear joint trained framework (ExtJT), scatter component analysis (SCA) and domain-invariant iterative partial least squares (DIPALS).

Gradual Domain Adaptation with Pseudo-Label Denoising for SAR Target Recognition When Using Only Synthetic Data for Training

Because of the high cost of data acquisition in synthetic aperture radar (SAR) target recognition, the application of synthetic (simulated) SAR data is becoming increasingly popular. Our study explores the problems encountered when training fully on synthetic data and testing on measured (real) data, and the distribution gap between synthetic and measured SAR data affects recognition performance under the circumstances. We propose a gradual domain adaptation recognition framework with pseudo-label denoising to solve this problem. As a warm-up, the feature alignment classification network is trained to learn the domain-invariant feature representation and obtain a relatively satisfactory recognition result. Then, we utilize the self-training method for further improvement. Some pseudo-labeled data are selected to fine-tune the network, narrowing the distribution difference between the training data and test data for each category. However, the pseudo-labels are inevitably noisy, and the wrong ones may deteriorate the classifier’s performance during fine-tuning iterations. Thus, we conduct pseudo-label denoising to eliminate some noisy pseudo-labels and improve the trained classifier’s robustness. We perform pseudo-label denoising based on the image similarity to keep the label consistent between the image and feature domains. We conduct extensive experiments on the newly published SAMPLE dataset, and we design two training scenarios to verify the proposed framework. For Training Scenario I, the framework matches the result of neural architecture searching and achieves 96.46% average accuracy. For Training Scenario II, the framework outperforms the results of other existing methods and achieves 97.36% average accuracy. These results illustrate the superiority of our framework, which can reach state-of-the-art recognition levels with appropriate stability.