Learning Domain-invariant Features Research Articles

Unsupervised and Semi-Supervised Robust Spherical Space Domain Adaptation.

Adversarial domain adaptation has been an effective approach for learning domain-invariant features by adversarial training. In this paper, we propose a novel adversarial domain adaptation approach defined in the spherical feature space, in which we define spherical classifier for label prediction and spherical domain discriminator for discriminating domain labels. In the spherical feature space, we develop a spherical robust pseudo-label loss to utilize pseudo-labels robustly, which weights the importance of the estimated labels of target domain data by the posterior probability of correct labeling, modeled by the Gaussian-uniform mixture model in the spherical space. Our proposed approach can be generally applied to both unsupervised and semi-supervised domain adaptation settings. In particular, to tackle the semi-supervised domain adaptation setting where a few labeled target domain data are available for training, we propose a novel reweighted adversarial training strategy for effectively reducing the intra-domain discrepancy within the target domain. We also present theoretical analysis for the proposed method based on the domain adaptation theory. Extensive experiments are conducted on multiple benchmarks for object recognition, digit recognition, and face recognition. The results show that our method either surpasses or is competitive compared with the recent methods for both unsupervised and semi-supervised domain adaptation. Ablation studies also confirm the effectiveness of the spherical classifier, spherical discriminator, spherical robust pseudo-label loss, and reweighted adversarial training strategy.

A domain adaptation network with feature scale preservation for remaining useful life prediction of rolling bearings under variable operating conditions

Transfer learning and domain adaptation (DA) methods have been utilized in bearing prognostic and health management, but most of the current DA methods do not take into account the feature scale change of degraded features when aligning the feature distribution, and these methods are more suitable for the classification problem, which is more robust to the feature scale change. However, they perform poorly in regression problems. In addition, most of the remaining useful life (RUL) prediction methods require preprocessing such as statistical feature extraction on the signal, which makes the prediction process complicated. To solve the above problems, a DA method based on the representation subspace distance (RSD) is proposed for predicting the bearing RUL under different operating conditions. First, the proposed convolutional neural network (CNN) self-attention (SA) long short term memory network model is utilized to extract the deep features from the original signal, which overcomes the limitations of the CNN in extracting time series. Then, the RSD in the Riemannian geometry of the Grassmann manifold is proposed as a domain transfer loss to learn domain invariant features. The modified method can align the feature distribution of the source domain and the target domain without changing the feature scale. At the same time, the bases mismatch penalization is introduced to avoid destroying the semantic information of the features in the process of domain alignment. Finally, the effectiveness of the proposed method is verified by experiments on four types of transfer tasks, and its superiority is also demonstrated by comparison with other advanced methods.

Fine-Grained Self-Supervision for Generalizable Semantic Segmentation

Unsupervised domain adaptative semantic segmentation is a powerful solution for the distribution shift problem between the source and target domains. However, such methods need specified target domain data that may be unavailable in actual applications due to excess expensive collection. Generalizable semantic segmentation as a new paradigm appears in recent research, which aims to generalize well on distinct unseen domains only using source domain data. The existing methods focus on learning domain-invariant features by using global distribution alignment strategies, which may lead to a decreased discriminability of the model. To cope with this challenge, we propose a fine-grained self-supervision (FGSS) framework for generalizable semantic segmentation that takes into account both discriminability and generalizability from the perspective of the intra-class relationship. The FGSS framework contains single-view and multi-view versions. In the single-view version, we propose a fine-grained self-supervision strategy to distinguish the sub-parts of the semantic class for better class discriminability. In the multi-view version, we propose a class prototype feature enhancement strategy to generate another view (i.e. another representation of the original representation). Then, we propose a multi-view mutual supervision loss to enforce consistency between different views and further enhance the generalizability of the model. Experimental results on five widely-used datasets, i.e., GTAV, SYNTHIA, BDD100K, Cityscapes, and Mapillary, demonstrate that our FGSS framework achieves superior performance compared to state-of-the-art methods.

Domain adaptation with label-aligned sampling (DALAS) for cross-domain fault diagnosis of rotating machinery under class imbalance

Though existing cross-domain fault-diagnosis methods have shown promising results under domain shift conditions, existing approaches are only valid for class-balanced data. However, situations of class imbalance are inevitable in industrial fields, due to the difficulty in acquiring fault data from in-use machinery. Thus, if existing cross-domain fault diagnosis approaches are directly applied when domain shift and class imbalance coexist, performance degradation can occur. Thus, this research develops a domain adversarial learning network for class-imbalanced data to address situations where class imbalance and domain shift coexist; this situation is referred to as the problem of class imbalance domain adaptation (CIDA). In the proposed method, domain adversarial training is implemented for learning domain-invariant features by reducing the domain shift, and a label-aligned sampling strategy is utilized to deal with the class imbalance. In addition, for further performance enhancement of label-aligned sampling by increasing the accuracy of pseudo labels, metric learning is introduced to enhance the feature distinctiveness by expanding the distance of samples from different classes while decreasing the distance of samples from the same class. The efficiency of the proposed method is validated by applying it to various circumstances using two bearing datasets. The proposed method demonstrates superior performance compared to conventional algorithms in addressing the CIDA problem, according to quantitative and qualitative evaluations.

A multi-domain mixture density network for tool wear prediction under multiple machining conditions

Accurate tool wear prediction is an essential task in machining processes because it helps to schedule efficient tool maintenance and maximise the tool's useful life, thus contributing to sustainable production via zero defect manufacturing (ZDM). However, there are limitations to existing methods; these cannot be used under multiple machining conditions, which is common practice. This problem not only hinders accurate tool wear monitoring but also necessitates the use of multiple models, which increases operation and modelling costs. Therefore, the multi-domain learning problem should be addressed to enable tool wear prediction under various machining conditions. To this end, this work presents a novel method, a multi-domain mixture density network (MD2N). In particular, a Bayesian learning-based feature extractor is proposed to learn domain-invariant representations. Additionally, an adversarial learning approach is developed to lead the predictive model in learning domain-invariant features. Lastly, a mixture density network-based predictor is used to generate probabilistic tool wear outputs. Experiments that use datasets from real-world milling processes under multiple conditions prove the proposed method's promising efficacy, with the best mean absolute error (MAE), root mean squared error (RMSE), and mean absolute percentage error (MAPE) of 2.1748, 5.6422, and 0.0350, respectively, indicating the ability to learn multi-domain representations.

Boosting domain generalization by domain-aware knowledge distillation

Deep neural networks often suffer performance degradation when the testing data distribution differs significantly from the training data distribution. To address this problem, most domain generalization (DG) approaches focus on learning domain-invariant features from multiple source domains. However, enforcing model invariance among different domains can also result in information that is not shared across training domains being discarded, even though it may still be relevant to unseen domains. We argue that a generalized model should be discriminative towards comprehensive features across different domains. To avoid unexpected information loss, we propose a two-stage learning scheme, named Domain-aware Knowledge Distillation (DAKD). In the first stage, we pre-train a parameter-efficient multi-expert model, where each expert is trained to be responsible for a specific source domain and preserve the domain-specific features. In the second stage, following the traditional knowledge distillation formulation, we train a new student network assisted by the pre-trained experts, where the prediction of the student for a certain domain is regularized by the soft output of the corresponding expert. A comprehensive ablation study and analysis highlight that our distilled model can preserve more source-domain-specific features, and obtain higher accuracy on unseen domains compared to its counterpart without distillation. We discuss the relationship between the generalization risk bound theory and our method. The upper bound is reduced by mitigating source-target domain discrepancy and reducing risk on source domains. Experiments demonstrate that our method provides state-of-the-art performance on PACS and other more challenging datasets such as Office-Home and DomainNet. The source code is available at https://github.com/ZZQ321/DAKD.git

Classification Certainty Maximization for Unsupervised Domain Adaptation

The bi-classifier paradigm is a common practice in unsupervised domain adaptation (UDA), where two classifiers are leveraged to guide the model to learn domain invariant features. Previous approaches only focused on the consistency of the outputs between classifiers, but ignored the classification certainty of each classifier. Therefore, existing methods in some cases may mislead the classifiers into the wrong direction of ambiguous outputs and subsequently undermine the discriminability. To challenge this problem, in this paper we propose Classification Certainty Maximization (CCM) which considers both the joint certainty between classifiers and the individual certainty of each classifier via a novel formulation, and derived their optimal weight ratios by theoretical derivation from the perspective of gradient. In addition, we also propose a dynamic centroid update strategy to mitigate the domain gaps at the feature level. Extensive experiments on four widely used UDA datasets show that CCM performs better than the existing state-of-the-art domain adaptation methods. Notably, our dynamic centroid update strategy can be used as a plug-and-play module for existing bi-classifier domain adaptation methods to boost classification accuracy.

Adaptive Texture Filtering for Single-Domain Generalized Segmentation

Domain generalization in semantic segmentation aims to alleviate the performance degradation on unseen domains through learning domain-invariant features. Existing methods diversify images in the source domain by adding complex or even abnormal textures to reduce the sensitivity to domain-specific features. However, these approaches depends heavily on the richness of the texture bank and training them can be time-consuming. In contrast to importing textures arbitrarily or augmenting styles randomly, we focus on the single source domain itself to achieve the generalization. In this paper, we present a novel adaptive texture filtering mechanism to suppress the influence of texture without using augmentation, thus eliminating the interference of domain-specific features. Further, we design a hierarchical guidance generalization network equipped with structure-guided enhancement modules, which purpose to learn the domain-invariant generalized knowledge. Extensive experiments together with ablation studies on widely-used datasets are conducted to verify the effectiveness of the proposed model, and reveal its superiority over other state-of-the-art alternatives.

Deep discriminative causal domain generalization

Domain generalization aims to generalize knowledge learned from multi-domains sources to a target domain whose statistical distribution is unknown. The mainstream approach to domain generalization involves learning domain-invariant features by aligning them across the domain distribution. However, these methods usually neglect the causal relationship between the instances and class labels; thus, the domain generalization model may learn the features that are not causally related to the label predictions and lead to potential misclassification. To address this problem, we propose a deep discriminative causal domain generalization (DDCDG), which can effectively learn causal feature representations. Specifically, we first imitate various possible intervention styles by applying an enriched data augmentation technique while preserving causal features relevant to the prediction task. We then designed a new regularization term for causal feature disentanglement, aiming to separate the causal and non-causal features. Additionally, we introduce centre alignment to approximate the representations of cross-domain homogeneous objects and enhance their discriminative ability. Extensive cross-domain visual recognition experiments were conducted, and the results show that our method can achieve a significant improvement in classification accuracy when compared with existing state-of-the-art methods on the Domainbed benchmark.

Active consistency network for multi-source domain generalization in brain tumor segmentation

Recent advances in deep learning for brain tumor segmentation demonstrate good performance when the training and test data share the same distribution. However, medical images from different medical institutions often face distribution shifts in clinical applications. To deal with domain shift in multi-source domain generalization, we propose Active Consistency Network (ACN) with a single encoder and dual decoders to learn domain invariant features. A causal feature learning strategy and uncertainty-based refinery strategy are proposed to obtain reliable soft labels of reliable samples (Active Consistency Labels, ACL). Active consistency labels remove the noisy and unreliable samples for model consistency. Causality-based Random Masking (CRM) is proposed to obtain more stable causal features by adversarial learning. Causal features help achieve stable uncertainty estimation. Besides, two decoders are supervised by the active consistency labels learned from each other’s decoders, which can help decoders learn domain invariant features. Experimental results show that ACN utilizes causal feature learning and active consistency labels improving segmentation performance on multi-institutional brain tumor segmentation and achieving state-of-the-art performance compared to other domain generalization methods.

CIFair: Constructing continuous domains of invariant features for image fair classifications

With learning models widely deployed in daily life, researchers are discovering that many of them generate discriminatory predictions towards sensitive attributes such as gender or race. A common way of tackling this problem is to learn fair features without sensitive information by removing features. This method requires extra heads for attribute predictions and eliminates the information via adversarial settings. Although the processes can impose fairness, they reduce the accuracy of the models relative to the originals. In this research, we generate continuous domains containing information from different subgroups with mixup operations. We then learn domain invariant features with similarity constraints. Different from previous methods, the proposed method, CIFair, can learn fair features without feature removal operations or task-irrelevant learning objectives. Finally, we evaluated our approach on the CelebA dataset with different sensitive attributes under multiple settings. All experimental results demonstrate that CIFair is able to impose enforce better fair prediction results than previous methods while maintaining model accuracy performance.

Toward a blind image quality evaluator in the wild by learning beyond human opinion scores

Nowadays, most existing blind image quality assessment (BIQA) models inthewild heavily rely on human ratings, which are extraordinarily labor-expensive to collect. Here, we propose an opinion−free BIQA method that learns from multiple annotators to assess the perceptual quality of images captured in the wild. Specifically, we first synthesize distorted images based on the pristine counterparts. We then randomly assemble a set of image pairs from the synthetic images, and use a group of IQA models to assign pseudo-binary labels for each pair indicating which image has higher quality as the supervisory signal. Based on the newly established pseudo-labeled dataset, we train a deep neural network (DNN)-based BIQA model to rank the perceptual quality, optimized for consistency with the binary rank labels. Since there exists domain shift, e.g., distortion shift and content shift, between the synthetic and in-the-wild images, we leverage two ways to alleviate this issue. First, the simulated distortions should be similar to authentic distortions as much as possible. Second, an unsupervised domain adaptation (UDA) module is further applied to encourage learning domain-invariant features between two domains. Extensive experiments demonstrate the effectiveness of our proposed opinion−free BIQA model, yielding SOTA performance in terms of correlation with human opinion scores, as well as gMAD competition. Our code is available at: https://github.com/wangzhihua520/OF_BIQA.

DMSTL: A Deep Multi-Scale Transfer Learning Framework for Unsupervised Cross-Position Human Activity Recognition

Human activity recognition (HAR) based on wearable sensors has been a prosperous research topic in recent years. Considering that the sensor may be worn at diverse body positions, obtaining enough labeled human activity data for each body-worn position is usually expensive and labor-intensive. Furthermore, the variability and diversity of data distribution induced by different body-worn positions make the HAR model trained on data collected from one body position perform poorly for other body-worn positions. To achieve accurate HAR with low labeling cost, which we call unsupervised cross-position HAR, in this article, we propose a deep multiscale transfer learning (DMSTL) model. In the model, we first introduce an unsupervised source selection method to select the most similar source domain for transferring domain knowledge. Then, we develop a multiscale spatial–temporal Net (MSSTNet) to learn comprehensive multimodal representations from multiple feature subspaces. Finally, we design the category-level adaptation module and the domain-level adversarial module for learning domain-invariant features. We conduct extensive experiments on three public HAR data sets and demonstrate the reasonable generalization performance of the DMSTL, which remarkably outperforms other state-of-the-art baselines.

Multi-Representation Domain Adaptation Network with Duplex Adversarial Learning for Hot-Rolling Mill Fault Diagnosis.

The multi-process manufacturing of steel rolling products requires the cooperation of complicated and variable rolling conditions. Such conditions pose challenges to the fault diagnosis of the key equipment of the rolling mill. The development of transfer learning has alleviated the problem of fault diagnosis under variable working conditions to a certain extent. However, existing diagnosis methods based on transfer learning only consider the distribution alignment from a single representation, which may only transfer part of the state knowledge and generate fuzzy decision boundaries. Therefore, this paper proposes a multi-representation domain adaptation network with duplex adversarial learning for hot rolling mill fault diagnosis. First, a multi-representation network structure is designed to extract rolling mill equipment status information from multiple perspectives. Then, the domain adversarial strategy is adopted to match the source and target domains of each pair of representations for learning domain-invariant features from multiple representation networks. In addition, the maximum classifier discrepancy adversarial algorithm is adopted to generate target features that are close to the source support, thereby forming a robust decision boundary. Finally, the average value of the predicted probabilities of the two classifiers is used as the final diagnostic result. Extensive experiments are conducted on an experimental platform of a four-high hot rolling mill to collect the fault state data of the reduction gearbox and roll bearing. The experimental results reveal that the method can effectively realize the fault diagnosis of rolling mill equipment under variable working conditions and can achieve average diagnostic rates of up to 99.15% and 99.40% on the data sets of the rolling mill gearbox and bearing, which are respectively 2.19% and 1.93% higher than the rates achieved by the most competitive method.

Deep transfer learning based on Bi-LSTM and attention for remaining useful life prediction of rolling bearing

Many transfer learning methods focus on training models between domains with large differences. However, the data feature distribution varies greatly in different bearing degradation processes, which affects the prediction accuracy of Remaining useful life (RUL). To solve this problem, a novel method for RUL prediction with more refined transfer is proposed, which added failure behavior judgment. Firstly, a failure behavior judgment method is proposed by using the convolutional autoencoder (CAE) and Pearson correlation coefficient to determine whether the bearing fails gradually or suddenly. Then, a multi-channel transfer network is proposed for extracting multi-scale features of bearing degradation. Each channel uses convolutional network and bidirectional long short-term memory (Bi-LSTM) to extract global and temporal information. To reduce the difference in feature distribution between the training and test bearing data, a domain adaptive structure is added after feature fusion in each channel to enable the model to learn domain invariant features. By applying this method to experimental data and comparing it with other methods, the superiority and effectiveness of the proposed method are verified.

Research on fault diagnosis method of planetary gearbox based on dynamic simulation and deep transfer learning

To address the issue of not having enough labeled fault data for planetary gearboxes in actual production, this research develops a simulation data-driven deep transfer learning fault diagnosis method that applies fault diagnosis knowledge from a dynamic simulation model to an actual planetary gearbox. Massive amounts of different fault simulation data are collected by creating a dynamic simulation model of a planetary gearbox. A fresh deep transfer learning network model is built by fusing one-dimensional convolutional neural networks, attention mechanisms, and domain adaptation methods. The network model is used to learn domain invariant features from simulated data, thereby enabling fault diagnosis on real data. The fault diagnosis experiment is verified by using the Drivetrain Diagnostics Simulator test bench. The validity of the proposed means is evaluated by comparing the diagnostic accuracy of various means on various diagnostic tasks.

Deep Adversarial Subdomain Adaptation Network for Intelligent Fault Diagnosis

Recently, domain adaptation has received extensive attention for solving intelligent fault diagnosis problems. It aims to reduce the distribution discrepancy between the source domain and target domain through learning domain-invariant features. However, most existing domain adaptation methods mainly focus on global domain adaptation and overlook subdomain adaptation, which results in the loss of fine-grained information and discriminative features. To address this problem, in this article, a deep adversarial subdomain adaptation network is proposed. This network aligns the relevant distributions of subdomains by minimizing the local maximum mean discrepancy loss of the same categories in the source domain and target domain. Under the constraints of global domain adaptation and subdomain adaptation, the distribution discrepancy is reduced from the domain and category levels. Four transfer tasks under different machine rotating speeds and six transfer tasks on different but related machines were used to evaluate the effectiveness of the proposed method. The results demonstrated the robustness and superiority of the proposed method over five other methods.

Online diagnosis for rolling bearings based on multi-channel convolution and transfer learning

In recent years, the fault diagnosis methods based on deep learning have been widely applied. In practical engineering, there are great distribution differences between the training and testing data in the network, leading to low diagnosis reliability. Transfer learning can solve such problems by learning domain invariant features. In this paper, a multi-channel convolutional online transfer network model for rolling bearing fault diagnosis is proposed. In the model, the offline stage merges the time domain and frequency domain features of the original data. A three-channel dataset is constructed as input of the network. And the domain invariant features can be learnt by fully training the offline stage network model. The online model is initialized by the parameters transferred from the offline network. The model also designs an online update strategy according to the prediction error. So that the model can adapt to new data, and finally realize the online diagnosis of the rolling bearing fault state. The validity and accuracy of the model are verified by the different laboratory measurement of rolling bearing operating datasets.

An adversarial transfer network with supervised metric for remaining useful life prediction of rolling bearing under multiple working conditions

Many existing domain adaptation-based methods try to derive domain invariant features to address domain shifts and obtain satisfactory remaining useful life (RUL) of bearings under multiple working conditions. However, most methods may not consider local semantics about degradation features and mutual information from target-specific data when aligning distribution discrepancies, thus resulting in limitations. Additionally, the use of contrastive learning to maintain mutual information may introduce unstable negative samples. To overcome these issues, a metric adversarial domain adaptation approach (MADA) is proposed to evaluate the bearing RULs under multiple working conditions. More specifically, an adversarial domain adaptation architecture with a supervised positive contrastive module is developed to consider mutual information without a negative sample, further learning domain invariant features. Also, the dual self-attention module is designed to extract multi-scale contextual semantics between degradation features. Meanwhile, extensive experiments are conducted in twelve cross-domain scenarios for two bearing cases. The experimental results show that the proposed method is more competitive.

Joint bi-adversarial learning for unsupervised domain adaptation

An important challenge of unsupervised domain adaptation (UDA) is how to sufficiently utilize the structure and information of the data distribution, so as to exploit the source domain knowledge for a more accurate classification of the unlabeled target domain. Currently, much research work has been devoted to UDA. However, existing works have mostly considered only distribution alignment or learning domain invariant features by adversarial techniques, ignoring feature processing and intra-domain category information. To this end, we design a new cross-domain discrepancy metric, namely joint distribution for maximum mean discrepancy (JD-MMD), and propose a deep unsupervised domain adaptation learning method, namely joint bi-adversarial learning for unsupervised domain adaptation (JBL-UDA). Specifically, JD-MMD measures cross-domain divergence in terms of both discrepancy and relevance by preserving cross-domain joint distribution discrepancy, as well as their class discriminability. Then, with such divergence measure, JBL-UDA models with two learning modalities, one is founded by the bi-adversarial learning from domains and classes implicitly, while the other explicitly addresses domains and classes alignment via the JD-MMD metric. Besides, JBL-UDA explores structural prior knowledge from data classes and domains to generate class-discriminative and domain-invariant representations. Finally, extensive evaluations exhibit state-of-the-art accuracy of the proposed methodology.