Extract Domain-invariant Features Research Articles

Bearing fault diagnosis based on partial domain adaptation adversarial network

The existing fault diagnosis algorithm based on domain adaptation solves the problem of degradation of model diagnosis performance due to different data distributions under variable working conditions and cross-machine conditions, and its excellent fault diagnosis performance relies on the assumption that the fault category space of source and target domains is the same; however, it is difficult to meet the above assumption in practical application scenarios. For this reason, focusing on the matter of imbalance within the fault category, this paper proposes a novel unsupervised partial domain adaptational fault diagnosis method—a partial domain adaptation adversarial network (PDAAN). On the one hand, it uses the source domain fault samples to expand the target domain and promotes the effective alignment of the fault feature area of the source domain and the target domain, in order for the model to effectively extract domain invariant features; on the other hand, class-level weights and weighted entropy weights are introduced into the loss function to suppress the uncertainty within the transfer process and avoid negative transfer of the model. Finally, experiments are conducted in the case of variable working conditions and cross-mechanical devices, and it is confirmed that the PDAAN model has high recognition accuracy in the case of class space asymmetry.

Domain Adaptation on Multiple Cloud Recognition From Different Types of Meteorological Satellite

Meteorological satellites have become an indispensable meteorological tool for earth observation, as aiding in areas such as cloud detection, which has important guiding significance for maritime activities. However, it is time-consuming and labor-intensive to obtain fine-grained annotations provided by artificial experience or mature satellite cloud products for multi-spectral maritime cloud imageries, especially when new satellites are launched. Moreover, due to the data discrepancy caused by different detection bands, existing models have inadequate generalization performance compared to new satellites, and some cannot be directly migrated. In this paper, to reduce the data distribution’s discrepancy, an approach is presented based on unsupervised domain adaption method for marine cloud detection task based on Himawari-8 satellite data as a source domain and Fengyun-4 satellite data as a target domain. The goal of the proposed method is to leverage the representation power of adversarial learning to extract domain-invariant features, consisting of a segmentation model, a feature extract model for target domain, and a domain discriminator. In addition, aiming to remedy the discrepancy of detection bands, a band mapping module is designed to implement consistency between different bands. The result of the experiments demonstrated the effectiveness of the proposed method with a 7% improvement compared with the comparative experiment. We also designed a series of statistical experiments on different satellite data to further study cloudy perception representation, including data visualization experiment and cloud type statistics.

An End-to-end Supervised Domain Adaptation Framework for Cross-Domain Change Detection

Change detection is a crucial but extremely challenging task in remote sensing image analysis, and much progress has been made with the rapid development of deep learning. However, most existing deep learning-based change detection methods try to elaborately design complicated neural networks with powerful feature representations. However, they ignore the universal domain shift induced by time-varying land cover changes, including luminance fluctuations and seasonal changes between pre-event and post-event images, thereby producing suboptimal results. In this paper, we propose an end-to-end supervised domain adaptation framework for cross-domain change detection named SDACD, to effectively alleviate the domain shift between bi-temporal images for better change predictions. Specifically, our SDACD presents collaborative adaptations from both image and feature perspectives with supervised learning. Image adaptation exploits generative adversarial learning with cycle-consistency constraints to perform cross-domain style transformation, which effectively narrows the domain gap in a two-side generation fashion. As for feature adaptation, we extract domain-invariant features to align different feature distributions in the feature space, which could further reduce the domain gap of cross-domain images. To further improve the performance, we combine three types of bi-temporal images for the final change prediction, including the initial input bi-temporal images and two generated bi-temporal images from the pre-event and post-event domains. Extensive experiments and analyses conducted on two benchmarks demonstrate the effectiveness and generalizability of our proposed framework. Notably, our framework pushes several representative baseline models up to new State-Of-The-Art records, achieving 97.34% and 92.36% on the CDD and WHU building datasets, respectively. The source code and models are publicly available at https://github.com/Perfect-You/SDACD.

Neuron Linear Transformation: Modeling the Domain Shift for Crowd Counting.

Cross-domain crowd counting (CDCC) is a hot topic due to its importance in public safety. The purpose of CDCC is to alleviate the domain shift between the source and target domain. Recently, typical methods attempt to extract domain-invariant features via image translation and adversarial learning. When it comes to specific tasks, we find that the domain shifts are reflected in model parameters' differences. To describe the domain gap directly at the parameter level, we propose a neuron linear transformation (NLT) method, exploiting domain factor and bias weights to learn the domain shift. Specifically, for a specific neuron of a source model, NLT exploits few labeled target data to learn domain shift parameters. Finally, the target neuron is generated via a linear transformation. Extensive experiments and analysis on six real-world data sets validate that NLT achieves top performance compared with other domain adaptation methods. An ablation study also shows that the NLT is robust and more effective than supervised and fine-tune training. Code is available at https://github.com/taohan10200/NLT.

Unsupervised domain adaptation via softmax-based prototype construction and adaptation

To mitigate the distribution difference between the source and target domains, there have been many unsupervised domain adaptation methods to achieve class-level alignment by aligning the prototypes of two domains. Since the labels of the target domain are unobserved, the target prototypes are constructed based on pseudo-labels. However, the inaccuracy of pseudo-labels may lead to biased prototypes, which introduce noise into the distribution alignment. Moreover, a shared feature extractor or two separate feature extractors are used to extract domain-invariant features. But the former is limited by the large domain gap and the latter increases the network parameters. To this end, we propose a Softmax-Based Prototype construction and Adaptation (SBPA) method, which constructs prototypes based on the softmax output of the classifier instead of ground-truth labels or pseudo-labels. SBPA performs domain-level alignment through adversarial training, and class-level alignment by aligning prototypes of the same class. In addition, SBPA contains a residual block to explicitly model the difference between the source and target domain features extracted by a shared feature extractor. We evaluate our method on four widely used datasets, and the results show that our method outperforms recent domain adaptation methods, especially on DomainNet, the hardest domain adaptation dataset by far.

Domain generalization using contrastive domain discrepancy optimization for interpretation-while-drilling

Lithology identification is a crucial component in reservoir characterization and geological exploration. Recently, lithology identification has been studied by many researchers using machine learning methods which can fit complex models. Most of the previous studies develop machine learning models in the premise of independent and identical distribution, i.e., the collected logging data has the same probability distribution even if it comes from different wells. In this way, the model trained on multiple drilled wells could be directly used in a new well. However, this assumption is invalid in practice since there is always a large difference in depositional environment and logging equipment. Therefore, the model trained on the interpreted data of drilled wells might not perform well in interpreting the logging data of a new well, especially while drilling. In this study, we investigate the domain generalization problem in lithology identification, where the logging data of the new unseen well is unavailable and the distribution varies greatly in different wells, to ensure the prediction accuracy on a new unseen well. To tackle such a problem, we propose a contrastive domain discrepancy based adversarial autoencoder (CDD-AAE) framework. Specifically, we utilize adversarial autoencoder to extract domain invariant features by imposing contrastive domain discrepancy objective in the latent space, i.e., the maximum mean discrepancy of inter-class is maximized, and the maximum mean discrepancy of intra-class is minimized. In addition, to avoid the over-fitting problem, a prior distribution is introduced to regularize the latent space in an adversarial training manner. Finally, we conduct extensive experiments on the logging data collected from Jiyang Depression, Bohai Bay Basin to certify the effectiveness of our proposed model. The experiments results demonstrate that our method achieves the best performance of 70.4% on average. Specifically, the macro recall of each target well are 68.7%, 75.3%, 65.4% and 72.2%, respectively. Besides, our method obtains relatively balanced predictions for all lithology types and achieves, indicating that our method can effectively mitigate the domain discrepancy and learn the discriminative features.

A dynamic-model-based fault diagnosis method for a wind turbine planetary gearbox using a deep learning network

The planetary gearbox is a critical part of wind turbines, and has great significance for their safety and reliability. Intelligent fault diagnosis methods for these gearboxes have made some achievements based on the availability of large quantities of labeled data. However, the data collected from the diagnosed devices are always unlabeled, and the acquisition of fault data from real gearboxes is time-consuming and laborious. As some gearbox faults can be conveniently simulated by a relatively precise dynamic model, the data from dynamic simulation containing some features are related to those from the actual machines. As a potential tool, transfer learning adapts a network trained in a source domain to its application in a target domain. Therefore, a novel fault diagnosis method combining transfer learning with dynamic model is proposed to identify the health conditions of planetary gearboxes. In the method, a modified lumped-parameter dynamic model of a planetary gear train is established to simulate the resultant vibration signal, while an optimized deep transfer learning network based on a one-dimensional convolutional neural network is built to extract domain-invariant features from different domains to achieve fault classification. Various groups of transfer diagnosis experiments of planetary gearboxes are carried out, and the experimental results demonstrate the effectiveness and the reliability of both the dynamic model and the proposed method.

The two-stage RUL prediction across operation conditions using deep transfer learning and insufficient degradation data

The remaining useful life (RUL) prediction provides an essential basis for improving mechanical equipment reliability. In practical application, the variant of working conditions and incomplete degradation data seriously deteriorate the performance of the prognostic models. In order to conquer this problem, a two-stage RUL prediction method is proposed for the cross-domain prognostic task with insufficient degradation data. At first, the two-level alarm mechanism is employed to detect the first predicting time (FPT) of each mechanical entity adaptively. Then, the deep separable convolutional network with the double transferable attention mechanism (DSCN-DTAM) is proposed to construct the cross-domain prognostic model. In DSCN-DTAM, multiple regularization strategies can guide the model to extract domain-invariant features, and the double transferable attention mechanism is designed to select the degradation information with high transferability. Finally, the proposed method is verified by multiple transfer prognostic tasks designed by two bearing datasets. Compared with other methods, the proposed method shows superior performance.

Learning to share by masking the non-shared for multi-domain sentiment classification

Multi-domain sentiment classification deals with the scenario where labeled data exists for multiple domains but is insufficient for training effective sentiment classifiers that work across domains. Thus, fully exploiting sentiment knowledge shared across domains is crucial for real-world applications. While many existing works try to extract domain-invariant features in high-dimensional space, such models fail to explicitly distinguish between shared and private features at the text level, which to some extent lacks interpretability. Based on the assumption that removing domain-related tokens from texts would help improve their domain invariance, we instead first transform original sentences to be domain-agnostic. To this end, we propose the BERTMasker model which explicitly masks domain-related words from texts, learns domain-invariant sentiment features from these domain-agnostic texts and uses those masked words to form domain-aware sentence representations. Empirical experiments on the benchmark multiple domain sentiment classification datasets demonstrate the effectiveness of our proposed model, which improves the accuracy on multi-domain and cross-domain settings by 1.91% and 3.31% respectively. Further analysis on masking proves that removing those domain-related and sentiment irrelevant tokens decreases texts’ domain separability, resulting in the performance degradation of a BERT-based domain classifier by over 12%.

Multi-Domain Weighted Transfer Adversarial Network for the Cross-Domain Intelligent Fault Diagnosis of Bearings

Transfer learning is a topic that has attracted attention for the intelligent fault diagnosis of bearings since it addresses bearing datasets that have different distributions. However, the traditional intelligent fault diagnosis methods based on transfer learning have the following two shortcomings. (1) The multi-mode structure characteristics of bearing datasets are neglected. (2) Some local regions of the bearing signals may not be suitable for transfer due to signal fluctuation. Therefore, a multi-domain weighted adversarial transfer network is proposed for the cross-domain intelligent fault diagnosis of bearings. In the proposed method, multi-domain adversarial and attention weighting modules are designed to consider bearing multi-mode structure characteristics and solve the influence of local non-transferability regions of signals, respectively. Two diagnosis cases are used to verify the proposed method. The results show that the proposed method is able to extract domain invariant features for different cross-domain diagnosis cases, and thus improves the accuracy of fault identification.

DAD: Domain Adversarial Defense System Against DDoS Attacks in Cloud

The DDoS attack nullifies the data availability in cloud by primarily exploiting the security vulnerabilities of the network protocol and cloud services. The emergence of deep-learning based defense systems have delivered outstanding success in handling the DDoS attacks. However, the deep-learning based systems relies on having similar attack features during training and evaluation. In this paper, we propose a domain adversarial defense (DAD) system to reduce the domain mismatch and generalize the model to handle real-time attacks in cloud. Two novel strategies are applied into our proposed DAD system. 1) Latent feature extractor to extract domain invariant features across multiple domains. 2) Adversarial training algorithm to perform unsupervised adaptation over a defense system using real-time attack samples. The capability of the DAD model is empirically substantiated across four datasets to understand its effectiveness across both known and unknown attacks. The experimental analysis shows that the proposed DAD model achieves substantial improvements with minimum overhead, reduced latency and better efficacy over existing defense mechanisms.

A Novel Symmetric Stacked Autoencoder for Adversarial Domain Adaptation Under Variable Speed

At present, most of the fault diagnosis methods with extensive research and good diagnostic effect are based on the premise that the sample distribution is consistent. However, in reality, the sample distribution of rotating machinery is inconsistent due to variable working conditions, and most of the fault diagnosis algorithms have poor diagnostic effects or even invalid. To dispose the above problems, a novel symmetric stacked autoencoder (NSSAE) for adversarial domain adaptation is proposed. Firstly, the symmetric stacked autoencoder network with shared weights is used as the feature extractor to extract features which can better express the original signal. Secondly, adding domain discriminator that constituting adversarial with feature extractor to enhance the ability of feature extractor to extract domain invariant features, thus confusing the domain discriminator and making it unable to correctly distinguish the features of the two domains. Finally, to assist the adversarial training, the maximum mean discrepancy (MMD) is added to the last layer of the feature extractor to align the features of the two domains in the high-dimensional space. The experimental results show that, under the condition of variable speed, the NSSAE model can extract domain invariant features to achieve the transfer between domains, and the transfer diagnosis accuracy is high and the stability is strong.

Two-Branch Attention Adversarial Domain Adaptation Network for Hyperspectral Image Classification

Recent studies have shown that deep domain adaptation (DA) techniques have good performance on cross-domain hyperspectral image (HSI) classification problems. However, most existing deep HSI DA approaches directly use deep networks to extract features from the data, which ignores the detailed information of HSI in spectral and spatial dimensions. To effectively exploit the spectral–spatial joint information for DA of HSIs, we propose a two-branch attention adversarial DA (TAADA) network in this article. In the TAADA network, a two-branch feature extraction (TBFE) subnetwork is first designed as a generator to extract the attention-based spectral–spatial features. Then, a discriminator based on two classifiers with the multilayer FC-BN-ReLU-Dropout structure is constructed. Based on adversarial learning between the generator and the discriminator, the ability of discriminative feature extraction and cross-domain classification is improved simultaneously. Finally, the TAADA network can adjust the distribution between the source and target domains and extract domain-invariant features. Experimental results on three cross-scene HSI classification tasks show that our proposed TAADA outperforms some existing DA methods.

DFAN: Dual-Branch Feature Alignment Network for Domain Adaptation on Point Clouds

Unsupervised domain adaptation (UDA) significantly reduces the gap between the source domain and the target domain in machine learning and computer vision tasks. Most UDA approaches are applied to images and videos, and only a few methods implement domain adaptation on 3-D computer vision problems. The existing UDA approaches operating on point clouds try to extract domain-invariant features in different domains for feature alignment. However, higher commonality brings less diversity and results a loss of detailed information. In this article, we propose a novel dual-branch feature alignment network (DFAN) architecture for domain adaptation on point cloud visual tasks to better exploit the respective characteristics of local and global features. Our approach specializes in the extraction and alignment of global and local features with different strategies in each branch to complement each other. We also introduce a hierarchical alignment strategy for local feature alignment and a distribution alignment strategy for global feature alignment. Experiments on the PointDA-10 and PointSegDA datasets show that our approach achieves state-of-the-art performance on the UDA of point cloud classification and segmentation tasks. The ablation study demonstrates the effectiveness of the dual-branch design and the feature alignment strategies.

Dual-Branch Meta-Learning Network With Distribution Alignment for Face Anti-Spoofing

Existing face anti-spoofing (FAS) methods fail to generalize well to unseen domains with different data distribution from the training domains, due to the distribution discrepancies between various domains. To extract domain-invariant features for unseen domains, this work proposes a Dual-Branch Meta-learning Network (DBMNet) with distribution alignment for face anti-spoofing. Specifically, DBMNet consists of a feature embedding (FE) branch and a depth estimating (DE) branch for real and fake face discrimination. Each branch acts as a meta-learner and is optimized by step-adjusted meta-learning that can adaptively select the best number of meta-train steps. In order to mitigate distribution discrepancies between domains, we introduce two distribution alignment losses to directly regularize the two meta-learners, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , the triplet loss for FE branch and the depth loss for DE branch, respectively. Both of them are designed as part of the meta-train and meta-test objectives, which contribute to higher-order derivatives on the parameters during the meta-optimization for further seeking domain-invariant features. Extensive ablation studies and comparisons with the state-of-the-art methods show the effectiveness of our method for better generalization.

An unsupervised domain adaptation model based on dual-module adversarial training

In this paper, we propose a dual-module network architecture that employs a domain discriminative feature module to encourage the domain invariant feature module to learn more domain invariant features. The proposed architecture can be applied to any model that utilizes domain invariant features for unsupervised domain adaptation to improve its ability to extract domain invariant features. We conduct experiments with the Domain-Adversarial Training of Neural Networks (DANN) model as a representative algorithm. In the training process, we supply the same input to the two modules and then extract their feature distribution and prediction results respectively. We propose a discrepancy loss to find the discrepancy of the prediction results and the feature distribution between the two modules. Through the adversarial training by maximizing the loss of their feature distribution and minimizing the discrepancy of their prediction results, the two modules are encouraged to learn more domain discriminative and domain invariant features respectively. Extensive comparative evaluations are conducted and the proposed approach outperforms the state-of-the-art in most unsupervised domain adaptation tasks.

Remaining useful life prediction of bearings under different working conditions using a deep feature disentanglement based transfer learning method

The data distribution discrepancy between the training and test samples makes it challenging for the remaining useful life (RUL) prediction under different working conditions. Although various transfer learning methods focusing on minimizing the distribution discrepancy of global cross-domain features have been applied to address this issue, the inherent properties of each domain are always ignored. The domain private representations caused by it has a negative impact on the RUL prediction of another domain. This paper proposes a novel method called Deep Feature Disentanglement Transfer Learning Network (DFDTLN) to extract domain-invariant features. In the proposed method, shared domain-invariant representations and private representations are disentangled by a pair of joint learning autoencoders. The effectiveness of the proposed method is verified using IEEE PHM Challenge 2012 dataset. The comparison results show the deep features extracted by DFDTLN are more domain-invariant and suitable for RUL prediction.

Feature-Aware Adaptation and Density Alignment for Crowd Counting in Video Surveillance.

With the development of deep neural networks, the performance of crowd counting and pixel-wise density estimation is continually being refreshed. Despite this, there are still two challenging problems in this field: 1) current supervised learning needs a large amount of training data, but collecting and annotating them is difficult and 2) existing methods cannot generalize well to the unseen domain. A recently released synthetic crowd dataset alleviates these two problems. However, the domain gap between the real-world data and synthetic images decreases the models' performance. To reduce the gap, in this article, we propose a domain-adaptation-style crowd counting method, which can effectively adapt the model from synthetic data to the specific real-world scenes. It consists of multilevel feature-aware adaptation (MFA) and structured density map alignment (SDA). To be specific, MFA boosts the model to extract domain-invariant features from multiple layers. SDA guarantees the network outputs fine density maps with a reasonable distribution on the real domain. Finally, we evaluate the proposed method on four mainstream surveillance crowd datasets, Shanghai Tech Part B, WorldExpo'10, Mall, and UCSD. Extensive experiments are evidence that our approach outperforms the state-of-the-art methods for the same cross-domain counting problem.

Deep domain adversarial method with central moment discrepancy for intelligent transfer fault diagnosis

Big data condition monitoring in the industrial of internet era is indispensable, and intelligent fault diagnosis plays an important role in it. The adversarial learning method is widely used because of its ability to extract domain invariant features to solve the variable speed fault diagnosis problem. However, its training process is often unstable and difficult to converge to the optimal solution, which brings great challenges to the fault detection of equipment. In view of this exasperating problem, a novel model, called deep domain adversarial method with central moment discrepancy, is proposed. The presented model mainly consists of four modules: a shared weight feature extraction network with wide convolution kernel, a supervised classification network, an adversarial domain classification network, and a CMD alignment network. Adversarial domain classification network is employed to extract features that have both category distinction and domain invariance in the process of mutual game learning between features of source domain and target domain. The CMD alignment network can be devoted to align the higher-order moments of two domain features to constrain the instability in adversarial learning. Through the above regularization method, the model shows a relatively stable and higher accuracy of transferring diagnosis in the non-standardized data. The public test data set and the private data set are applied to validate the model. The results show that the proposed model successfully solves the problem of training instability in adversarial learning and has a relatively high diagnostic accuracy.

DeepMerge – II. Building robust deep learning algorithms for merging galaxy identification across domains

ABSTRACT In astronomy, neural networks are often trained on simulation data with the prospect of being used on telescope observations. Unfortunately, training a model on simulation data and then applying it to instrument data leads to a substantial and potentially even detrimental decrease in model accuracy on the new target data set. Simulated and instrument data represent different data domains, and for an algorithm to work in both, domain-invariant learning is necessary. Here, we employ domain adaptation techniques – Maximum Mean Discrepancy as an additional transfer loss and Domain Adversarial Neural Networks – and demonstrate their viability to extract domain-invariant features within the astronomical context of classifying merging and non-merging galaxies. Additionally, we explore the use of Fisher loss and entropy minimization to enforce better in-domain class discriminability. We show that the addition of each domain adaptation technique improves the performance of a classifier when compared to conventional deep learning algorithms. We demonstrate this on two examples: between two Illustris-1 simulated data sets of distant merging galaxies, and between Illustris-1 simulated data of nearby merging galaxies and observed data from the Sloan Digital Sky Survey. The use of domain adaptation techniques in our experiments leads to an increase of target domain classification accuracy of up to ${\sim }20{{\ \rm per\ cent}}$. With further development, these techniques will allow astronomers to successfully implement neural network models trained on simulation data to efficiently detect and study astrophysical objects in current and future large-scale astronomical surveys.