Domain Adaptation with Representation Learning and Nonlinear Relation for Time Series

Unsupervised domain adaptation without source data for estimating occupancy and recognizing activities in smart buildings

Machine learning methods have proven to be effective in analyzing vast amounts of data in various formats to obtain patterns, detect trends, gain insight, and predict outcomes based on historical data. However, training models from scratch across various real-world applications is costly in terms of both time and data consumption. Model adaptation (Domain Adaptation) is a promising methodology to tackle this problem. It can reuse the knowledge embedded in an existing model to train another model. However, model adaptation is a challenging task due to dataset bias or domain shift. In addition, data access from both the original (source) domain and the destination (target) domain is often an issue in the real world, due to data privacy and cost issues (gathering additional data may cost money). Several domain adaptation algorithms and methodologies have introduced in recent years; they reuse trained models from one source domain for a different but related target domain. Many existing domain adaptation approaches aim at modifying the trained model structure or adjusting the latent space of the target domain using data from the source domain. Domain adaptation techniques can be evaluated over several criteria, namely, accuracy, knowledge transfer, training time, and budget. In this paper, we start from the notion that in many real-world scenarios, the owner of the trained model restricts access to the model structure and the source dataset. To solve this problem, we propose a methodology to efficiently select data from the target domain (minimizing consumption of target domain data) to adapt the existing model without accessing the source domain, while still achieving acceptable accuracy. Our approach is designed for supervised and semi-supervised learning and extendable to unsupervised learning.

Domain adaptation leverages the knowledge in one domain-the source domain-to improve learning efficiency in another domain-the target domain. Existing heterogeneous domain adaptation research is relatively well-progressed but only in situations where the target domain contains at least a few labeled instances. In contrast, heterogeneous domain adaptation with an unlabeled target domain has not been well-studied. To contribute to the research in this emerging field, this article presents: 1) an unsupervised knowledge transfer theorem that guarantees the correctness of transferring knowledge and 2) a principal angle-based metric to measure the distance between two pairs of domains: one pair comprises the original source and target domains and the other pair comprises two homogeneous representations of two domains. The theorem and the metric have been implemented in an innovative transfer model, called a Grassmann-linear monotonic maps-geodesic flow kernel (GLG), which is specifically designed for heterogeneous unsupervised domain adaptation (HeUDA). The linear monotonic maps (LMMs) meet the conditions of the theorem and are used to construct homogeneous representations of the heterogeneous domains. The metric shows the extent to which the homogeneous representations have preserved the information in the original source and target domains. By minimizing the proposed metric, the GLG model learns the homogeneous representations of heterogeneous domains and transfers knowledge through these learned representations via a geodesic flow kernel (GFK). To evaluate the model, five public data sets were reorganized into ten HeUDA tasks across three applications: cancer detection, the credit assessment, and text classification. The experiments demonstrate that the proposed model delivers superior performance over the existing baselines.

The focus of Source-free Unsupervised Domain Adaptation (SFUDA) is to effectively transfer a well-trained model from the source domain to an unlabelled target domain. During the target domain adaptation, the source domain data is no longer accessible. Prevalent methodologies attempt to synchronize the data distributions between the source and target domains, utilizing pseudo-labels to impart categorical information, which has made some progress in improving the model’s performance. However, performance impairments persist due to the introduction of learning bias from the source model and the impact of noisy pseudo-labels generated for the target domain. In this research, we reveal that the central cause for feature misalignment during domain transition is the learning bias, which is generated by the discrepancy of information between source and target domain data. The source domain data may contain distinguishable features that do not appear on the target domain, which causes the pre-trained source model to fail to work during domain adaptation. To overcome the information discrepancy, we propose a Prototypical Feature Compensation (PFC) Network. The network extracts representative feature maps of the source domain. Then use them to minimize the discrepancy information in the target domain feature maps. This mechanism facilitates feature alignment across different domains, allowing the model to generate more accurate categorical data through pseudo-labelling. The experimental results and ablation studies demonstrate exceptional performance on three SFUDA datasets and provide evidence of the proposed PFC method’s ability to adjust the feature distribution of both source and target domain data, ensuring their overlap in the latent space.

Domain adaptation (DA) offers an effective way to align feature distributions of the source domain (SD) and the target domain (TD) without requiring any target label samples. As a method of DA, representation learning effectively realizes the alignment of feature distributions in different domains by transferring domain knowledge. However, existing representation learning methods often focus on unilateral representation transfer, which potentially results in transfer bias. Additionally, most methods ignore the connection between domain alignment and discrimination during the DA process, which easily causes negative transfer. This paper proposes a dynamic weighted dual-driven domain adaptation (DW-D 3 A) model that effectively addresses the aforementioned issues through bilateral feature transfer between domains and a dynamic weighted scheme. Technically, we first propose a dual-driven domain adaptation (D 3 A) model, which employs symmetrical structures to facilitate the knowledge transfer of bilateral representations between source and target domain samples, learning the subspaces of two domains and reducing distribution discrepancies between subspaces via joint distribution-driven alignment. This process mitigates transfer bias and goes beyond previous unilateral transfer methods. Then, to alleviate strong constraints on projecting SD and TD into the same subspace in existing approaches, we apply a relaxed subspace constraint to bring the projections of SD and TD closer. Furthermore, data reconstruction is incorporated to preserve discriminant information from the original data. Lastly, we expand (D 3 A) to DW-D 3 A using a dynamic weighted scheme, which adjusts the weights assigned to domain alignment and discrimination based on their significance to inhibit negative transfer. Extensive experimentation on three datasets indicates that DW-D 3 A outperforms seven other DA methods, showing its superior performance.

Representation learning via serial autoencoders for domain adaptation

Smart buildings have gained increasing interest recently by providing several advanced solutions, especially AI-based solutions. Activity recognition and occupancy estimation are among the outcomes of smart buildings that can help provide several advantages such as energy management and security solutions. Previously, domain adaptation (DA) has been widely considered by researchers to transfer knowledge from source domains, where we have abundant labeled data, to a target domain where labeled data is scarce. It is a tedious and time-consuming task to label data, especially with smart building applications which is why researchers have considered unsupervised DA where we do have labeled data in the source domain and unlabeled data in the target domain. Semi-supervised DA (SSDA) adaptation has also been considered by researchers where we have a small amount of labeled data in the target domain. Most unsupervised DA (UDA) and SSDA methods transfer knowledge from one source to one target. However, it is possible to exploit knowledge from multiple source domains instead of one single domain to enhance the performance of the target domain. Multi-source DA (MSDA) is more difficult than single-source DA but also it is more efficient. In this research, we adapt several MDSA methods and evaluate them using sensorial datasets.

Generalized zero-shot domain adaptation via coupled conditional variational autoencoders

Domain adaptation aims to transfer knowledge from a domain with adequate labeled samples to a domain with scarce labeled samples. Prior research has introduced various open set domain adaptation settings in the literature to extend the applications of domain adaptation methods in real-world scenarios. This paper focuses on the type of open set domain adaptation setting where the target domain has both private ('unknown classes') label space and the shared ('known classes') label space. However, the source domain only has the 'known classes' label space. Prevalent distribution-matching domain adaptation methods are inadequate in such a setting that demands adaptation from a smaller source domain to a larger and diverse target domain with more classes. For addressing this specific open set domain adaptation setting, prior research introduces a domain adversarial model that uses a fixed threshold for distinguishing known from unknown target samples and lacks at handling negative transfers. We extend their adversarial model and propose a novel adversarial domain adaptation model with multiple auxiliary classifiers. The proposed multi-classifier structure introduces a weighting module that evaluates distinctive domain characteristics for assigning the target samples with weights which are more representative to whether they are likely to belong to the known and unknown classes to encourage positive transfers during adversarial training and simultaneously reduces the domain gap between the shared classes of the source and target domains. A thorough experimental investigation shows that our proposed method outperforms existing domain adaptation methods on a number of domain adaptation datasets. © 1999-2012 IEEE.

Geometric Knowledge Embedding for unsupervised domain adaptation

This paper considers the unsupervised domain adaptation problem, in which we want to find a good prediction function on the unlabeled target domain, by utilizing the information provided in the labeled source domain. A common approach to the domain adaptation problem is to learn a representation space where the distributional discrepancy of the source and target domains is small. Existing methods generally tend to match the marginal distributions of the two domains, while the label information in the source domain is not fully exploited. In this paper, we propose a representation learning approach for domain adaptation, which is addressed as JODAWAT. We aim to adapt the joint distributions of the feature-label pairs in the shared representation space for both domains. In particular, we minimize the Wasserstein distance between the source and target domains, while the prediction performance on the source domain is also guaranteed. The proposed approach results in a minimax adversarial training procedure that incorporates a novel split gradient penalty term. A generalization bound on the target domain is provided to reveal the efficacy of representation learning for joint distribution adaptation. We conduct extensive evaluations on JODAWAT, and test its classification accuracy on multiple synthetic and real datasets. The experimental results justify that our proposed method is able to achieve superior performance compared with various domain adaptation methods.

Electroencephalogram (EEG) is one of the most widely used signals in motor imagery (MI) based brain-computer interfaces (BCIs). Domain adaptation has been frequently used to improve the accuracy of EEG-based BCIs for a new user (target domain), by making use of labeled data from a previous user (source domain). However, this raises privacy concerns, as EEG contains sensitive health and mental information. It is very important to perform privacy-preserving domain adaptation, which simultaneously improves the classification accuracy for a new user and protects the privacy of a previous user. We propose augmentation-based source-free adaptation (ASFA), which consists of two parts: 1) source model training, where a novel data augmentation approach is proposed for MI EEG signals to improve the cross-subject generalization performance of the source model; and, 2) target model training, which simultaneously considers uncertainty reduction for domain adaptation and consistency regularization for robustness. ASFA only needs access to the source model parameters, instead of the raw EEG data, thus protecting the privacy of the source domain. We further extend ASFA to a stricter privacy-preserving scenario, where the source model's parameters are also inaccessible. Experimental results on four MI datasets demonstrated that ASFA outperformed 15 classical and state-of-the-art MI classification approaches. This is the first work on completely source-free domain adaptation for EEG-based BCIs. Our proposed ASFA achieves high classification accuracy and strong privacy protection simultaneously, important for the commercial applications of EEG-based BCIs.

Unsupervised domain adaptation methods based on a single source domain assume that the target domain data does not contain label information, and diagnostic knowledge can be borrowed from a single source. These methods eliminate dependence on label information in target domain data, and as a result, they have been widely studied in the field of equipment fault diagnosis in recent years. However, in real-world scenarios, there are more challenging application cases: the diagnostic knowledge to be leveraged comes from multiple different operating conditions, or the target domain data lacks labeled information. Single-source domain methods struggle to effectively handle the differences between the source domain and the target domain. Additionally, existing diagnostic methods primarily focus on extracting common features between the source domain and target domain and aligning their distributions in a shared feature space, but they neglect the differences between source domains, between source domains and the target domain, and the decision boundaries between different fault categories. To address these issues, this paper proposes a domain adaptation fault diagnosis method based on dual-stage progressive alignment. First, several domain adaptation feature extractors are built to explicitly describe the structural variations between source domains, achieving feature adaptation in the first step, as opposed to standard multi-source transfer methods that handle all sources equally. In order to achieve feature adaptation in the second stage and more domain-adaptive results, a multi-source response classifier is designed to use the structural information of decision boundaries between source domains to guide the target domain based on feature-level alignment. Ultimately, the high-speed aviation bearing dataset is used for extensive experiments, and the outcomes show that the proposed approach excels at intelligent diagnostic tasks involving multiple source domains.

Domain adaptation aims at generalizing a high-performance learner on a target domain via utilizing the knowledge distilled from a source domain which has a different but related data distribution. One solution to domain adaptation is to learn domain invariant feature representations while the learned representations should also be discriminative in prediction. To learn such representations, domain adaptation frameworks usually include a domain invariant representation learning approach to measure and reduce the domain discrepancy, as well as a discriminator for classification. Inspired by Wasserstein GAN, in this paper we propose a novel approach to learn domain invariant feature representations, namely Wasserstein Distance Guided Representation Learning (WDGRL). WDGRL utilizes a neural network, denoted by the domain critic, to estimate empirical Wasserstein distance between the source and target samples and optimizes the feature extractor network to minimize the estimated Wasserstein distance in an adversarial manner. The theoretical advantages of Wasserstein distance for domain adaptation lie in its gradient property and promising generalization bound. Empirical studies on common sentiment and image classification adaptation datasets demonstrate that our proposed WDGRL outperforms the state-of-the-art domain invariant representation learning approaches.

The performance of the supervised learning algorithms such as k-nearest neighbor (k-NN) depends on the labeled data. For some applications (Target Domain), obtaining such labeled data is very expensive and labor-intensive. In a real-world scenario, the possibility of some other related application (Source Domain) is always accompanied by sufficiently labeled data. However, there is a distribution discrepancy between the source domain and the target domain application data as the background of collecting both the domains data is different. Therefore, source domain application with sufficient labeled data cannot be directly utilized for training the target domain classifier. Domain Adaptation (DA) or Transfer learning (TL) provides a way to transfer knowledge from source domain application to target domain application. Existing DA methods may not perform well when there is a much discrepancy between the source and the target domain data, and the data is non-linear separable. Therefore, in this paper, we provide a Kernelized Unified Framework for Domain Adaptation (KUFDA) that minimizes the discrepancy between both the domains on linear or non-linear data-sets and aligns them both geometrically and statistically. The substantial experiments verify that the proposed framework outperforms state-of-the-art Domain Adaptation and the primitive methods (Non- Domain Adaptation) on real-world Office-Caltech and PIE Face data-sets. Our proposed approach (KUFDA) achieved mean accuracies of 86.83% and 74.42% for all possible tasks of Office-Caltech with VGG-Net features and PIE Face data-sets.