Cross-domain aspect term extraction using pre-trained language models with pre-training and fine-tuning strategy

The focus of Source-free Unsupervised Domain Adaptation (SFUDA) is to effectively transfer a well-trained model from the source domain to an unlabelled target domain. During the target domain adaptation, the source domain data is no longer accessible. Prevalent methodologies attempt to synchronize the data distributions between the source and target domains, utilizing pseudo-labels to impart categorical information, which has made some progress in improving the model’s performance. However, performance impairments persist due to the introduction of learning bias from the source model and the impact of noisy pseudo-labels generated for the target domain. In this research, we reveal that the central cause for feature misalignment during domain transition is the learning bias, which is generated by the discrepancy of information between source and target domain data. The source domain data may contain distinguishable features that do not appear on the target domain, which causes the pre-trained source model to fail to work during domain adaptation. To overcome the information discrepancy, we propose a Prototypical Feature Compensation (PFC) Network. The network extracts representative feature maps of the source domain. Then use them to minimize the discrepancy information in the target domain feature maps. This mechanism facilitates feature alignment across different domains, allowing the model to generate more accurate categorical data through pseudo-labelling. The experimental results and ablation studies demonstrate exceptional performance on three SFUDA datasets and provide evidence of the proposed PFC method’s ability to adjust the feature distribution of both source and target domain data, ensuring their overlap in the latent space.

Source-Free Domain Adaptation (SFDA) aims to address the challenge of effectively transferring a source domain model to a target domain when the target domain data is unlabeled and the source domain data is unavailable. Traditional Unsupervised Domain Adaptation (UDA) methods rely on simultaneous access to both source and target domain data. However, in many practical scenarios, such as medical data privacy protection or resource-constrained devices, direct access to source domain data is not feasible. SFDA leverages only a pre-trained source domain model and unlabeled target domain data to update the model, avoiding the direct use of source domain data and thereby meeting privacy and security requirements. This paper provides a systematic classification and review of SFDA research methods, categorizing them into three main types: data-related methods, model-related methods, and loss-related methods. Data-related methods replace missing source data by extracting data or feature augmentation information from pre-trained models; model-related methods reduce domain discrepancies by optimizing feature representations or utilizing information in the feature space; and loss-related methods enhance the model's generalization ability through specific loss functions. This paper aims to offer a clear research roadmap for researchers in the field by systematically classifying and analyzing existing SFDA methods, facilitating the selection of appropriate methods or the development of new strategies to address specific problems.

The unsupervised domain adaptive classification task can learn domain-invariant features between the unlabeled target domain data and the labeled source domain data, thereby improving the classification performance of the classifier in the target domain. However, privacy protection and memory-constrained often make it difficult to obtain labeled source domain samples, which will bring bottlenecks to the existing domain adaptation tasks. To this end, we propose a novel source free domain adaptive classification model, that is, without any source domain data, a classifier with good performance in the target domain can be obtained only by using the source domain pre-trained classifier and the target domain data. The method first proposes a novel conditional information generative adversarial module based on combined discriminators. Through the confrontation between combined discriminators and the generator, the middle domain with pseudo-labels is generated to solve the problem of missing source domain. Then when training the new classifier in the domain adaptation module, we add a distillation loss mechanism to deal with the lack of source domain data supervision, thereby minimizing the difference between the old classifier response and the new classifier response to ensure that the network output retains the source domain information. We conducted experiments on three groups of 10 data sets, which proved that our method can effectively solve the problem of source free domain adaptive classification and effectively improve the classification accuracy of the model in each domain.

Domain-transfer learning is a machine learning task to explore a source domain data set to help the learning problem in a target domain. Usually, the source domain has sufficient labeled data, while the target domain does not. In this paper, we propose a novel domain-transfer convolutional model by mapping a target domain data sample to a proxy in the source domain and applying a source domain model to the proxy for the purpose of prediction. In our framework, we firstly represent both source and target domains to feature vectors by two convolutional neural networks and then construct a proxy for each target domain sample in the source domain space. The proxy is supposed to be matched to the corresponding target domain sample convolutional representation vector well. To measure the matching quality, we proposed to maximize their squared-loss mutual information (SMI) between the proxy and target domain samples. We further develop a novel neural SMI estimator based on a parametric density ratio estimation function. Moreover, we also propose to minimize the classification error of both source domain samples and target domain proxies. The classification responses are also smoothened by manifolds of both the source domain and proxy space. By minimizing an objective function of SMI, classification error, and manifold regularization, we learn the convolutional networks of both source and target domains. In this way, the proxy of a target domain sample can be matched to the source domain data and thus benefits from the rich supervision information of the source domain. We design an iterative algorithm to update the parameters alternately and test it over benchmark data sets of abnormal behavior detection in video, Amazon product reviews sentiment analysis, etc.

Expanding visual categorization into a novel domain without the need of extra annotation has been a long-term interest for multimedia intelligence. Previously, this challenge has been approached by unsupervised domain adaptation (UDA). Given labeled data from a source domain and unlabeled data from a target domain, UDA seeks for a deep representation that is both discriminative and domain-invariant. While UDA focuses on the target domain, we argue that the performance on both source and target domains matters, as in practice which domain a test example comes from is unknown. In this article, we extend UDA by proposing a new task called unsupervised domain expansion (UDE), which aims to adapt a deep model for the target domain with its unlabeled data, meanwhile maintaining the model’s performance on the source domain. We propose Knowledge Distillation Domain Expansion (KDDE) as a general method for the UDE task. Its domain-adaptation module can be instantiated with any existing model. We develop a knowledge distillation-based learning mechanism, enabling KDDE to optimize a single objective wherein the source and target domains are equally treated. Extensive experiments on two major benchmarks, i.e., Office-Home and DomainNet, show that KDDE compares favorably against four competitive baselines, i.e., DDC, DANN, DAAN, and CDAN, for both UDA and UDE tasks. Our study also reveals that the current UDA models improve their performance on the target domain at the cost of noticeable performance loss on the source domain.

Unsupervised open-set domain adaptation (UODA) is a realistic problem where unlabeled target data contain unknown classes. Prior methods rely on the coexistence of both source and target domain data to perform domain alignment, which greatly limits their applications when source domain data are restricted due to privacy concerns. In this paper we address the challenging hypothesis transfer setting for UODA, where data from source domain are no longer available during adaptation on target domain. Specifically, we propose to use pseudo-labels and a novel consistency regularization on target data, where using conventional formulations fails in this open-set setting. Firstly, our method discovers confident predictions on target domain and performs classification with pseudo-labels. Then we enforce the model to output consistent and definite predictions on semantically similar transformed inputs, discovering all latent class semantics. As a result, unlabeled data can be classified into discriminative classes coincided with either source classes or unknown classes. We theoretically prove that under perfect semantic transformation, the proposed objective that enforces consistency can recover the information of true labels in prediction. Experimental results show that our model outperforms state-of-the-art methods on UODA benchmarks.

Recently, deep neural networks have achieved great success in bearing fault diagnosis. Most existing methods are developed under the assumption that the bearing vibration signals are collected under the same machine operating conditions. However, bearing fault diagnosis under cross-domain conditions will suffer from domain shift problems if the neural network is only trained with the source domain data. Moreover, acquiring enough labeled data from the target domain will be expensive and time-consuming. To address the above problems, this paper proposes an end-to-end multi-adversarial cross-domain neural network for bearing fault diagnosis, which takes labeled source domain data and unlabeled target domain data to achieve the cross-domain bearing fault diagnosis under cross-load conditions and cross-machine conditions. The proposed method employs multi-adversarial training to automatically extract the domain-invariant features from source and target domains instead of manually designing features, which combines domain-adversarial learning and mini-max entropy adversarial learning to adversarially reduce the domain discrepancy between the source and target domains and alleviate the class misalignment problem. The results of the cross-load and the cross-machine experiments prove the effectiveness of the proposed method, and the proposed method provides a promising tool for cross-domain bearing fault diagnosis.

Sensors are devices that output signals for sensing physical phenomena and are widely used in all aspects of our social production activities. The continuous recording of physical parameters allows effective analysis of the operational status of the monitored system and prediction of unknown risks. Thanks to the development of deep learning, the ability to analyze temporal signals collected by sensors has been greatly improved. However, models trained in the source domain do not perform well in the target domain due to the presence of domain gaps. In recent years, many researchers have used deep unsupervised domain adaptation techniques to address the domain gap between signals collected by sensors in different scenarios, i.e., using labeled data in the source domain and unlabeled data in the target domain to improve the performance of models in the target domain. This survey first summarizes the background of recent research on unsupervised domain adaptation with time series sensor data, the types of sensors used, the domain gap between the source and target domains, and commonly used datasets. Then, the paper classifies and compares different unsupervised domain adaptation methods according to the way of adaptation and summarizes different adaptation settings based on the number of source and target domains. Finally, this survey discusses the challenges of the current research and provides an outlook on future work. This survey systematically reviews and summarizes recent research on unsupervised domain adaptation for time series sensor data to provide the reader with a systematic understanding of the field.

In this paper, we focus on the adaptation problem that has a large labeled data in the source domain and a large but unlabeled data in the target domain. Our aim is to learn reliable information from unlabeled target domain data for dependency parsing adaptation. Current state-of-the-art statistical parsers perform much better for shorter dependencies than for longer ones. Thus we propose an adaptation approach by learning reliable information on shorter dependencies in an unlabeled target data to help parse longer distance words. The unlabeled data is parsed by a dependency parser trained on labeled source domain data. The experimental results indicate that our proposed approach outperforms the baseline system, and is better than current state-of-the-art adaptation techniques.

Deep\footnote learning-based semantic segmentation methods require a huge amount of training images with pixel-level annotations. Unsupervised domain adaptation (UDA) for semantic segmentation enables transferring knowledge learned from the synthetic data (source domain) with low-cost annotations to the real images (target domain). However, current UDA methods mostly require full access to the source domain data for feasible adaptation, which limits their applications in real-world scenarios with privacy, storage, or transmission issues. To this end, this paper identifies and addresses a more practical but challenging problem of UDA for semantic segmentation, where access to the original source domain data is forbidden. In other words, only the pre-trained source model and unlabelled target domain data are available for adaptation. To tackle the problem, we propose to construct a set of source domain virtual data to mimic the source domain distribution by identifying the target domain high-confidence samples predicted by the pre-trained source model. Then by analyzing the data properties in the cross-domain semantic segmentation tasks, we propose an uncertainty and prior distribution-aware domain adaptation method to align the virtual source domain and the target domain with both adversarial learning and self-training strategies. Extensive experiments on three cross-domain semantic segmentation datasets with in-depth analyses verify the effectiveness of the proposed method.

The goal of domain adaptation is to train a high-performance predictive model on the target domain data by using knowledge from the source domain data, which has different but related data distribution. In this paper, we consider unsupervised domain adaptation where we have labelled source domain data but unlabelled target domain data. Our solution to unsupervised domain adaptation is to learn a domain-invariant representation that is also category discriminative. Domain-invariant representations are realized by minimizing the domain discrepancy. To minimize the domain discrepancy, we propose a novel graph-matching metric between the source and target domain representations. Minimizing this metric allows the source and target representations to be in support of each other. We further exploit confident unlabelled target domain samples and their pseudo-labels to refine our proposed model. We expect the refining step to improve the performance further. This is validated by performing experiments on standard image classification adaptation datasets. Results showed our proposed approach out-perform previous domain-invariant representation learning approaches.

Unsupervised multi-source domain adaptation methods transfer knowledge learned from multiple labeled source domains to an unlabeled target domain. Existing methods assume that all source domain data can be accessed directly. However, such an assumption is unrealistic and causes data privacy concerns, especially when the source domain labels include personal information. In such a setting, it is prohibited to minimize domain gaps by pairwise calculation of the data from the source and target domains. Therefore, this work addresses the source-free unsupervised multi-source domain adaptation problem, where only the source models are available during the adaptation. We propose trust center sample-based source-free domain adaptation (TSDA) method to solve this problem. The key idea is to leverage the pre-trained models from the source domain and progressively train the target model in a self-learning manner. Because target samples with low entropy measured from the pre-trained source model achieve high accuracy, the trust center samples are selected first using the entropy function. Then pseudo labels are assigned for target samples based on a self-supervised pseudo-labeling strategy. For multiple source domains, corresponding target models are learned based on the assigned pseudo labels. Finally, multiple target models are integrated to predict the label for unlabeled target data. Extensive experiment results on some benchmark datasets and generated adversarial samples demonstrate that our approach outperforms existing UMDA methods, even though some methods can always access source data.

Transfer learning aims to enhance performance in a target domain by exploiting useful information from auxiliary or source domains when the labeled data in the target domain are insufficient or difficult to acquire. In some real-world applications, the data of source domain are provided in advance, but the data of target domain may arrive in a stream fashion. This kind of problem is known as online transfer learning. In practice, there can be several source domains that are related to the target domain. The performance of online transfer learning is highly associated with selected source domains, and simply combining the source domains may lead to unsatisfactory performance. In this paper, we seek to promote classification performance in a target domain by leveraging labeled data from multiple source domains in online setting. To achieve this, we propose a new online transfer learning algorithm that merges and leverages the classifiers of the source and target domain with an ensemble method. The mistake bound of the proposed algorithm is analyzed, and the comprehensive experiments on three real-world data sets illustrate that our algorithm outperforms the compared baseline algorithms.

Small data sets are an extremely challenging problem in the machine learning (ML) realm, and in specific, in regression scenarios, as the lack of relevant data can lead to ML models that have large bias. However, there are many applications for which a purely data‐driven procedure would be advantageous, but a large amount of data are not available. This article proposes a novel regression‐based transfer learning (TL) model to address this challenge, where TL is defined as knowledge transfer from a large, relevant data set (source domain data) to a small data set (target domain data). The proposed TL model is termed double‐weighted support vector transfer regression (DW‐SVTR), which couples least squares support vector machines for regression (LS‐SVMR) with two weight functions. The first weight function uses kernel mean matching (KMM) to reweight the source domain data such that the mean values of the source and target domain data in a reproduced kernel Hilbert space (RKHS) are close. In this way, the source domain data points relevant to the target domain points have a larger weight than irrelevant source domain points. The second weight is a function of estimated residuals, which aims to further reduce the negative interference of irrelevant source domain points. The proposed approach is assessed and validated via simulated data and by enhanced shear strength prediction of nonductile columns based on limited availability of nonductile column data. Specifically, the results for the latter show that the proposed DW‐SVTR can reduce the root mean square error (RMSE) by 34% and enhance the coefficient of determination (R2) by 229%. These numerical results demonstrate that the DW‐SVTR significantly reduces the effect of small sample bias and improves prediction performance compared to standard ML methods.

Existing intelligent gearbox fault diagnosis approaches have two shortcomings: (a) their performance is mostly confined to manual handcrafted features, and (b) they follow a general assumption that the distribution of the data in the source domain (labeled data on which the model is trained) is similar to the target domain (unlabeled data on which the model is tested), which might not be the case in real-world applications. Substantial human expertise and domain knowledge is required for manual feature extraction, and moreover, deploying the same model for a target domain whose distribution is different from the source domain would lead to poor generalization. Since deep learning methods can automatically learn high dimensional feature representations from raw measurement data, this paper proposes a novel deep learning-based domain adaptation (DA) method for gearbox fault diagnosis under significant speed variations. A deep convolutional neural network is used as the main architecture. The paper proposes to minimize the summation of cross-entropy loss (between the labeled source domain data) and maximum mean discrepancy loss (between the labeled source and unlabeled target datasets) simultaneously to adapt the source domain model to be applied in the target domain. The proposed deep learning DA approach is evaluated using experimental data from a gearbox under variable speeds and multiple health conditions. An appropriate benchmarking with both traditional machine learning methods and other DA methods demonstrate the superiority of the proposed method.