Transfer Learning Problem Research Articles

An Adaptive Transfer Learning Framework for Functional Classification

In this paper, we study the transfer learning problem in functional classification, aiming to improve the classification accuracy of the target data by leveraging information from related source datasets. To facilitate transfer learning, we propose a novel transferability function tailored for classification problems, enabling a more accurate evaluation of the similarity between source and target dataset distributions. Interestingly, we find that a source dataset can offer more substantial benefits under certain conditions than another dataset with an identical distribution to the target dataset. This observation renders the commonly-used debiasing step in the parameter-based transfer learning algorithm unnecessary under some circumstances to the classification problem. In particular, we propose two adaptive transfer learning algorithms based on the functional Distance Weighted Discrimination (DWD) classifier for scenarios with and without prior knowledge regarding informative sources. Furthermore, we establish the upper bound on the excess risk of the proposed classifiers, providing the statistical gain via transfer learning mathematically provable. Simulation studies are conducted to thoroughly examine the finite-sample performance of the proposed algorithms. Finally, we implement the proposed method to Beijing air-quality data, and significantly improve the prediction of the PM 2.5 level of a target station by effectively incorporating information from source datasets.

Transfer Learning for Logistic Regression with Differential Privacy

Transfer learning, as a machine learning approach enhancing model generalization across different domains, has extensive applications in various fields. However, the risk of privacy leakage remains a crucial consideration during the transfer learning process. Differential privacy, with its rigorous mathematical foundation, has been proven to offer consistent and robust privacy protection. This study delves into the logistic regression transfer learning problem supported by differential privacy. In cases where transferable sources are known, we propose a two-step transfer learning algorithm. For scenarios with unknown transferable sources, a non-algorithmic, cross-validation-based transferable source detection method is introduced, to mitigate adverse effects from non-informative sources. The effectiveness of the proposed algorithm is validated through simulations and experiments with real-world data.

Transfer Learning in Machine Learning: A Review of Methods and Applications

Transfer learning has gained significant traction and popularity in the field of machine learning due to its wide range of potential applications. This review article offers a thorough examination of transfer learning techniques and their wide-ranging applications in several fields. This text provides a thorough evaluation of the literature, focusing on important research and the methodology used. Furthermore, a comparative table highlighting transfer learning research across several areas provides valuable insights into the wide range of applications. The inclusion criteria were centred on recent articles published within the past five years that comprehensively examined transfer learning methodologies, applications, frameworks, problems, and future directions. The review articles highlight the widespread use of transfer learning models, the effectiveness of data augmentation strategies, and the capability of transfer learning to tackle issues particular to different domains. Nevertheless, some constraints like as biases in the dataset, difficulties in interpreting the model, and problems with scalability have been recognised. These limitations provide opportunities for future research to focus on creating transfer learning algorithms that are more resilient and easier to read.

Computing Speed-of-Sound From Ultrasound: User-Agnostic Recovery and a New Benchmark.

Medical ultrasound is one of the most accessible imaging modalities, but is a challenging modality for quantitative parameters comparison across vendors and sonographers. B-Mode imaging, with limited exceptions, provides a map of tissue boundaries; crucially, it does not provide diagnostically relevant physical quantities of the interior of organ domains.This can be remedied: the raw ultrasound signal carries significantly more information than is present in the B-Mode image. Specifically, the ability to recover speed-of-sound and attenuation maps from the raw ultrasound signal transforms the modality into a tissue-property modality. Deep learning was shown to be a viable tool for recovering speed-of-sound maps. A major hold-back towards deployment is the domain transfer problem, i.e., generalizing from simulations to real data. This is due in part to dependence on the (hard-to-calibrate) system response. We explore a remedy to the problem of operator-dependent effects on the system response by introducing a novel approach utilizing the phase information of the IQ demodulated signal. We show that the IQ-phase information effectively decouples the operator-dependent system response from the data, significantly improving the stability of speed-of-sound recovery. We also introduce an improvement to the network topology providing faster and improved results to the state-of-the-art. We present the first publicly available benchmark for this problem: a simulated dataset for raw ultrasound plane wave processing. The consideration of the phase of the IQ-signals presents a promising appeal to traversing the transfer learning problem, advancing the goal of real-time speed-of-sound imaging.

Multi-label deep transfer learning method for coupling fault diagnosis

With the development of complexity and integration of machines, the multiple components of the system are prone to simultaneous failures and the multi-fault signals may be measured in limited sensors. This fault mode is defined as coupling fault in this paper and becomes more common and attracts great attention. Current coupling fault diagnosis approaches rely on labeled data under consistent working conditions. However, in practical varying working conditions, the data is difficult to label and the traditional methods perform poorly. To address the challenge problem, we utilize some related annotated data under other working conditions for auxiliary learning and abstract this problem as a multi-label transfer learning problem which is a new research topic in fault diagnosis. We proposed a novel deep transfer coupling fault diagnosis method from the global and local views to achieve the effect of transfer, which is the first attempt to address this practical problem as far as we know. At the global level, we used Maximum Mean Discrepancy (MMD) at multiple stages of the network to reduce the hypothesis space of the model. Furthermore, we design a special manifold framework with multi-level similarity to constrain the hierarchy of the distance of the sample and the hierarchy of the labels being consistent between the two domains, to further improve the effect of local alignment. The proposed method has high accuracy in multiple transfer tasks for both public and laboratory datasets, powerfully demonstrating its effectiveness.

Reprint of: Some thoughts about transfer learning. What role for the source domain?

Transfer learning is called for when the training and test data do not share the same input distributions (PXS≠PXT) or/and not the same conditional ones (PY|XS≠PY|XT). In the most general case, the input spaces and/or output spaces can be different: XS≠XT and/or YS≠YT. However, most work assume that XS=XT. Furthermore, a common held assumption is that it is necessary that the source hypothesis be good on the source training data and that the “distance” between the source and the target domains be as small as possible in order to get a good (transferred) target hypothesis.This paper revisits the reasons for these beliefs and discusses the relevance of these conditions. An algorithm is presented which can deal with transfer learning problems where XS≠XT, and that furthermore brings a fresh perspective on the role of the source hypothesis (it does not have to be good) and on what is important in the distance between the source and the target domains (translations between them should belong to a limited set). Experiments illustrate the properties of the method and confirm the theoretical analysis.Determining beforehand a relevant source hypothesis remains an open problem, but the vista provided here helps understanding its role.

Privacy-Preserving Distributed Transfer Learning and its Application in Intelligent Transportation

With the rapid development of intelligent transportation systems (ITS), more and more intelligent applications for ITS have received widespread attention, such as the vehicle detection, inference of typical routes, and traffic forecasting. In these applications, deep learning is widely used as a key artificial intelligence technology. However, most ITS providers fail to collect enough labeled traffic data for model training. As a complement to deep learning, transfer learning is an effective way to solve the scarcity of labeled data, which can transfer knowledge from labeled datasets to unlabeled datasets, thus improving the accuracy of prediction and classification. Nevertheless, when the labeled dataset and the unlabeled dataset are held by different entities, it is still unrealistic for two mutually distrustful entities to cooperate in transfer learning regarding data security and privacy preservation. Although some existing works provide privacy-preserving transfer learning methods, such methods fail to apply to traffic data with high sample dimensions due to their high computational cost and round complexity. To address this problem, we design an efficient privacy-preserving distributed transfer learning protocol, which is appropriate for traffic data. Compared to existing works, our protocol addresses the privacy-preserving problem of transfer learning for traffic data with high sample dimensions. In addition, our protocol has fewer interaction rounds and can be proved in the semi-honest model. Finally, we validate the effectiveness, efficiency and security of the proposed protocol via experiments. Furthermore, we show the application of the proposed protocol in intelligent transportation systems.

Renewing Iterative Self-Labeling Domain Adaptation With Application to the Spine Motion Prediction

The area of transfer learning comprises supervised machine learning methods that cope with the issue when the training and testing data have different input feature spaces or distributions. In this work, we propose a novel transfer learning algorithm called Renewing Iterative Self-labeling Domain Adaptation (Re-ISDA). This work is motivated by a cervical spine motion prediction problem, with the goal of predicting the cervical spine motion of different subjects using the measurements of “exterior features”. The joint distribution of the exterior features for each subject may vary with one’s distinct BMI, sex and other characteristics; the sample size of this problem is limited due to the high experimental cost. The proposed method is well suited for transfer learning problems with limited training samples. The learning problem is formulated as a dynamic programming model and the latter is then solved by an efficient greedy algorithm. Numerical studies show that the proposed method outperforms prevailing transfer learning methods. The proposed method also achieves high prediction accuracy for the cervical spine motion problem. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper is motivated by the problem of using human body exterior features (e.g., contours, surface markers) acquired by computer vision or other measurement means to determine the underlying physics or mechanics. The goal is to build a subject-specific model to predict cervical spine motion based on surface-based measurements. Conventional supervised learning and transfer learning methods fail to give an accurate prediction due to the subject-to-subject variation and/or the small sample size of the training data. This paper proposes a novel transfer learning method called renewing iterative self-labeling domain adaptation to modify the possible error of the pseudo labels and thus make the training process more stable. The proposed method improves the accuracy by over <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$35\%$</tex-math> </inline-formula> on the traditional transfer learning methods. This method is applicable to limited-sample problems with different training and testing input distributions, and is particularly suitable for time-series data.

Heterogeneous Domain Adaptation With Generalized Similarity and Dissimilarity Regularization.

Heterogeneous domain adaptation (HDA) aims to address the transfer learning problems where the source domain and target domain are represented by heterogeneous features. The existing HDA methods based on matrix factorization have been proven to learn transferable features effectively. However, these methods only preserve the original neighbor structure of samples in each domain and do not use the label information to explore the similarity and separability between samples. This would not eliminate the cross-domain bias of samples and may mix cross-domain samples of different classes in the common subspace, misleading the discriminative feature learning of target samples. To tackle the aforementioned problems, we propose a novel matrix factorization-based HDA method called HDA with generalized similarity and dissimilarity regularization (HGSDR). Specifically, we propose a similarity regularizer by establishing the cross-domain Laplacian graph with label information to explore the similarity between cross-domain samples from the identical class. And we propose a dissimilarity regularizer based on the inner product strategy to expand the separability of cross-domain labeled samples from different classes. For unlabeled target samples, we keep their neighbor relationship to preserve the similarity and separability between them in the original space. Hence, the generalized similarity and dissimilarity regularization is built by integrating the above regularizers to facilitate cross-domain samples to form discriminative class distributions. HGSDR can more efficiently match the distributions of the two domains both from the global and sample viewpoints, thereby learning discriminative features for target samples. Extensive experiments on the benchmark datasets demonstrate the superiority of the proposed method against several state-of-the-art methods.

Some thoughts about transfer learning. What role for the source domain?

Transfer learning is called for when the training and test data do not share the same input distributions (PXS≠PXT) or/and not the same conditional ones (PY|XS≠PY|XT). In the most general case, the input spaces and/or output spaces can be different: XS≠XT and/or YS≠YT. However, most work assume that XS=XT. Furthermore, a common held assumption is that it is necessary that the source hypothesis be good on the source training data and that the “distance” between the source and the target domains be as small as possible in order to get a good (transferred) target hypothesis.This paper revisits the reasons for these beliefs and discusses the relevance of these conditions. An algorithm is presented which can deal with transfer learning problems where XS≠XT, and that furthermore brings a fresh perspective on the role of the source hypothesis (it does not have to be good) and on what is important in the distance between the source and the target domains (translations between them should belong to a limited set). Experiments illustrate the properties of the method and confirm the theoretical analysis.Determining beforehand a relevant source hypothesis remains an open problem, but the vista provided here helps understanding its role.

Optimal Transport-Embedded Neural Network for Fairness Transfer Problem

Research on neuromorphic computing has gained popularity in recent years. In particular, regularized embedded neural systems have been applied in several significant real-world situations, such as recommendation systems and transfer learning. This paper deals with the fairness transfer learning problem, which has been insufficiently explored. In fairness transfer settings, the source domain has limit-tagged training samples, which may lead to performance degradation in the target domain. To solve such problems, a linear data-augmentation-based optimal transport-embedded neural network is proposed in this paper. It can augment the source samples to make the distribution of the source domain balanced and can align the source and target distributions simultaneously. Moreover, the distribution of the augmented data by mixup is limited to a certain bound that can avoid the abnormal samples generated. The effectiveness of the proposed method has been demonstrated in several transfer learning tests, including regression and classification. In 1-shot and 3-shot classification tasks on the Office dataset, our method’s accuracy is 4.8 and 3.9% better, respectively, than the second-best model. Additionally, our model’s performance is about 2–3 percentage points superior to the second-best model in the OfficeHome dataset. It is simple yet effective, making it perfect for low-power edge AI applications.

A Bayesian approach to (online) transfer learning: Theory and algorithms

Transfer learning is a machine learning paradigm where knowledge from one problem is utilized to solve a new but related problem. While conceivable that knowledge from one task could help solve a related task, if not executed properly, transfer learning algorithms can impair the learning performance instead of improving it – commonly known as negative transfer. In this paper, we use a parametric statistical model to study transfer learning from a Bayesian perspective. Specifically, we study three variants of transfer learning problems, instantaneous, online, and time-variant transfer learning. We define an appropriate objective function for each problem and provide either exact expressions or upper bounds on the learning performance using information-theoretic quantities, which allow simple and explicit characterizations when the sample size becomes large. Furthermore, examples show that the derived bounds are accurate even for small sample sizes. The obtained bounds give valuable insights into the effect of prior knowledge on transfer learning, at least with respect to our Bayesian formulation of the transfer learning problem. In particular, we formally characterize the conditions under which negative transfer occurs. Lastly, we devise several (online) transfer learning algorithms that are amenable to practical implementations, some of which do not require the parametric assumption. We demonstrate the effectiveness of our algorithms with real data sets, focusing primarily on when the source and target data have strong similarities.

UNSUPERVISED TRANSUDATIVE TL FEATURE LEARNING FOR IMAGE FEATURE EXTRACTION AND REPRESENTATION

In this study, we address the problem of unsupervised transductive transfer learning for image feature extraction and representation. While transfer learning has shown promising results in various domains, its application to image feature extraction in an unsupervised transductive setting remains relatively unexplored. The research gap lies in the scarcity of methods that can effectively learn meaningful image representations without access to labeled data in the target domain, hindering the broader applicability of transfer learning in computer vision. Our research seeks to bridge this gap by proposing a novel framework that leverages unsupervised feature learning to enhance the adaptability of models across different image domains, thus contributing to the advancement of transfer learning in the field of computer vision. Experimental results demonstrate the effectiveness of our method in addressing this critical research gap and its potential for real-world applications.

A linear adjustment-based approach to posterior drift in transfer learning.

We present new models and methods for the posterior drift problem where the regression function in the target domain is modelled as a linear adjustment, on an appropriate scale, of that in the source domain, and study the theoretical properties of our proposed estimators in the binary classification problem. The core idea of our model inherits the simplicity and the usefulness of generalized linear models and accelerated failure time models from the classical statistics literature. Our approach is shown to be flexible and applicable in a variety of statistical settings, and can be adopted for transfer learning problems in various domains including epidemiology, genetics and biomedicine. As concrete applications, we illustrate the power of our approach (i) through mortality prediction for British Asians by borrowing strength from similar data from the larger pool of British Caucasians, using the UK Biobank data, and (ii) in overcoming a spurious correlation present in the source domain of the Waterbirds dataset.

Operational adjustment modeling approach based on Bayesian network transfer learning for new flotation process under scarce data

Establishing an intelligent operational adjustment model is significant for optimizing flotation performance and realizing the stable control of the flotation process. When sufficient historical data is available, the Bayesian network (BN) has been used to achieve this goal due to its powerful uncertainty modeling and reasoning capabilities. However, for a new flotation process in the early stage with scarce data, an accurate BN model can hardly be established to make reliable decisions. In this scenario, it is an effective way to transfer data or knowledge from other similar flotation processes to the new flotation process to help complete the target BN modeling task. Therefore, the operational adjustment modeling problem is transformed into the BN transfer learning problem. Moreover, this article proposes a new BN transfer learning approach, including structure and parameters transfer learning techniques. For structure transfer learning, a majority voting mechanism is proposed to get an integrated result of conditional independence tests and generates the network structure based on them. For parameters transfer learning, the sources are judged whether to be used to learn the parameters first, and then the suitable sources are used to learn the network parameters based on the weight determined by the similarity. The Asia network first assesses the proposed approach and then applies it to real-world flotation operational adjustment decisions. Experimental results on the Asia network show that the proposed approach can learn more accurate structure and parameters according to F1 score, Hamming distance, and KL divergence indicators compared to existing advanced methods. Furthermore, the experimental results of the flotation process show that the inference accuracy is improved from 57.1% to 85.7% compared with the traditional modeling method.

Federated transfer learning in fault diagnosis under data privacy with target self-adaptation

The past decades have witnessed great developments and applications of the data-driven machinery fault diagnosis methods. Due to the difficulties and significant expenses in collecting labeled data, multiple industrial users generally would like to collaborate for training powerful models. However, for data privacy concerns, the local data generally cannot be shared with others for centralized training. Moreover, the data at different users are usually obtained under different conditions. The model generalization problem is quite challenging since no simultaneous data access is permitted. To address these issues, a federated transfer learning method is proposed. Fake data of additional classes are generated to suppress the decision boundaries, and only the models are communicated across different clients. A prediction alignment scheme is proposed at the target client for self-adaptation. A model and instance-level consensus scheme is introduced to enhance model performance. The proposed method achieves promising knowledge transfer effects with only communications of models, and the challenging partial transfer learning problems can be also well addressed. Experiments on real-world machinery datasets validate the proposed method, which is promising for applications in the real industries.

Learning Enhanced Acoustic Latent Representation for Small Scale Affective Corpus with Adversarial Cross Corpora Integration

Achieving robust cross contexts speech emotion recognition (SER) has become a critical next direction of research for wide adoption of SER technology. The core challenge is in the large variability of affective speech that is highly contextualized. Prior works have worked on this as a transfer learning problem that mostly focuses on developing domain adaptation strategy. However, many of the existing speech emotion corpora, even those considered as large scale, are still limited in size resulting in an unsatisfactory transfer result. On the other hand, directly collecting context-specific corpus often results in an even smaller data size leading to an inevitably non-robust accuracy. In order to mitigate this issue, we propose the concept of enhancing the affect-related variability when learning the in-context acoustic latent representation by integrating out-of-context emotion data. Specifically, we utilize adversarial autoencoder network as our backbone with multiple out-of-context emotion labels derived for each in-context samples that serve as an auxiliary constraint in learning the latent representation. We extensively evaluate our framework using three in-context databases with three out-of-context databases. In this work, we demonstrate not only an improved recognition accuracy but also a comprehensive analysis on the effectiveness of this representation learning strategy.

Hybrid synthetic data generation pipeline that outperforms real data

Fine-tuning a pretrained model with real data for a machine learning task requires many hours of manual work, especially for computer vision tasks, where collection and annotation of data can be very time-consuming. We present a framework and methodology for synthetic data collection that is not only efficient in terms of time taken to collect and annotate data, making use of free- and open-source software tools and 3D assets but also beats the state-of-the-art against real data, which is the ultimate test for any similar-to-real approach. We test our approach on a set of image classes from ObjectNet, which is a challenging image classification benchmark test dataset that is designed to be similar in many respects to ImageNet but with a wider variety of viewpoints, rotations, and backgrounds, which can make it more difficult for transfer learning problems. The novelty of our approach stems from the way we create complex backgrounds for 3D models using 2D images laid out as decals in a 3D game engine, where synthetic images are captured programmatically with a large number of systematic variations. We demonstrate that our approach is highly effective, resulting in a deep learning model with a top-1 accuracy of 72% on the ObjectNet data, which is a new state-of-the-art result. In addition, we present an efficient strategy for learning rate tuning that is an order of magnitude faster than regular grid search.

Differentially private transferrable deep learning with membership-mappings

Despite a recent surge of research interest in privacy and transferrable deep learning, optimizing the tradeoff between privacy requirements and performance of machine learning models remains a challenge. This motivates the development of an approach that optimizes both privacy-preservation mechanism and learning of the deep models for achieving a robust performance. This paper considers the problem of semi-supervised transfer and multi-task learning under differential privacy framework. An alternative conception of deep autoencoder, referred to as Conditionally Deep Membership-Mapping Autoencoder (CDMMA), is considered for transferrable deep learning. Under practice-oriented settings, an analytical solution for the learning of CDMMA can be derived by means of variational optimization. The paper proposes a transfer and multi-task learning approach that combines CDMMA with a tailored noise adding mechanism to transfer knowledge from source to target domain in a privacy-preserving manner.

Curriculum learning for human compositional generalization

Generalization (or transfer) is the ability to repurpose knowledge in novel settings. It is often asserted that generalization is an important ingredient of human intelligence, but its extent, nature, and determinants have proved controversial. Here, we examine this ability with a paradigm that formalizes the transfer learning problem as one of recomposing existing functions to solve unseen problems. We find that people can generalize compositionally in ways that are elusive for standard neural networks and that human generalization benefits from training regimes in which items are axis aligned and temporally correlated. We describe a neural network model based around a Hebbian gating process that can capture how human generalization benefits from different training curricula. We additionally find that adult humans tend to learn composable functions asynchronously, exhibiting discontinuities in learning that resemble those seen in child development.