Existing Transfer Learning Methods Research Articles

Selective Random Walk for Transfer Learning in Heterogeneous Label Spaces.

Transfer learning has been widely used in different scenarios, especially in those lacking enough labeled data. However, most of the existing transfer learning methods are based on the assumption that the source and target domains should share the label space entirely or partially, which greatly limits their application scopes. In this article, a Selective Random Walk (SRW) method for transfer learning in heterogeneous label spaces is proposed to make full use of unlabeled auxiliary data, which acts as a bridge for knowledge transfer from the source domain to the target domain. The proposed SRW method can explicitly identify transfer sequences between source and target instances via auxiliary instances based on random walk techniques. Since not all of the transfer sequences generated by random walk are credible for the target task, the SRW method can learn to weight transfer sequences adaptively. Based on the weights of the transfer sequences, the SRW method leverages knowledge by forcing adjacent data points in the transfer sequence to be similar and making the target data point in the sequence represented by other data points in the same sequence. Experiments show that the SRW method outperforms state-of-the-art models in plenty of transfer learning tasks with heterogeneous label spaces constructed within and across several benchmark datasets.

DynamicAug: Enhancing Transfer Learning Through Dynamic Data Augmentation Strategies Based on Model State

Transfer learning has made significant advancements, however, the issue of overfitting continues to pose a major challenge. Data augmentation has emerged as a highly promising technique to counteract this challenge. Current data augmentation methods are fixed in nature, requiring manual determination of the appropriate intensity prior to the training process. However, this entails substantial computational costs. Additionally, as the model approaches convergence, static data augmentation strategies can become suboptimal. In this paper, we introduce the concept of Dynamic Data Augmentation (DynamicAug), a method that autonomously adjusts the intensity of data augmentation, taking into account the convergence state of the model. During each iteration of the model’s forward pass, we utilize a Gaussian distribution based sampler to stochastically sample the current intensity of data augmentation. To ensure that the sampled intensity is aligned with the convergence state of the model, we introduce a learnable expectation to the sampler and update the expectation iteratively. In order to assess the convergence status of the model, we introduce a novel loss function called the convergence loss. Through extensive experiments conducted over 27 vision datasets, we have demonstrated that DynamicAug can significantly enhance the performance of existing transfer learning methods.

Distributed Heterogeneous Transfer Learning

Transfer learning has proved to be effective for building predictive models even in complex conditions with a low amount of available labeled data, by constructing a predictive model for a target domain also using the knowledge coming from a separate domain, called source domain. However, several existing transfer learning methods assume identical feature spaces between the source and the target domains. This assumption limits the possible real-world applications of such methods, since two separate, although related, domains could be described by totally different feature spaces. Heterogeneous transfer learning methods aim to overcome this limitation, but they usually i) make other assumptions on the features, such as requiring the same number of features, ii) are not generally able to distribute the workload over multiple computational nodes, iii) cannot work in the Positive-Unlabeled (PU) learning setting, which we also considered in this study, or iv) their applicability is limited to specific application domains, i.e., they are not general-purpose methods.In this manuscript, we present a novel distributed heterogeneous transfer learning method, implemented in Apache Spark, that overcomes all the above-mentioned limitations. Specifically, it is able to work also in the PU learning setting by resorting to a clustering-based approach, and can align totally heterogeneous feature spaces, without exploiting peculiarities of specific application domains. Moreover, our distributed approach allows us to process large source and target datasets.Our experimental evaluation was performed in three different application domains that can benefit from transfer learning approaches, namely the reconstruction of the human gene regulatory network, the prediction of cerebral stroke in hospital patients, and the prediction of customer energy consumption in power grids. The results show that the proposed approach is able to outperform 4 state-of-the-art heterogeneous transfer learning approaches and 3 baselines, and exhibits ideal performances in terms of scalability.

Cross-sensor transfer learning for fire smoke scene detection using variable-bands multi-spectral satellite imagery aided by spectral patterns

ABSTRACT This paper addresses the challenge of training deep learning models for fire smoke scene detection from multi-sensor, multi-spectral satellite imagery, where spectral bands vary and training data is scarce, especially for new sensors. We introduce a novel cross-sensor transfer learning approach enhanced by spectral pattern extraction, notably aided by a deep learning (DL) module called Input Amplification (IA). Our contribution includes the creation of two datasets, Landsat_smk2770 and Sentinel2_smk, with varying spectral bands. We employ a model named IA_VIB_SD which incorporates IA and is designed specifically for fire smoke scene detection. We investigated the transferability of the Landsat_smk2770-trained IA_VIB_SD model to the Sentinel2_smk dataset using various transfer learning techniques, and compared the performance of the transferred model with the benchmark IA_VIB_SD model exclusively trained on pure Sentinel2_smk data. Our proposed transfer learning approach resulted in a transferred model with an average accuracy 5% higher than the benchmark model. Notably, Our proposed transfer learning approach outperformed existing transfer learning methods by more than 1% in terms of the accuracy, even when trained on a mere 10% of the Sentinel2_smk dataset. This research advances fire smoke scene detection from multi-sensor, multi-spectral satellite imagery.

Multitree Genetic Programming With Feature-Based Transfer Learning for Symbolic Regression on Incomplete Data.

Data incompleteness is a serious challenge in real-world machine-learning tasks. Nevertheless, it has not received enough attention in symbolic regression (SR). Data missingness exacerbates data shortage, especially in domains with limited available data, which in turn limits the learning ability of SR algorithms. Transfer learning (TL), which aims to transfer knowledge across tasks, is a potential solution to solve this issue by making amends for the lack of knowledge. However, this approach has not been adequately investigated in SR. To fill this gap, a multitree genetic programming-based TL method is proposed in this work to transfer knowledge from complete source domains (SDs) to incomplete related target domains (TDs). The proposed method transforms the features from a complete SD to an incomplete TD. However, having many features complicates the transformation process. To mitigate this problem, we integrate a feature selection mechanism to eliminate unnecessary transformations. The method is examined on real-world and synthetic SR tasks with missing values to consider different learning scenarios. The obtained results not only show the effectiveness of the proposed method but also show its training efficiency compared with the existing TL methods. Compared to state-of-the-art methods, the proposed method reduced an average of more than 2.58% and 4% regression error on heterogeneous and homogeneous domains, respectively.

Selective transfer subspace learning for small-footprint end-to-end cross-domain keyword spotting

In small-footprint end-to-end keyword spotting, it is often expensive and time-consuming to acquire sufficient labels in various speech scenarios. To overcome this problem, transfer learning leverages the rich knowledge of the auxiliary domain to annotate the unlabeled target data. However, most existing transfer learning methods typically learn a domain-invariant feature representation while ignoring the negative transfer problem. In this paper, we propose a new and general cross-domain keyword spotting framework called selective transfer subspace learning (STSL) that avoid negative transfer and dramatically improve the accuracy for cross-domain keyword spotting by actively selecting appropriate source samples. Specifically, STSL first aligns geometrical relationship and weighted distribution discrepancy to learn a domain-invariant projection subspace. Then, it actively selects appropriate source samples that are similar to the target domain for transfer learning to avoid negative transfer. Finally, we formulate a minimization problem that alternately optimizes the projection subspace and source active selection, giving an effective optimization. Experimental results on 10 groups of cross-domain keyword spotting tasks show that our STSL outperforms some state-of-the-art transfer learning methods and no transfer learning methods.

Fault state identification of rolling bearings based on deep transfer convolutional autoencoder and a new health indicator

For the problem of bearing fault state identification, in recent years, the main research is how to distinguish the normal state and the fault state. However, it is still difficult to accurately identify the more detailed fault severity in the cross-working environment. Therefore, this paper proposes a new health indicator (HI) and deep transferable convolutional autoencoder network (DTCAE) to solve the problem of rolling bearing early fault monitoring and fault state recognition. Firstly, the envelope spectrum of the vibration signal is calculated, and then the ratio of the energy at the maximum amplitude of the envelope spectrum to other energies is obtained, and the final HI is obtained by weighting the standard deviation with it, which has good generalization and engineering practicability. HI is used for early bearing fault monitoring and classifying bearing fault status in source domain. Finally, aiming at the problem that the existing transfer learning methods do not consider the weakening of the target domain data features after domain adaptation, DTCAE is proposed to identify the bearing fault state. Specifically, in addition to the distribution difference loss and classification loss commonly seen in transfer learning models, a decoder is also added after feature extraction to calculate the difference between the reconstructed data of the target domain and the original data, and the feature distribution difference between the reconstructed source domain and the target domain is also calculated. This allows the source-domain bearing and the target-domain bearing features to be matched in such a way that both not only have the same feature distribution, but also retain the degradation trend characteristics of the target-domain bearing. Experiments in three challenging cases have achieved good results and are superior to some existing methods.

An Optimization Method of Deep Transfer Learning for Vegetation Segmentation under Rainy and Dry Season Differences in a Dry Thermal Valley.

Deep learning networks might require re-training for different datasets, consuming significant manual labeling and training time. Transfer learning uses little new data and training time to enable pre-trained network segmentation in relevant scenarios (e.g., different vegetation images in rainy and dry seasons); however, existing transfer learning methods lack systematicity and controllability. So, an MTPI method (Maximum Transfer Potential Index method) was proposed to find the optimal conditions in data and feature quantity for transfer learning (MTPI conditions) in this study. The four pre-trained deep networks (Seg-Net (Semantic Segmentation Networks), FCN (Fully Convolutional Networks), Mobile net v2, and Res-Net 50 (Residual Network)) using the rainy season dataset showed that Res-Net 50 had the best accuracy with 93.58% and an WIoU (weight Intersection over Union) of 88.14%, most worthy to transfer training in vegetation segmentation. By obtaining each layer's TPI performance (Transfer Potential Index) of the pre-trained Res-Net 50, the MTPI method results show that the 1000-TDS and 37-TP were estimated as the best training speed with the smallest dataset and a small error risk. The MTPI transfer learning results show 91.56% accuracy and 84.86% WIoU with 90% new dataset reduction and 90% iteration reduction, which is informative for deep networks in segmentation tasks between complex vegetation scenes.

Fault Diagnosis of Planetary Gearbox Based on Dynamic Simulation and Partial Transfer Learning.

To address the problem of insufficient real-world data on planetary gearboxes, which makes it difficult to diagnose faults using deep learning methods, it is possible to obtain sufficient simulation fault data through dynamic simulation models and then reduce the difference between simulation data and real data using transfer learning methods, thereby applying diagnostic knowledge from simulation data to real planetary gearboxes. However, the label space of real data may be a subset of the label space of simulation data. In this case, existing transfer learning methods are susceptible to interference from outlier label spaces in simulation data, resulting in mismatching. To address this issue, this paper introduces multiple domain classifiers and a weighted learning scheme on the basis of existing domain adversarial transfer learning methods to evaluate the transferability of simulation data and adaptively measure their contribution to label predictor and domain classifiers, filter the interference of unrelated categories of simulation data, and achieve accurate matching of real data. Finally, partial transfer experiments are conducted to verify the effectiveness of the proposed method, and the experimental results show that the diagnostic accuracy of this method is higher than existing transfer learning methods.

Knowledge sharing-based multi-block federated learning for few-shot oil layer identification

Oil Layer Identification (OLI) plays a significant role in petroleum development, but is unavoidably negatively affected by small samples, non-reservoirs interference, and layer classes imbalance. Existing transfer learning methods partially address the problem of insufficient samples in OLI. Nevertheless, they ignore the geological differences between blocks. This paper introduces Multi-Block Federated Learning (MBFL) to train a generalized pre-trained model, which consists of a Mask Attention Network (MAN), Class Balance Module (CBM), and Dynamic Weighted Fusion (DWF). MBFL uses MAN to dynamically avoid non-reservoir interference while learning the relationship between reservoirs and non-reservoirs; MBFL uses CBM to overcome layer classes imbalance and uses DWF to optimize traditional federated learning for addressing geological differences among blocks. Experimental results demonstrate that MBFL achieves an average accuracy of 89.32%, and an average F1 score of 81.74% with a real-world dataset. The results of two public datasets demonstrate the generalization of MBFL.

Meta-transfer learning for emotion recognition

Deep learning has been widely adopted in automatic emotion recognition and has lead to significant progress in the field. However, due to insufficient training data, pre-trained models are limited in their generalisation ability, leading to poor performance on novel test sets. To mitigate this challenge, transfer learning performed by fine-tuning pr-etrained models on novel domains has been applied. However, the fine-tuned knowledge may overwrite and/or discard important knowledge learnt in pre-trained models. In this paper, we address this issue by proposing a PathNet-based meta-transfer learning method that is able to (i) transfer emotional knowledge learnt from one visual/audio emotion domain to another domain and (ii) transfer emotional knowledge learnt from multiple audio emotion domains to one another to improve overall emotion recognition accuracy. To show the robustness of our proposed method, extensive experiments on facial expression-based emotion recognition and speech emotion recognition are carried out on three bench-marking data sets: SAVEE, EMODB, and eNTERFACE. Experimental results show that our proposed method achieves superior performance compared with existing transfer learning methods.

Contrastive domain adaptation with consistency match for automated pneumonia diagnosis.

Pneumonia can be difficult to diagnose since its symptoms are too variable, and the radiographic signs are often very similar to those seen in other illnesses such as a cold or influenza. Deep neural networks have shown promising performance in automated pneumonia diagnosis using chest X-ray radiography, allowing mass screening and early intervention to reduce the severe cases and death toll. However, they usually require many well-labelled chest X-ray images for training to achieve high diagnostic accuracy. To reduce the need for training data and annotation resources, we propose a novel method called Contrastive Domain Adaptation with Consistency Match (CDACM). It transfers the knowledge from different but relevant datasets to the unlabelled small-size target dataset and improves the semantic quality of the learnt representations. Specifically, we design a conditional domain adversarial network to exploit discriminative information conveyed in the predictions to mitigate the domain gap between the source and target datasets. Furthermore, due to the small scale of the target dataset, we construct a feature cloud for each target sample and leverage contrastive learning to extract more discriminative features. Lastly, we propose adaptive feature cloud expansion to push the decision boundary to a low-density area. Unlike most existing transfer learning methods that aim only to mitigate the domain gap, our method instead simultaneously considers the domain gap and the data deficiency problem of the target dataset. The conditional domain adaptation and the feature cloud generation of our method are learning jointly to extract discriminative features in an end-to-end manner. Besides, the adaptive feature cloud expansion improves the model's generalisation ability in the target domain. Extensive experiments on pneumonia and COVID-19 diagnosis tasks demonstrate that our method outperforms several state-of-the-art unsupervised domain adaptation approaches, which verifies the effectiveness of CDACM for automated pneumonia diagnosis using chest X-ray imaging.

Variable-Modulation Specific Emitter Identification With Domain Adaptation

Specific emitter identification (SEI) is a technique of identifying individual emitters via unique characteristics of different emitters. In this paper, we consider a SEI problem with transmitter changing modulations scenario. There have been few previous studies on this type of scenario. To cope with the daunting challenge, a variable-modulation SEI framework with domain adaptation is proposed. The components characteristics of transmitter are analyzed and the distortion models are established for simulation dataset generation. The received in-phase/quadrature (I/Q) signals are demodulated and reconstructed to obtain baseband ideal modulation signals. The received signals and the ideal modulation signals corresponding to demodulation and reconstruction are merged and embedded into the feature extraction network. Domain adversarial neural network (DANN) is added into the SEI framework to generate domain-invariant fingerprint features, thus realizing variable-modulation SEI. To better align the distortion features of emitters with variable modulations, Gaussian Encoder is designed to project fingerprint features into Gaussian distribution space. Numerous experiments show that the proposed SEI framework can improve recognition accuracy of individual emitter for single modulation and variable transfer greatly, and outperform the existing transfer learning methods. The ablation study demonstrates the components of framework are complementary. The complexity of framework is acceptable and it can extend to large-scale use. The robustness of framework is verified through modulation transfer among PSK and QAM.

Transfer Learning on Electromyography (EMG) Tasks: Approaches and Beyond.

Machine learning on electromyography (EMG) has recently achieved remarkable success on various tasks, while such success relies heavily on the assumption that the training and future data must be of the same data distribution. However, this assumption may not hold in many real-world applications. Model calibration is required via data re-collection and label annotation, which is generally very expensive and time-consuming. To address this issue, transfer learning (TL), which aims to improve target learners' performance by transferring knowledge from related source domains, is emerging as a new paradigm to reduce the amount of calibration effort. This survey assesses the eligibility of more than fifty published peer-reviewed representative transfer learning approaches for EMG applications. Unlike previous surveys on purely transfer learning or EMG-based machine learning, this survey aims to provide insight into the biological foundations of existing transfer learning methods on EMG-related analysis. Specifically, we first introduce the muscles' physiological structure, the EMG generating mechanism, and the recording of EMG to provide biological insights behind existing transfer learning approaches. Further, we categorize existing research endeavors into data based, model based, training scheme based, and adversarial based. This survey systematically summarizes and categorizes existing transfer learning approaches for EMG related machine learning applications. In addition, we discuss possible drawbacks of existing works and point out the future direction of better EMG transfer learning algorithms to enhance practicality for real-world applications.

Cross-scenario automatic sleep stage classification using transfer learning and single-channel EEG

Sleep staging via manual inspection is time-consuming and inefficient, and as such, automatic sleep stage classification methods have been proposed and successfully implemented. However, these conventional supervised learning models are not suitable for cross-scenario sleep staging tasks due to the shift data distribution problem. This study aims to explore the use of transfer learning to transfer knowledge from a labeled source dataset to a new target domain where labels are difficult to obtain, so as to help solve the classification problem in a new domain in practical sleep staging scenarios. A novel end-to-end deep learning network that includes a feature extractor, a sleep stage classifier, and a domain adaptation network was proposed. The proposed domain adaptation network can accomplish distribution alignment between source and target domains, through the application of both joint distribution loss and stage transition loss. Cross-channel, cross-subject and channel, and cross-dataset experiments were designed to verify the feasibility of the proposed network using three publicly available datasets. The results indicate an average accuracy improvement ranging from 0.1% to 6.1% (average of 2.9%), a macro-averaging F1-score improvement ranging from 0.1% to 5.1% (average of 2.6%), and a Cohen’s kappa coefficient improvement ranging from 0.016 to 0.083 (average of 0.045) after using transfer learning, when compared with a non-transfer model. Relative to similar existing transfer learning methods for sleep staging, the proposed method was implemented in an unsupervised manner and achieved success in cross-scenario sleep staging tasks, which is more suitable for practical applications in daily life.

Multi-source manifold feature transfer learning with domain selection for brain-computer interfaces

Transfer learning uses the knowledge in source domains to improve the learning performance in the target domain, which is useful in electroencephalogram (EEG) based brain-computer interfaces (BCIs) with small training datasets. However, the existing transfer learning methods for EEG based BCI mainly consider the knowledge transfer from single-to-single (STS) domain or simply merge different source domains into a bigger one. In this paper, we propose a multi-source manifold feature transfer learning (MMFT) framework to transfer multi-source knowledge for EEG signals classification. MMFT minimizes marginal probability distribution on the Riemannian manifold using Riemannian alignment and Grassmann manifold feature learning, then transfers the manifold features with a conditional probability distribution adaptation in the structural risk minimization (SRM) function. Based on MMFT, w-MMFT is proposed to tackle the class imbalance issue for SRM, and the label similarity analysis (LSA) is proposed to select source domains for MMFT, forming a new LSA-MMFT framework. Experimental results on six datasets demonstrate that the proposed MMFT has achieved superior performance in classification accuracy and computational efficiency compared to state-of-the-art methods. The LSA-MMFT can get more stable performance than two other domain selection methods.

Multi-Source Contribution Learning for Domain Adaptation.

Transfer learning becomes an attractive technology to tackle a task from a target domain by leveraging previously acquired knowledge from a similar domain (source domain). Many existing transfer learning methods focus on learning one discriminator with single-source domain. Sometimes, knowledge from single-source domain might not be enough for predicting the target task. Thus, multiple source domains carrying richer transferable information are considered to complete the target task. Although there are some previous studies dealing with multi-source domain adaptation, these methods commonly combine source predictions by averaging source performances. Different source domains contain different transferable information; they may contribute differently to a target domain compared with each other. Hence, the source contribution should be taken into account when predicting a target task. In this article, we propose a novel multi-source contribution learning method for domain adaptation (MSCLDA). As proposed, the similarities and diversities of domains are learned simultaneously by extracting multi-view features. One view represents common features (similarities) among all domains. Other views represent different characteristics (diversities) in a target domain; each characteristic is expressed by features extracted in a source domain. Then multi-level distribution matching is employed to improve the transferability of latent features, aiming to reduce misclassification of boundary samples by maximizing discrepancy between different classes and minimizing discrepancy between the same classes. Concurrently, when completing a target task by combining source predictions, instead of averaging source predictions or weighting sources using normalized similarities, the original weights learned by normalizing similarities between source and target domains are adjusted using pseudo target labels to increase the disparities of weight values, which is desired to improve the performance of the final target predictor if the predictions of sources exist significant difference. Experiments on real-world visual data sets demonstrate the superiorities of our proposed method.

Repetition-Based Approach for Task Adaptation in Imitation Learning.

Transfer learning is an effective approach for adapting an autonomous agent to a new target task by transferring knowledge learned from the previously learned source task. The major problem with traditional transfer learning is that it only focuses on optimizing learning performance on the target task. Thus, the performance on the target task may be improved in exchange for the deterioration of the source task’s performance, resulting in an agent that is not able to revisit the earlier task. Therefore, transfer learning methods are still far from being comparable with the learning capability of humans, as humans can perform well on both source and new target tasks. In order to address this limitation, a task adaptation method for imitation learning is proposed in this paper. Being inspired by the idea of repetition learning in neuroscience, the proposed adaptation method enables the agent to repeatedly review the learned knowledge of the source task, while learning the new knowledge of the target task. This ensures that the learning performance on the target task is high, while the deterioration of the learning performance on the source task is small. A comprehensive evaluation over several simulated tasks with varying difficulty levels shows that the proposed method can provide high and consistent performance on both source and target tasks, outperforming existing transfer learning methods.

Doubly Supervised Transfer Classifier for Computer-Aided Diagnosis With Imbalanced Modalities.

Transfer learning (TL) can effectively improve diagnosis accuracy of single-modal-imaging-based computer-aided diagnosis (CAD) by transferring knowledge from other related imaging modalities, which offers a way to alleviate the small-sample-size problem. However, medical imaging data generally have the following characteristics for the TL-based CAD: 1) The source domain generally has limited data, which increases the difficulty to explore transferable information for the target domain; 2) Samples in both domains often have been labeled for training the CAD model, but the existing TL methods cannot make full use of label information to improve knowledge transfer. In this work, we propose a novel doubly supervised transfer classifier (DSTC) algorithm. In particular, DSTC integrates the support vector machine plus (SVM+) classifier and the low-rank representation (LRR) into a unified framework. The former makes full use of the shared labels to guide the knowledge transfer between the paired data, while the latter adopts the block-diagonal low-rank (BLR) to perform supervised TL between the unpaired data. Furthermore, we introduce the Schatten-p norm for BLR to obtain a tighter approximation to the rank function. The proposed DSTC algorithm is evaluated on the Alzheimer's disease neuroimaging initiative (ADNI) dataset and the bimodal breast ultrasound image (BBUI) dataset. The experimental results verify the effectiveness of the proposed DSTC algorithm.

A weak prior embedding-based method for transfer fault diagnosis of rolling bearing

Transfer learning extends the application scope of deep intelligent fault diagnosis models. However, when operation conditions are significantly varied, existing transfer learning methods may encounter a bottleneck. The unsupervised methods will mismatch features, while supervised methods are constrained by insufficient target labeled instances. A weak prior embedding-based domain adaption network (WPEDAN) is proposed to resolve this problem. Specifically, the cluster ability of source data pre-trained network to target data is emphasized and enhanced directly and indirectly so as to obtain discriminative feature clusters. The target prior template samples are embedded and automatically delivered to the most relevant feature clusters, so the class information of target instances can be estimated based on similarity and supervised training on target is following. Moreover, an improved conditional maximum mean discrepancy (ICMMD) is developed to further align conditional distribution. Experiments on two challenging bearing datasets show that proposed method achieves more than 99% diagnosis accuracy in all tasks, and is significantly ahead of the comparison methods in complex transfer tasks.