Existing Transfer Learning Research Articles

Robust manifold discriminative distribution adaptation for transfer subspace learning

Transfer learning is an effective means by leveraging rich information of the labeled samples from the source domain to build an appropriate classifier for annotating the target domain. Existing transfer learning algorithms usually try to minimize the distribution difference between different domains by means of Maximum Mean Discrepancy (MMD) based metric criteria. However, the domain adaptation methods based on MMD usually have the following two problems: 1) the local geometry information of the inter-domain data is ignored, which may cause to negative transfer effect. 2) The classifier and the feature learning are not integrated into one framework, in which the global optimal solution may be not obtained. Therefore, a Robust Manifold Discriminative Distribution Adaptation (RMDDA) model is presented for transfer subspace learning. Because the difference between domains leads to the large deviation of the original features of data in the source and target domains, it is difficult to construct the manifold structure of inter-domain data. A new feature termed Histogram Feature of Neighbors (HFON) is firstly introduced to rightly capture the domain locality. Secondly, the marginal distribution, conditional distribution, and discriminative regression are integrated into a model, which can solve the problem of separating feature learning from classifier training. It is considered that the dimension of feature extraction is limited by the number of classes which is determined by discriminative regression, we introduce double projection matrices into the model. Lastly, the numerous experiments on six benchmark cross-domain image datasets prove the superiority of RMDDA algorithm compared with some state-of-the-art algorithms.

An Optimization Method of Deep Transfer Learning for Vegetation Segmentation under Rainy and Dry Season Differences in a Dry Thermal Valley.

Deep learning networks might require re-training for different datasets, consuming significant manual labeling and training time. Transfer learning uses little new data and training time to enable pre-trained network segmentation in relevant scenarios (e.g., different vegetation images in rainy and dry seasons); however, existing transfer learning methods lack systematicity and controllability. So, an MTPI method (Maximum Transfer Potential Index method) was proposed to find the optimal conditions in data and feature quantity for transfer learning (MTPI conditions) in this study. The four pre-trained deep networks (Seg-Net (Semantic Segmentation Networks), FCN (Fully Convolutional Networks), Mobile net v2, and Res-Net 50 (Residual Network)) using the rainy season dataset showed that Res-Net 50 had the best accuracy with 93.58% and an WIoU (weight Intersection over Union) of 88.14%, most worthy to transfer training in vegetation segmentation. By obtaining each layer's TPI performance (Transfer Potential Index) of the pre-trained Res-Net 50, the MTPI method results show that the 1000-TDS and 37-TP were estimated as the best training speed with the smallest dataset and a small error risk. The MTPI transfer learning results show 91.56% accuracy and 84.86% WIoU with 90% new dataset reduction and 90% iteration reduction, which is informative for deep networks in segmentation tasks between complex vegetation scenes.

Patch-Level Consistency Regularization in Self-Supervised Transfer Learning for Fine-Grained Image Recognition

Fine-grained image recognition aims to classify fine subcategories belonging to the same parent category, such as vehicle model or bird species classification. This is an inherently challenging task because a classifier must capture subtle interclass differences under large intraclass variances. Most previous approaches are based on supervised learning, which requires a large-scale labeled dataset. However, such large-scale annotated datasets for fine-grained image recognition are difficult to collect because they generally require domain expertise during the labeling process. In this study, we propose a self-supervised transfer learning method based on Vision Transformer (ViT) to learn finer representations without human annotations. Interestingly, it is observed that existing self-supervised learning methods using ViT (e.g., DINO) show poor patch-level semantic consistency, which may be detrimental to learning finer representations. Motivated by this observation, we propose a consistency loss function that encourages patch embeddings of the overlapping area between two augmented views to be similar to each other during self-supervised learning on fine-grained datasets. In addition, we explore effective transfer learning strategies to fully leverage existing self-supervised models trained on large-scale labeled datasets. Contrary to the previous literature, our findings indicate that training only the last block of ViT is effective for self-supervised transfer learning. We demonstrate the effectiveness of our proposed approach through extensive experiments using six fine-grained image classification benchmark datasets, including FGVC Aircraft, CUB-200-2011, Food-101, Oxford 102 Flowers, Stanford Cars, and Stanford Dogs. Under the linear evaluation protocol, our method achieves an average accuracy of 78.5%, outperforming the existing transfer learning method, which yields 77.2%.

Fault Diagnosis of Planetary Gearbox Based on Dynamic Simulation and Partial Transfer Learning.

To address the problem of insufficient real-world data on planetary gearboxes, which makes it difficult to diagnose faults using deep learning methods, it is possible to obtain sufficient simulation fault data through dynamic simulation models and then reduce the difference between simulation data and real data using transfer learning methods, thereby applying diagnostic knowledge from simulation data to real planetary gearboxes. However, the label space of real data may be a subset of the label space of simulation data. In this case, existing transfer learning methods are susceptible to interference from outlier label spaces in simulation data, resulting in mismatching. To address this issue, this paper introduces multiple domain classifiers and a weighted learning scheme on the basis of existing domain adversarial transfer learning methods to evaluate the transferability of simulation data and adaptively measure their contribution to label predictor and domain classifiers, filter the interference of unrelated categories of simulation data, and achieve accurate matching of real data. Finally, partial transfer experiments are conducted to verify the effectiveness of the proposed method, and the experimental results show that the diagnostic accuracy of this method is higher than existing transfer learning methods.

Deep cross-domain transfer for emotion recognition via joint learning

Deep learning has been applied to achieve significant progress in emotion recognition from multimedia data. Despite such substantial progress, existing approaches are hindered by insufficient training data, leading to weak generalisation under mismatched conditions. To address these challenges, we propose a learning strategy which jointly transfers emotional knowledge learnt from rich datasets to source-poor datasets. Our method is also able to learn cross-domain features, leading to improved recognition performance. To demonstrate the robustness of the proposed learning strategy, we conducted extensive experiments on several benchmark datasets including eNTERFACE, SAVEE, EMODB, and RAVDESS. Experimental results show that the proposed method surpassed existing transfer learning schemes by a significant margin.

Enhancing sugarcane disease classification with ensemble deep learning: A comparative study with transfer learning techniques

Deep learning practices in the agriculture sector can address many challenges faced by the farmers such as disease detection, yield estimation, soil profile estimation, etc. In this paper, disease classification for the sugarcane plant and the experimentation involved thereby is thoroughly discussed. Experimental results include the performances of the well-known existing transfer learning techniques and proposed ensemble deep learning based architecture that incorporates stack ensemble of two networks with one having level-wise spatial attention helping to provide better generalization. A Self-created database of sugarcane leaf diseases is introduced to the research community through this paper. It involves 5 categories with a total of 2569 images. Here, it is observed that best performing transfer learning method, MobileNet-V2 shows an accuracy of around 84% with the lowest number of parameters whereas ensemble model reaching to 86.53% with less epochs and with acceptable number of parameters.

Knowledge sharing-based multi-block federated learning for few-shot oil layer identification

Oil Layer Identification (OLI) plays a significant role in petroleum development, but is unavoidably negatively affected by small samples, non-reservoirs interference, and layer classes imbalance. Existing transfer learning methods partially address the problem of insufficient samples in OLI. Nevertheless, they ignore the geological differences between blocks. This paper introduces Multi-Block Federated Learning (MBFL) to train a generalized pre-trained model, which consists of a Mask Attention Network (MAN), Class Balance Module (CBM), and Dynamic Weighted Fusion (DWF). MBFL uses MAN to dynamically avoid non-reservoir interference while learning the relationship between reservoirs and non-reservoirs; MBFL uses CBM to overcome layer classes imbalance and uses DWF to optimize traditional federated learning for addressing geological differences among blocks. Experimental results demonstrate that MBFL achieves an average accuracy of 89.32%, and an average F1 score of 81.74% with a real-world dataset. The results of two public datasets demonstrate the generalization of MBFL.

Non-IID Transfer Learning on Graphs

Transfer learning refers to the transfer of knowledge or information from a relevant source domain to a target domain. However, most existing transfer learning theories and algorithms focus on IID tasks, where the source/target samples are assumed to be independent and identically distributed. Very little effort is devoted to theoretically studying the knowledge transferability on non-IID tasks, e.g., cross-network mining. To bridge the gap, in this paper, we propose rigorous generalization bounds and algorithms for cross-network transfer learning from a source graph to a target graph. The crucial idea is to characterize the cross-network knowledge transferability from the perspective of the Weisfeiler-Lehman graph isomorphism test. To this end, we propose a novel Graph Subtree Discrepancy to measure the graph distribution shift between source and target graphs. Then the generalization error bounds on cross-network transfer learning, including both cross-network node classification and link prediction tasks, can be derived in terms of the source knowledge and the Graph Subtree Discrepancy across domains. This thereby motivates us to propose a generic graph adaptive network (GRADE) to minimize the distribution shift between source and target graphs for cross-network transfer learning. Experimental results verify the effectiveness and efficiency of our GRADE framework on both cross-network node classification and cross-domain recommendation tasks.

Pseudo-label self-training model for transfer learning algorithm

When aligning joint distributions between domains, the existing transfer learning algorithms usually assign pseudo labels due to the lack of labels in target domain. However, the noise in pseudo labels will affect the performance of transfer learning. Pseudo-label self-training for transfer learning (PST-TL) model is proposed to generate reliable pseudo labels for target domain and have a wide range of applications in existing algorithms. Pseudo labels are predicted by an ensemble classifier using absolute majority vote, and labels predicted successfully are considered to be high confidence. The training of ensemble classifier applies the self-training of joint pseudo labels strategy, adding strongly stable data to training set of the classifier. The semi-supervised and unsupervised transfer learning tasks in experiment show that the existing transfer learning algorithm can significantly improve the transfer performance after embedded by PST-TL model.

Lightweight Source-Free Transfer for Privacy-Preserving Motor Imagery Classification

Transfer learning utilizes data or knowledge in one problem to help solve a related problem. It is particularly useful in electroencephalogram (EEG) based motor imagery (MI) classification, to handle high intra-subject and/or cross-subject variations. This paper considers offline unsupervised cross-subject MI classification, i.e., we have labeled EEG trials from several source subjects, but only unlabeled EEG trials from the target subject. Existing transfer learning approaches usually make use of the source domain data directly in target model learning. To protect the privacy of the source subjects, we propose lightweight source-free transfer (LSFT), which first generates source models locally and encapsulates them as model application programming interfaces (APIs), then constructs a virtual intermediate domain to transfer the knowledge in the source domains to the target domain, and finally performs feature adaptation learning. Compared with existing deep transfer learning approaches, LSFT does not need to transfer from massive source data or models, is computationally efficient, and has a small number of parameters. Experiments on four benchmark MI datasets demonstrated that LSFT outperformed 13 different approaches, including several state-of-the-art transfer learning approaches that make use of the source domain samples or model parameters directly.

To Transfer or Not to Transfer and Why? Meta-Transfer Learning for Explainable and Controllable Cross-Individual Activity Recognition

Human activity recognition (HAR) plays a central role in ubiquitous computing applications such as health monitoring. In the real world, it is impractical to perform reliably and consistently over time across a population of individuals due to the cross-individual variation in human behavior. Existing transfer learning algorithms suffer the challenge of “negative transfer”. Moreover, these strategies are entirely black-box. To tackle these issues, we propose X-WRAP (eXplain, Weight and Rank Activity Prediction), a simple but effective approach for cross-individual HAR, which improves the performance, transparency, and ease of control for stakeholders in HAR. X-WRAP works by wrapping transfer learning into a meta-learning loop that identifies the approximately optimal source individuals. The candidate source domains are ranked using a linear scoring function based on interpretable meta-features capturing the properties of the source domains. X-WRAP is optimized using Bayesian optimization. Experiments conducted on a publicly available dataset show that the model can effectively improve the performance of transfer learning models consistently. In addition, X-WRAP can provide interpretable analysis according to the meta-features, making it possible for stakeholders to get a high-level understanding of selective transfer. In addition, an extensive empirical analysis demonstrates the promise of the approach to outperform in data-sparse situations.

Multi-Class Transfer Learning and Domain Selection for Cross-Subject EEG Classification

Transfer learning (TL) has been proven to be one of the most significant techniques for cross-subject classification in electroencephalogram (EEG)-based brain-computer interfaces (BCI). Hence, it is widely used to address the challenges of cross-session and cross-subject variability with more accurate intention prediction. In this case, TL utilizes knowledge (signal features) in the source domain(s) to improve the classification in the target domain. However, current existing transfer learning approaches on EEG-based BCI are mostly limited to two-class cross-subject classification problems, while multi-class problems are only implemented with a focus on within-subject classification due to the complexity of multi-class cross-subject classification problems. In this paper, we first extended the transfer learning approaches to a multi-class cross-subject scenario, then investigated the reason for transfer learning performance being poor in multi-class cross-subject classification. Secondly, we address the challenge of significant sessional and subject-to-subject variations originating from both known and unknown factors. It is discovered that such variations have a massive influence on the classification because of the negative transfer (NT) across domains. Based on this discovery, we propose a multi-class transfer learning approach based on multi-source manifold feature transfer learning (MMFT) framework and an enhanced version to minimize the effects of NT. The proposed multi-class transfer learning approach extends the existing MMFT to multi-class cases. Then enhanced multi-class MMFT firstly searches for domains with high transferability and selects only the best combination among source domains (SD), then utilize the best-selected combination of domains for transfer learning. Experimental results illustrate that the proposed multi-class MMFT can be employed in the cross-subject classification of both three-class and four-class problems. Experimental results also demonstrated that the enhanced multi-class MMFT could effectively minimize the effect of negative transfer and significantly increase the prediction rates across individual target domains (TD). The highest classification accuracy (CA) of 98% is obtained by the enhanced multi-class MMFT.

Visual Domain Adaptation through Locality Information

Traditional machine learning methods have always performed the learning tasks solely. The models of these conventional methods have to be built from scratch. The idea of domain adaptation can overcome this learning procedure by adapting the knowledge gained from one domain to another. With datasets from several domains, this research proposes a novel strategy for unsupervised visual domain adaptation. Existing transfer learning approaches try to reduce the domain shift using maximum mean discrepancy metric. We propose a combined framework termed Visual Domain Adaptation through Locality Information (DALI) that lowers the geometrical distance between the domains while probabilistically addressing the shortcomings. The proposed approach employs the input data features to develop a technique that assists in inducing the transfer of information from one domain to another. Extensive experimental findings demonstrate that the proposed method DALI excels over traditional domain adaptation algorithms as well as CNN architectures on a variety of cross-domain object, facial, and digit recognition tasks including Office-Caltech, Office-Home, PIE and COIL datasets. Our proposed model has improved the mean accuracy to 69.69%, exceeding the most recent state-of-the-art techniques when performed a comprehensive analysis on the Office-Home data.

Deep Attention Relation Network: A Zero-Shot Learning Method for Bearing Fault Diagnosis Under Unknown Domains

Deep learning (DL) method are extensively used for bearing fault diagnosis (BFD). Due to severe data distribution difference under variable working conditions, they have unsatisfactory performance of the BFD. Although the existing transfer learning (TL) methods might improve the diagnostic performance in different data distributions, fault data from these different domains in training have to be obtained. When a given bearing operates in a new working condition and fault data are not available, the TL methods might be invalid, and the BFD would be postponed. To solve the above problem, a novel zero-shot learning method named deep attention relation network (DARN) is proposed for the BFD under multiple unknown domains. The built DARN only trained by the data from a known domain might be used to diagnose fault types from unknown, but related domains without prior data input. In this method, a feature extraction module is constructed to generate representations of input samples, and a relation module is designed to calculate the relation score between the sample pairs to determine their categories. Meanwhile, a parallel attention mechanism is introduced into the DARN so as to enhance the representative ability of the built model. The results of experimental study indicate that the proposed method can make use of fault knowledge learnt from the single known domain for the BFD in the several unknown domains. The proposed DARN significantly outperforms the existing popular TL methods in diagnostic performance.

Cross-Domain Open Set Fault Diagnosis Based on Weighted Domain Adaptation with Double Classifiers.

The application of transfer learning in fault diagnosis has been developed in recent years. It can use existing data to solve the problem of fault recognition under different working conditions. Due to the complexity of the equipment and the openness of the working environment in industrial production, the status of the equipment is changeable, and the collected signals can have new fault classes. Therefore, the open set recognition ability of the transfer learning method is an urgent research direction. The existing transfer learning model can have a severe negative transfer problem when solving the open set problem, resulting in the aliasing of samples in the feature space and the inability to separate the unknown classes. To solve this problem, we propose a Weighted Domain Adaptation with Double Classifiers (WDADC) method. Specifically, WDADC designs the weighting module based on Jensen-Shannon divergence, which can evaluate the similarity between each sample in the target domain and each class in the source domain. Based on this similarity, a weighted loss is constructed to promote the positive transfer between shared classes in the two domains to realize the recognition of shared classes and the separation of unknown classes. In addition, the structure of double classifiers in WDADC can mitigate the overfitting of the model by maximizing the discrepancy, which helps extract the domain-invariant and class-separable features of the samples when the discrepancy between the two domains is large. The model's performance is verified in several fault datasets of rotating machinery. The results show that the method is effective in open set fault diagnosis and superior to the common domain adaptation methods.

Meta-transfer learning for emotion recognition

Deep learning has been widely adopted in automatic emotion recognition and has lead to significant progress in the field. However, due to insufficient training data, pre-trained models are limited in their generalisation ability, leading to poor performance on novel test sets. To mitigate this challenge, transfer learning performed by fine-tuning pr-etrained models on novel domains has been applied. However, the fine-tuned knowledge may overwrite and/or discard important knowledge learnt in pre-trained models. In this paper, we address this issue by proposing a PathNet-based meta-transfer learning method that is able to (i) transfer emotional knowledge learnt from one visual/audio emotion domain to another domain and (ii) transfer emotional knowledge learnt from multiple audio emotion domains to one another to improve overall emotion recognition accuracy. To show the robustness of our proposed method, extensive experiments on facial expression-based emotion recognition and speech emotion recognition are carried out on three bench-marking data sets: SAVEE, EMODB, and eNTERFACE. Experimental results show that our proposed method achieves superior performance compared with existing transfer learning methods.

Transfer Learning on Electromyography (EMG) Tasks: Approaches and Beyond.

Machine learning on electromyography (EMG) has recently achieved remarkable success on various tasks, while such success relies heavily on the assumption that the training and future data must be of the same data distribution. However, this assumption may not hold in many real-world applications. Model calibration is required via data re-collection and label annotation, which is generally very expensive and time-consuming. To address this issue, transfer learning (TL), which aims to improve target learners' performance by transferring knowledge from related source domains, is emerging as a new paradigm to reduce the amount of calibration effort. This survey assesses the eligibility of more than fifty published peer-reviewed representative transfer learning approaches for EMG applications. Unlike previous surveys on purely transfer learning or EMG-based machine learning, this survey aims to provide insight into the biological foundations of existing transfer learning methods on EMG-related analysis. Specifically, we first introduce the muscles' physiological structure, the EMG generating mechanism, and the recording of EMG to provide biological insights behind existing transfer learning approaches. Further, we categorize existing research endeavors into data based, model based, training scheme based, and adversarial based. This survey systematically summarizes and categorizes existing transfer learning approaches for EMG related machine learning applications. In addition, we discuss possible drawbacks of existing works and point out the future direction of better EMG transfer learning algorithms to enhance practicality for real-world applications.

Variable-Modulation Specific Emitter Identification With Domain Adaptation

Specific emitter identification (SEI) is a technique of identifying individual emitters via unique characteristics of different emitters. In this paper, we consider a SEI problem with transmitter changing modulations scenario. There have been few previous studies on this type of scenario. To cope with the daunting challenge, a variable-modulation SEI framework with domain adaptation is proposed. The components characteristics of transmitter are analyzed and the distortion models are established for simulation dataset generation. The received in-phase/quadrature (I/Q) signals are demodulated and reconstructed to obtain baseband ideal modulation signals. The received signals and the ideal modulation signals corresponding to demodulation and reconstruction are merged and embedded into the feature extraction network. Domain adversarial neural network (DANN) is added into the SEI framework to generate domain-invariant fingerprint features, thus realizing variable-modulation SEI. To better align the distortion features of emitters with variable modulations, Gaussian Encoder is designed to project fingerprint features into Gaussian distribution space. Numerous experiments show that the proposed SEI framework can improve recognition accuracy of individual emitter for single modulation and variable transfer greatly, and outperform the existing transfer learning methods. The ablation study demonstrates the components of framework are complementary. The complexity of framework is acceptable and it can extend to large-scale use. The robustness of framework is verified through modulation transfer among PSK and QAM.

Bi-modal email spam detection using recurrent and convolution neural network techniques

The increasing adoption of electronic emails as a means of communication, both at the commercial, government, and individual levels, serves as an impetus for attackers to compromise communication. Consequently, numerous machine learning techniques have been developed for identifying unwanted emails, commonly known as spam. Despite the significant progress reported in existing literature, most studies do not integrate the detection of both textual and image-based spam. In this paper, two deep learning techniques that detect both textual and image-based spam were evaluated. First, the Recurrent Neural Network (RNN) and Convolutional Neural Network (CNN) is studied, training them on various textimage features to explore their effectiveness on an improved dataset. Subsequently, in an effort to outsmart current spam detection techniques, a bi-modal architecture capable of detecting textual spam, image spam, and mixed spam is designed. The experimental results in conjunction with existing transfer learning for effective spam detection is provided.

Discriminative Fisher Embedding Dictionary Transfer Learning for Object Recognition.

In transfer learning model, the source domain samples and target domain samples usually share the same class labels but have different distributions. In general, the existing transfer learning algorithms ignore the interclass differences and intraclass similarities across domains. To address these problems, this article proposes a transfer learning algorithm based on discriminative Fisher embedding and adaptive maximum mean discrepancy (AMMD) constraints, called discriminative Fisher embedding dictionary transfer learning (DFEDTL). First, combining the label information of source domain and part of target domain, we construct the discriminative Fisher embedding model to preserve the interclass differences and intraclass similarities of training samples in transfer learning. Second, an AMMD model is constructed using atoms and profiles, which can adaptively minimize the distribution differences between source domain and target domain. The proposed method has three advantages: 1) using the Fisher criterion, we construct the discriminative Fisher embedding model between source domain samples and target domain samples, which encourages the samples from the same class to have similar coding coefficients; 2) instead of using the training samples to design the maximum mean discrepancy (MMD), we construct the AMMD model based on the relationship between the dictionary atoms and profiles; thus, the source domain samples can be adaptive to the target domain samples; and 3) the dictionary learning is based on the combination of source and target samples which can avoid the classification error caused by the difference among samples and reduce the tedious and expensive data annotation. A large number of experiments on five public image classification datasets show that the proposed method obtains better classification performance than some state-of-the-art dictionary and transfer learning methods. The code has been available at https://github.com/shilinrui/DFEDTL.