Transfer Learning Framework Research Articles

A residual convolution transfer framework based on slow feature for cross-domain machinery fault diagnosis

Intelligent fault diagnosis plays a vital role in ensuring the stable, reliable and safe operation of machinery equipment. However, data distribution doesn’t meet the assumption of the same distribution in the practical scene due to environmental changes. Traditional transfer learning methods can solve the situation of data distribution shift, but they encounter obstacles without adequately considering the possibility of introducing feature pre-extraction into deep learning and fusing pre-extracted low-dimensional feature with high-dimensional feature. In this study, a residual convolution transfer learning framework based on slow feature, which is an invariant or slowly varing signal learned from a vector signal, is presented to address this issue. Firstly, slow feature analysis is employed to implement preliminary feature extraction, which obtains latent variable that reflects the fundamental information of the equipment. Then, slow feature is sent to a residual convolution network for further abstraction and low-dimensional feature aggregates with high-dimensional feature by a bypass branch. Furthermore, maximum mean discrepancy is introduced to calculate the distance between two distributions in the Reproducing Kernel Hilbert Space (RKHS) and reduce the discrepancy of feature distribution. The proposed method is evaluated on Xi’an Jiao Tong University (XJTU) dataset and Case Western Reserve University (CWRU) dataset, and the average accuracy is higher than 99%. Experimental results show that the proposed framework has superior transferability and robustness under different working conditions.

Federated transfer learning for auxiliary classifier generative adversarial networks: framework and industrial application.

Machine learning with considering data privacy-preservation and personalized models has received attentions, especially in the manufacturing field. The data often exist in the form of isolated islands and cannot be shared because of data privacy in real industrial scenarios. It is difficult to gather the data to train a personalized model without compromising data privacy. To address this issue, we proposed a Federated Transfer Learning framework based on Auxiliary Classifier Generative Adversarial Networks named ACGAN-FTL. In the framework, Federated Learning (FL) trains a global model on decentralized datasets of the clients with data privacy-preservation and Transfer Learning (TL) transfers the knowledge from the global model to a personalized model with a relatively small data volume. ACGAN acts as a data bridge to connect FL and TL by generating similar probability distribution data of clients since the client datasets in FL cannot be directly used in TL for data privacy-preservation. A real industrial scenario of pre-baked carbon anode quality prediction is applied to verify the performance of the proposed framework. The results show that ACGAN-FTL can not only obtain acceptable performance on 0.81 accuracy, 0.86 precision, 0.74 recall, and 0.79 F1 but also ensure data privacy-preservation in the whole learning process. Compared to the baseline method without FL and TL, the former metrics have increased by 13%, 11%, 16%, and 15% respectively. The experiments verify that the performance of the proposed ACGAN-FTL framework fulfills the requirements of industrial scenarios.

Voice disorder classification using convolutional neural network based on deep transfer learning

Voice disorders are very common in the global population. Many researchers have conducted research on the identification and classification of voice disorders based on machine learning. As a data-driven algorithm, machine learning requires a large number of samples for training. However, due to the sensitivity and particularity of medical data, it is difficult to obtain sufficient samples for model learning. To address this challenge, this paper proposes a pretrained OpenL3-SVM transfer learning framework for the automatic recognition of multi-class voice disorders. The framework combines a pre-trained convolutional neural network, OpenL3, and a support vector machine (SVM) classifier. The Mel spectrum of the given voice signal is first extracted and then input into the OpenL3 network to obtain high-level feature embedding. Considering the effects of redundant and negative high-dimensional features, model overfitting easily occurs. Therefore, linear local tangent space alignment (LLTSA) is used for feature dimension reduction. Finally, the obtained dimensionality reduction features are used to train the SVM for voice disorder classification. Fivefold cross-validation is used to verify the classification performance of the OpenL3-SVM. The experimental results show that OpenL3-SVM can effectively classify voice disorders automatically, and its performance exceeds that of the existing methods. With continuous improvements in research, it is expected to be considered as auxiliary diagnostic tool for physicians in the future.

A CNN-Based Chest Infection Diagnostic Model: A Multistage Multiclass Isolated and Developed Transfer Learning Framework

In 2019, a deadly coronaviral infection (COVID-19) that infected millions of people globally was detected in China. This fatal virus affects the respiratory system and currently spreads to more than 200 nations worldwide. COVID-19 may be found using a chest X-ray scan, a reliable imaging method. Although an expert may examine an X-ray scan manually, this process takes a lot of time. Therefore, deep convolutional neural networks (CNNs) may be utilized to automate this procedure. In this work, at the first step, a novel isolated 19-layer CNN model is developed from scratch to detect chest infections using X-rays. Then, the developed model is reutilized to distinguish the type of chest infection, such as COVID-19, fibrosis, pneumonia, and tuberculosis, using the transfer learning approach. Stochastic gradient descent with momentum is utilized to optimize the model. The proposed multistage framework shows 98.85% and 97% classification accuracies for chest infection detection (binary classification between normal and patient) and four-class subclassification (COVID-19, fibrosis, pneumonia, and tuberculosis) for an online chest X-ray dataset. The reliability of the proposed multistage CNN model was further validated through a new dataset, showing an accuracy of 98.5%. The proposed multistage methodology took minimal training time compared to publically available pretrained models. Therefore, the presented multistage deep learning framework can help doctors in clinical practices.

Time Series Prediction Method of Industrial Process With Limited Data Based on Transfer Learning

Industrial time series, as a kind of data that responds to production process information, can be analyzed and predicted for effective monitoring of industrial production processes. There are problems of data shortage and algorithm cold start in industrial modeling process caused by complex working conditions, change of data acquisition environment and short running time of equipment. As a result, the accuracy of the existing data-driven industrial time series prediction algorithm is greatly limited. To address the above problems, we propose a new time series prediction method for industrial processes under limited data based on dynamic transfer learning in this work. This method aims to effectively use historical data of similar equipment or working conditions rather than discard them to help establish an industrial time series prediction model with limited target data. In this method, first, historical data are divided into multiple batches, and then a new multi-source transfer learning framework with dynamic maximum mean difference (MMD) loss is established according to the distribution distance between each batch of historical data and the limited target data at the current moment. The framework also combines multi-task learning methods to establish multi-step prediction model for online learning in industrial processes. Compared with other commonly used methods, experiments on two real-world datasets of solar power generation prediction and heating furnace temperature prediction demonstrate the effectiveness of the proposed method.

Knowledge transduction for cross-domain few-shot learning

Cross-Domain Few-Shot Learning (CDFSL) aims to classify new categories from new domains with few samples. It confronts a greater domain shift than Few-Shot Learning (FSL). Based on the transfer learning framework, we propose a Knowledge Transduction method (KT) to alleviate domain shift and achieve few-shot recognition. First, a feature adaptation module based on feed-forward attention is constructed to learn domain-adapted features. The feature adaptation module weakens domain shift by transducing knowledge from an auxiliary dataset to the new dataset. Second, a feature transduction module based on deep sparse representation is developed to gather class semantics from limited support images. The feature transduction module transduces knowledge from support images to query images for few-shot recognition. In addition, a stochastic image augmentation method is proposed for FSL to train a more generalized model through consistency representation learning. Our method achieves competitive accuracy on four CDFSL datasets and four FSL datasets compared to state-of-the-art methods. The source code is available at https://github.com/XDUpfLi/KT.

Multi-Class Transfer Learning and Domain Selection for Cross-Subject EEG Classification

Transfer learning (TL) has been proven to be one of the most significant techniques for cross-subject classification in electroencephalogram (EEG)-based brain-computer interfaces (BCI). Hence, it is widely used to address the challenges of cross-session and cross-subject variability with more accurate intention prediction. In this case, TL utilizes knowledge (signal features) in the source domain(s) to improve the classification in the target domain. However, current existing transfer learning approaches on EEG-based BCI are mostly limited to two-class cross-subject classification problems, while multi-class problems are only implemented with a focus on within-subject classification due to the complexity of multi-class cross-subject classification problems. In this paper, we first extended the transfer learning approaches to a multi-class cross-subject scenario, then investigated the reason for transfer learning performance being poor in multi-class cross-subject classification. Secondly, we address the challenge of significant sessional and subject-to-subject variations originating from both known and unknown factors. It is discovered that such variations have a massive influence on the classification because of the negative transfer (NT) across domains. Based on this discovery, we propose a multi-class transfer learning approach based on multi-source manifold feature transfer learning (MMFT) framework and an enhanced version to minimize the effects of NT. The proposed multi-class transfer learning approach extends the existing MMFT to multi-class cases. Then enhanced multi-class MMFT firstly searches for domains with high transferability and selects only the best combination among source domains (SD), then utilize the best-selected combination of domains for transfer learning. Experimental results illustrate that the proposed multi-class MMFT can be employed in the cross-subject classification of both three-class and four-class problems. Experimental results also demonstrated that the enhanced multi-class MMFT could effectively minimize the effect of negative transfer and significantly increase the prediction rates across individual target domains (TD). The highest classification accuracy (CA) of 98% is obtained by the enhanced multi-class MMFT.

Gear contact fatigue life prediction based on transfer learning

The gear contact fatigue test is characterised by extraordinarily high costs and a long period of testing. A transfer learning algorithm was applied to develop a gear contact fatigue life prediction strategy to fully use other low-cost fatigue test data, such as twin-disc test data. The twin-disc contact fatigue test data were incorporated into the neural network model to explore the combined effects of surface integrity and stress level on fatigue life, which was further transferred for gear contact fatigue life prediction. The results indicate that this method enables an effective gear fatigue life prediction under minor gear sample conditions. With a small sample amount of 7, the prediction error in the fatigue life of the model with transfer learning is 46.33%, while the typical back-propagation neural network (BPNN) model without transfer learning is 101.66%. To control life prediction error to within 50%, the model without transfer learning requires at least 17 gear samples, while the transfer model calls for only 7 gear samples. The proposed data transfer learning framework can be used to predict the gear contact fatigue S-N curve under minor sample conditions and is straightforwardly applied to extended applications.

Cross-subject EEG emotion recognition using multi-source domain manifold feature selection

Recent researches on emotion recognition suggests that domain adaptation, a form of transfer learning, has the capability to solve the cross-subject problem in Affective brain-computer interface (aBCI) field. However, traditional domain adaptation methods perform single to single domain transfer or simply merge different source domains into a larger domain to realize the transfer of knowledge, resulting in negative transfer. In this study, a multi-source transfer learning framework was proposed to promote the performance of multi-source electroencephalogram (EEG) emotion recognition. The method first used the data distribution similarity ranking (DDSA) method to select the appropriate source domain for each target domain off-line, and reduced data drift between domains through manifold feature mapping on Grassmann manifold. Meanwhile, the minimum redundancy maximum correlation algorithm (mRMR) was employed to select more representative manifold features and minimized the conditional distribution and marginal distribution of the manifold features, and then learned the domain-invariant classifier by summarizing structural risk minimization (SRM). Finally, the weighted fusion criterion was applied to further improve recognition performance. We compared our method with several state-of-the-art domain adaptation techniques using the SEED and DEAP dataset. Results showed that, compared with the conventional MEDA algorithm, the recognition accuracy of our proposed algorithm on SEED and DEAP dataset were improved by 6.74% and 5.34%, respectively. Besides, compared with TCA, JDA, and other state-of-the-art algorithms, the performance of our proposed method was also improved with the best average accuracy of 86.59% on SEED and 64.40% on DEAP. Our results demonstrated that the proposed multi-source transfer learning framework is more effective and feasible than other state-of-the-art methods in recognizing different emotions by solving the cross-subject problem.

A novel transfer learning framework for sorghum biomass prediction using UAV-based remote sensing data and genetic markers.

Yield for biofuel crops is measured in terms of biomass, so measurements throughout the growing season are crucial in breeding programs, yet traditionally time- and labor-consuming since they involve destructive sampling. Modern remote sensing platforms, such as unmanned aerial vehicles (UAVs), can carry multiple sensors and collect numerous phenotypic traits with efficient, non-invasive field surveys. However, modeling the complex relationships between the observed phenotypic traits and biomass remains a challenging task, as the ground reference data are very limited for each genotype in the breeding experiment. In this study, a Long Short-Term Memory (LSTM) based Recurrent Neural Network (RNN) model is proposed for sorghum biomass prediction. The architecture is designed to exploit the time series remote sensing and weather data, as well as static genotypic information. As a large number of features have been derived from the remote sensing data, feature importance analysis is conducted to identify and remove redundant features. A strategy to extract representative information from high-dimensional genetic markers is proposed. To enhance generalization and minimize the need for ground reference data, transfer learning strategies are proposed for selecting the most informative training samples from the target domain. Consequently, a pre-trained model can be refined with limited training samples. Field experiments were conducted over a sorghum breeding trial planted in multiple years with more than 600 testcross hybrids. The results show that the proposed LSTM-based RNN model can achieve high accuracies for single year prediction. Further, with the proposed transfer learning strategies, a pre-trained model can be refined with limited training samples from the target domain and predict biomass with an accuracy comparable to that from a trained-from-scratch model for both multiple experiments within a given year and across multiple years.

Urban network-wide traffic volume estimation under sparse deployment of detectors

ABSTRACT Sensing network-wide traffic information is fundamental for the sustainable development of urban planning and traffic management. However, owing to the limited budgets or device maintenance costs, detector deployment is usually sparse. Obtaining full-scale network volume using detectors is neither effective nor practical. Existing works primarily focus on improving the estimation accuracy using multi-correlation of networks and ignore the underlying challenges, particularly for these entire undetected road segments in sparse detector deployment scenarios. Here our study proposes a tailored transfer learning framework called the transfer learning-based least square support vector regression (TL-LSSVR) model. Network-wide volume can be estimated by fusing active detectors (taxi GPS data) and fixed passive detectors (license plate recognition data). Numerical experiments are carried out on a real-world road network in Nanjing, China. It is demonstrated that our approach achieves high performance even under sparse deployment of detectors and outperforms other baselines significantly.

A two-stage renal disease classification based on transfer learning with hyperparameters optimization.

Renal diseases are common health problems that affect millions of people around the world. Among these diseases, kidney stones, which affect anywhere from 1 to 15% of the global population and thus; considered one of the leading causes of chronic kidney diseases (CKD). In addition to kidney stones, renal cancer is the tenth most prevalent type of cancer, accounting for 2.5% of all cancers. Artificial intelligence (AI) in medical systems can assist radiologists and other healthcare professionals in diagnosing different renal diseases (RD) with high reliability. This study proposes an AI-based transfer learning framework to detect RD at an early stage. The framework presented on CT scans and images from microscopic histopathological examinations will help automatically and accurately classify patients with RD using convolutional neural network (CNN), pre-trained models, and an optimization algorithm on images. This study used the pre-trained CNN models VGG16, VGG19, Xception, DenseNet201, MobileNet, MobileNetV2, MobileNetV3Large, and NASNetMobile. In addition, the Sparrow search algorithm (SpaSA) is used to enhance the pre-trained model's performance using the best configuration. Two datasets were used, the first dataset are four classes: cyst, normal, stone, and tumor. In case of the latter, there are five categories within the second dataset that relate to the severity of the tumor: Grade 0, Grade 1, Grade 2, Grade 3, and Grade 4. DenseNet201 and MobileNet pre-trained models are the best for the four-classes dataset compared to others. Besides, the SGD Nesterov parameters optimizer is recommended by three models, while two models only recommend AdaGrad and AdaMax. Among the pre-trained models for the five-class dataset, DenseNet201 and Xception are the best. Experimental results prove the superiority of the proposed framework over other state-of-the-art classification models. The proposed framework records an accuracy of 99.98% (four classes) and 100% (five classes).

DDPG with Transfer Learning and Meta Learning Framework for Resource Allocation in Underlay Cognitive Radio Network

DDPG with Transfer Learning and Meta Learning Framework for Resource Allocation in Underlay Cognitive Radio Network

Modeling and predictive control of nonlinear processes using transfer learning method

AbstractThis work develops a transfer learning (TL) framework for modeling and predictive control of nonlinear systems using recurrent neural networks (RNNs) with the knowledge obtained in modeling one process transferred to another. Specifically, transfer learning uses a pretrained model developed based on a source domain as the starting point, and adapts the model to a target process with similar configurations. The generalization error for TL‐based RNN (TL‐RNN) is first derived to demonstrate the generalization capability on the target process. The theoretical error bound that depends on model capacity and the discrepancy between source and target domains is then utilized to guide the development of pretrained models for improved model transferability. Subsequently, the TL‐RNN model is utilized as the prediction model in model predictive controller (MPC) for the target process. Finally, the simulation study of chemical reactors via Aspen Plus Dynamics is used to demonstrate the benefits of transfer learning.

Transferability investigation of a Sim2Real deep transfer learning framework for cross-building energy prediction

Data-driven methods generally require large-scale historical data volume to obtain a better building energy prediction performance. Transfer learning methods have been widely used to address this issue by utilizing knowledge learned from similar tasks. However, most studies on building energy prediction are mainly focused on the effect of a single transfer learning model. There are few studies to systematically investigate the effect of building types, climate zones and data volume on energy prediction performance by using different transfer learning models with the auxiliary simulation data. This study proposes a Sim2Real transfer learning framework using simulation datasets for building energy prediction to fill this research gap. Three different experiment scenarios are conducted to investigate the effect of different building types, climate zones and data volume on the prediction performance from the perspective of simulation to reality by using the proposed Sim2Real transfer learning framework. Extensive experiment cases demonstrate that the proposed Sim2Real transfer learning framework can effectively enhance the MAPE by 1.06%∼29.05% with proper transfer learning models compared with the baseline model LSTM. In addition, transfer learning strategies are not necessary when the training data volume of target building is over five weeks. The proposed Sim2Real transfer learning framework can provide the bridge to enhance the building energy prediction performance from the perspective of simulation to reality.

Integrating physics-informed recurrent Gaussian process regression into instance transfer for predicting tool wear in milling process

Effective management of tool condition is of key importance to produce precision parts with desirable structural shape and excellent surface integrity. Due to the variable cutting conditions and the limited tool wear data in practice, it is a great challenge to predict tool wear with traditional machine learning methods. To overcome the dual challenges, this study proposes a physics-informed transfer learning (PITL) framework to predict tool wear under variable working conditions. Specifically, in order to predict tool wear under new tasks with limited data, instance transfer learning algorithm Two-stage TrAdaBoost.R2 is used to improve the generalization ability of the prediction model by transferring useful knowledge from source domains. Then, to compensate for the shortcomings of applying pure data-driven model as base learner in Two-stage TrAdaBoost.R2, physics-informed recurrent Gaussian process regression (PRGPR) is utilized as base learner to improve the extrapolation performance of the model in the target domain. For PRGPR, on the one hand, the tool wear predicted in the previous step is incorporated into the input vector to address the time-accumulation effect. On the other hand, the mean function of the model is generated by deriving equations from prior degradation knowledge to guide the forecasting process. Finally, the experimental results indicate that the proposed method has favorable foresight of the degradation process and can further improve the prediction accuracy of tool wear under variable working conditions.

Application of the deep transfer learning framework for hydatid cyst classification using CT images

Application of the deep transfer learning framework for hydatid cyst classification using CT images

A Systems Theory of Transfer Learning

Existing frameworks for transfer learning are incomplete from a systems theoretic perspective. They place emphasis on notions of domain and task, and neglect notions of structure and behavior. In doing so, they limit the extent to which formalism can be carried through into the elaboration of their frameworks. Herein, we use the Mesarovician systems theory to define transfer learning as a relation on sets, and subsequently, characterize the general nature of transfer learning as a mathematical construct. We interpret existing frameworks in terms of ours and go beyond existing frameworks to define notions of transferability, transfer roughness, and transfer distance. Importantly, despite its formalism, our framework avoids the detailed mathematics of the learning theory or machine learning solution methods without excluding their consideration. As such, we provide a formal, general systems framework for modeling transfer learning that offers a rigorous foundation for system design and analysis.

Federated Transfer Learning for Low-dose PET Denoising: A Pilot Study with Simulated Heterogeneous Data.

Positron emission tomography (PET) with a reduced injection dose, i.e., low-dose PET, is an efficient way to reduce radiation dose. However, low-dose PET reconstruction suffers from a low signal-to-noise ratio (SNR), affecting diagnosis and other PET-related applications. Recently, deep learning-based PET denoising methods have demonstrated superior performance in generating high-quality reconstruction. However, these methods require a large amount of representative data for training, which can be difficult to collect and share due to medical data privacy regulations. Moreover, low-dose PET data at different institutions may use different low-dose protocols, leading to non-identical data distribution. While previous federated learning (FL) algorithms enable multi-institution collaborative training without the need of aggregating local data, it is challenging for previous methods to address the large domain shift caused by different low-dose PET settings, and the application of FL to PET is still under-explored. In this work, we propose a federated transfer learning (FTL) framework for low-dose PET denoising using heterogeneous low-dose data. Our experimental results on simulated multi-institutional data demonstrate that our method can efficiently utilize heterogeneous low-dose data without compromising data privacy for achieving superior low-dose PET denoising performance for different institutions with different low-dose settings, as compared to previous FL methods.

Identifying Humans Based on Biometric Iris Recognition using an Interactive Transfer Learning Framework

Identifying Humans Based on Biometric Iris Recognition using an Interactive Transfer Learning Framework