Multi-source Transfer Learning Research Articles

Disk anomaly detection for remote maintenance control system of natural gas pipeline based on multi-source domain transfer and incremental learning

Accurate disk anomaly detection is of great significance for maintaining the safe and stable operation of the remote control system of natural gas pipelines. Most disk transfer algorithms use only a single-source domain model for offline transfer learning, resulting in lower classifier performance. This paper proposes a disk online anomaly detection algorithm combining transfer learning and incremental learning. A transfer learning algorithm based on data distribution similarity and dynamic index weighted ensemble is proposed. According to multiple source domain models and target domain model negative data in hyperspace, k neighbour data of each target domain negative data is extracted, and then different test criteria are adopted for each sub-model according to the number of target domain positive data, and index reweighting is performed according to the model detection capability. An incremental learning algorithm based on online automatic annotation is proposed. The data popped up by the sliding window is automatically marked at the current disk state, and the update period of the prediction model is dynamically adjusted according to the false detection rate and the missed detection rate. By setting three different experiments for comparison, the effectiveness and advanced nature of the method proposed in this paper are illustrated.

Classification of tumor from computed tomography images: A brain-inspired multisource transfer learning under probability distribution adaptation.

Preoperative diagnosis of gastric cancer and primary gastric lymphoma is challenging and has important clinical significance. Inspired by the inductive reasoning learning of the human brain, transfer learning can improve diagnosis performance of target task by utilizing the knowledge learned from the other domains (source domain). However, most studies focus on single-source transfer learning and may lead to model performance degradation when a large domain shift exists between the single-source domain and target domain. By simulating the multi-modal information learning and transfer mechanism of human brain, this study designed a multisource transfer learning feature extraction and classification framework, which can enhance the prediction performance of the target model by using multisource medical data (domain). First, this manuscript designs a feature extraction network that takes the maximum mean difference based on the Wasserstein distance as an adaptive measure of probability distribution and extracts the domain-specific invariant representations between source and target domain data. Then, aiming at the random generation of parameters bringing uncertainties to prediction accuracy and generalization ability of extreme learning machine network, the 1-norm regularization is used to implement sparse constraints of the output weight matrix and improve the robustness of the model. Finally, some experiments are carried out on the data of two medical centers. The experimental results show that the area under curves (AUCs) of the method are 0.958 and 0.929 in the two validation cohorts, respectively. The method in this manuscript can provide doctors with a better diagnostic reference, which has certain practical significance.

Similarity-based Multi-source Transfer Learning Approach for Time Series Classification

This study aims to develop an effective method of classification concerning time series signals for machine state prediction to advance predictive maintenance (PdM). Conventional machine learning (ML) algorithms are widely adopted in PdM, however, most existing methods assume that the training (source) and testing (target) data follow the same distribution, and that labeled data are available in both source and target domains. For real-world PdM applications, the heterogeneity in machine original equipment manufacturers (OEMs), operating conditions, facility environment, and maintenance records collectively lead to heterogeneous distribution for data collected from different machines. This will significantly limit the performance of conventional ML algorithms in PdM. Moreover, labeling data is generally costly and time-consuming. Finally, industrial processes incorporate complex conditions, and unpredictable breakdown modes lead to extreme complexities for PdM. In this study, similarity-based multi-source transfer learning (SiMuS-TL) approach is proposed for real-time classification of time series signals. A new domain, called "mixed domain," is established to model the hidden similarities among the multiple sources and the target. The proposed SiMuS-TL model mainly includes three key steps: 1) learning group-based feature patterns, 2) developing group-based pre-trained models, and 3) weight transferring. The proposed SiMuS-TL model is validated by observing the state of the rotating machinery using a dataset collected on the Skill boss manufacturing system, publicly available standard bearing datasets, Case Western Reserve University (CWRU), and Paderborn University (PU) bearing datasets. The results of the performance comparison demonstrate that the proposed SiMuS-TL method outperformed conventional Support Vector Machine (SVM), Artificial Neural Network (ANN), and Transfer learning with neural networks (TLNN) without similarity-based transfer learning methods.

Multi-source manifold feature transfer learning with domain selection for brain-computer interfaces

Transfer learning uses the knowledge in source domains to improve the learning performance in the target domain, which is useful in electroencephalogram (EEG) based brain-computer interfaces (BCIs) with small training datasets. However, the existing transfer learning methods for EEG based BCI mainly consider the knowledge transfer from single-to-single (STS) domain or simply merge different source domains into a bigger one. In this paper, we propose a multi-source manifold feature transfer learning (MMFT) framework to transfer multi-source knowledge for EEG signals classification. MMFT minimizes marginal probability distribution on the Riemannian manifold using Riemannian alignment and Grassmann manifold feature learning, then transfers the manifold features with a conditional probability distribution adaptation in the structural risk minimization (SRM) function. Based on MMFT, w-MMFT is proposed to tackle the class imbalance issue for SRM, and the label similarity analysis (LSA) is proposed to select source domains for MMFT, forming a new LSA-MMFT framework. Experimental results on six datasets demonstrate that the proposed MMFT has achieved superior performance in classification accuracy and computational efficiency compared to state-of-the-art methods. The LSA-MMFT can get more stable performance than two other domain selection methods.

A three-stage transfer learning framework for multi-source cross-project software defect prediction

Transfer learning techniques have been proved to be effective in the field of Cross-project defect prediction (CPDP). However, some questions still remain. First, the conditional distribution difference between source and target projects has not been considered. Second, facing multiple source projects, most studies only rarely consider the issues of source selection and multi-source data utilization; instead, they use all available projects and merge multi-source data together to obtain one final dataset. To address these issues, in this paper, we propose a three-stage weighting framework for multi-source transfer learning (3SW-MSTL) in CPDP. In stage 1, a source selection strategy is needed to select a suitable number of source projects from all available projects. In stage 2, a transfer technique is applied to minimize marginal differences. In stage 3, a multi-source data utilization scheme that uses conditional distribution information is needed to help guide researchers in the use of multi-source transferred data. First, we have designed five source selection strategies and four multi-source utilization schemes and chosen the best one to be used in stage 1 and 3 in 3SW-MSTL by comparing their influences on prediction performance. Second, to validate the performance of 3SW-MSTL, we compared it with four multi-source and six single-source CPDP methods, a baseline within-project defect prediction (WPDP) method, and two unsupervised methods on the data from 30 widely used open-source projects. Through experiments, bellwether and weighted vote are separately chosen as a source selection strategy and a multi-source utilization scheme used in 3SW-MSTL. And, our results indicate that 3SW-MSTL outperforms four multi-source, six single-source CPDP methods and two unsupervised methods. And, 3SW-MSTL is comparable to the WPDP method. The proposed 3SW-MSTL model is more effective for considering the two issues mentioned before.

A fault diagnosis method of bearings based on deep transfer learning

In recent years, many deep transfer learning methods have been widely used in bearing fault diagnosis under varying working conditions to solve the problem of data distribution shift. However, the deep transfer learning methods used in bearing fault diagnosis have low accuracy when source domain data differ much and feature distribution misaligns, so a deep multi-source transfer learning model is proposed in this paper. To solve the problem of the large discrepancy in source domain data, we use maximum mean discrepancy (MMD) to select suitable source domain data to form a new source domain to help model training, we utilize independent domain-specific feature extractors to extract domain features to avoid poor source domains affecting feature extraction of other domains. At the same time, classifier output is aligned using Wasserstein distance to reduce the probability of misclassification of boundary samples, and the weighted decision is used to impose more weight on the better source domains. In addition, aiming at the problem of data distribution misalignment, we propose an alignment method combining MMD, local maximum mean discrepancy (LMMD), and triplet loss. Experimental results show that the proposed model can achieve more than the accuracy of 95% in the four working conditions of the Paderborn bearing fault dataset, and is suitable for bearing fault diagnosis under varying working conditions.

The interaction of normalisation and clustering in sub-domain definition for multi-source transfer learning based time series anomaly detection

This paper examines how data normalisation and clustering interact in the definition of sub-domains within multi-source transfer learning systems for time series anomaly detection. The paper introduces a distinction between (i) clustering as a primary/direct method for anomaly detection, and (ii) clustering as a method for identifying sub-domains within the source or target datasets. Reporting the results of three sets of experiments, we find that normalisation after feature extraction and before clustering results in the best performance for anomaly detection. Interestingly, we find that in the multi-source transfer learning scenario clustering on the target dataset and identifying subdomains in the target data can result in improved model performance, as compared to identifying sub-domains through defining clusters using the multi-source dataset.

Fault Identification of Electric Submersible Pumps Based on Unsupervised and Multi-Source Transfer Learning Integration

The ratio between normal data and fault data generated by electric submersible pumps (ESPs) in production is prone to imbalance, and the information carried by the fault data generally as a minority sample is easily overwritten by the normal data as a majority sample, which seriously interferes with the fault identification effect. For the problem that data imbalance under different working conditions of ESPs causes the failure data to not be effectively identified, a fault identification method of ESPs based on unsupervised feature extraction integrated with migration learning was proposed. Firstly, new features were extracted from the data using multiple unsupervised methods to enhance the representational power of the data. Secondly, multiple samples of the source domain were obtained by multiple random sampling of the training set to fully train minority samples. Thirdly, the variation between the source domain and target domain was reduced by combining weighted balanced distribution adaptation (W-BDA). Finally, several basic learners were constructed and combined to integrate a stronger classifier to accomplish the ESP fault identification tasks. Compared with other fault identification methods, our method not only effectively enhances the performance of fault data features and improves the identification of a few fault data, but also copes with fault identification under different working conditions.

A novel approach for COVID-19 Infection forecasting based on multi-source deep transfer learning

COVID-19 is a contagious disease; so, predicting its future infections in a provincial region requires the consideration of the related data (i.e., rates of infection, mortality and recovery, etc.) over a period of time. Clearly, the COVID-19 data of a particular provincial region can be easily modelled as a time-series. However, predicting the future COVID-19 infections in a particular region is quite challenging when the availability of COVID-19 dataset of the province is of little quantity. Accordingly, ML models when deployed for such tasks usually results in low infection prediction accuracy. To overcome such issues of low variance and high bias in a model due to data scarcity, multi-source transfer learning (MSTL) along with deep learning may be quite useful and effective. Therefore, this paper proposes a novel technique based on multi-source deep transfer learning (MSDTL) to efficiently forecast the future COVID-19 infections in the provinces with insufficient COVID-19 data. The proposed approach is a novel contribution as it considers the fact that future COVID-19 transmission in a region also depends on its population density and economic conditions (GDP) for accurate forecasting of the infections to tackle the pandemic efficiently. The importance of this feature selection is experimentally proved in this paper. Our proposed approach employs the well-known recurrent neural network architecture, the Long-short term memory (LSTM), a popular deep-learning model for history-dependent tasks. A comparative analysis has been performed with existing state-of-art algorithms to portray the efficiency of LSTM. Thus, formation of MSDTL approach enhances the predictive precision capability of the LSTM. We evaluate the proposed methodology over the COVID-19 dataset from sixty-two provinces belonging to different nations. We then empirically evaluate the performance of the proposed approach using two different evaluation metrics, viz. The mean absolute percentage error and the coefficient of determination. We show that our proposed MSDTL based approach is better in terms of the accuracy of the future infection prediction, and produces improvements up to 96% over its without-TL counterpart.

Motor Imagery Recognition Method Based on Multisource Transfer Learning and Multiclassifier Fusion

There are two common problems in the field of motor imagery (MI) recognition, which are poor generalization and low recognition performance. A recognition method based on multisource transfer learning and multiclassifier fusion is therefore proposed to realize the MI classification. In this approach, multisource transfer learning method is used to transfer samples from multiple source domains to target domain. The source domain selection method based on distribution similarity is designed to select those source domains whose distribution is similar to the target domain, and samples with high information entropy are selected from these source domains for transferring. Then, an MI classification method is proposed through the fusion of multiple classifiers. The classifiers are trained by labeled samples in the target domain and the transferred samples in multiple source domains. The new sample in the target domain can be identified by the weight fusion of the results of these classifiers. In order to verify the effectiveness of the proposed method, four types of motor imagery in the BCI Competition IV dataset 2a were used to evaluate the recognition ability, and the results approved an excellent recognition and generalization performance as well as a better training efficiency comparing to the well-applied methods nowadays.

Multi-Source Selection Transfer Learning with Privacy-Preserving

Transfer learning has ability to create learning task of weakly labeled or unlabeled target domain by using knowledge of source domain to help, which can effectively improve the performance of target learning task. At present, the increased awareness of privacy protection restricts access to data sources and poses new challenges to the development of transfer learning. However, the research on privacy protection in transfer learning is very rare. The existing work mainly uses differential privacy technology and does not consider the distribution difference between data sources, or does not consider the conditional probability distribution of data, which causes negative transfer to harm the effect of algorithm. Therefore, this paper proposes multi-source selection transfer learning algorithm with privacy-preserving MultiSTLP, which is used in scenarios where target domain contains unlabeled data sets with only a small amount of group probability information and multiple source domains with a large number of labeled data sets. Group probability means that the class label of each sample in target data set is unknown, but the probability of each class in a given data group is available, and multiple source domains indicate that there are more than two source domains. The number of data set contains more than two data sets of source domain and one data set of target domain. The algorithm adapts to the marginal probability distribution and conditional probability distribution differences between domains, and can protect the privacy of target data and improve classification accuracy by fusing the idea of multi-source transfer learning and group probability into support vector machine. At the same time, it can select the representative dataset in source domains to improve efficiency relied on speeding up the training process of algorithm. Experimental results on several real datasets show the effectiveness of MultiSTLP, and it also has some advantages compared with the state-of-the-art transfer learning algorithm.

Multi-source transfer learning guided ensemble LSTM for building multi-load forecasting

Generally, it is difficult to establish an accurate building load forecasting model by using insufficient energy data. Although the transfer of knowledge from similar buildings can effectively solve this problem, there is still a lack of effective methods for both the selection of source domain buildings and the use of transfer knowledge when many candidate buildings are available. In view of this, this paper proposes a multi-source transfer learning guided ensemble LSTM method for building multi-load forecasting (MTE-LSTM). Firstly, a two-stage source-domain building matching method based on dominance comparison is developed to find multiple source-domain buildings similar to the target building. Next, an LSTM modeling strategy combining transfer learning and fine-tune technology is proposed, which uses multiple source-domain data to generate multiple basic load forecasting models for the target building. Following that, a model ensemble strategy based on similarity degree is given to weight the output results of basic forecasting models. Applications in many real buildings shows that the proposed building multi-energy load forecasting method can obtain high-precision load forecasting results when the target building data is relatively few.

Multisource Deep Transfer Learning Based on Balanced Distribution Adaptation.

The current traditional unsupervised transfer learning assumes that the sample is collected from a single domain. From the aspect of practical application, the sample from a single-source domain is often not enough. In most cases, we usually collect labeled data from multiple domains. In recent years, multisource unsupervised transfer learning with deep learning has focused on aligning in the common feature space and then seeking to minimize the distribution difference between the source and target domains, such as marginal distribution, conditional distribution, or both. Moreover, conditional distribution and marginal distribution are often treated equally, which will lead to poor performance in practical applications. The existing algorithms that consider balanced distribution are often based on a single-source domain. To solve the above-mentioned problems, we propose a multisource transfer learning algorithm based on distribution adaptation. This algorithm considers adjusting the weights of two distributions to solve the problem of distribution adaptation in multisource transfer learning. A large number of experiments have shown that our method MTLBDA has achieved significant results in popular image classification datasets such as Office-31.

Multisource Mobile Transfer Learning Algorithm Based on Dynamic Model Compression

With the development of the Internet of Things, the application of computer vision on mobile phones is becoming more and more extensive and people have higher and higher requirements for the timeliness of the recognition results returned and the processing capabilities of the mobile phone for image recognition. However, the processing capability and storage capability of the user terminal equipment cannot meet the needs of identifying and storing a large number of pictures, and the data transmission process will cause high energy consumption of the terminal equipment. At the same time, multisource deep transfer learning has outstanding performance in computer vision and image classification. However, due to the huge amount of calculation of the deep network model, it is impossible to use the existing excellent network model to realize image recognition and classification on the mobile terminal. In order to solve the abovementioned problems, we propose a multisource mobile transfer learning algorithm based on dynamic model compression, this algorithm considers the realization of multisource transfer learning computing in the case of multiple mobile device computing source domains, and the method also guarantees data privacy and security for each device (origin domain). Meanwhile, extensive experiments show that our method can achieve remarkable results in popular image classification datasets.

A multi-source information transfer learning method with subdomain adaptation for cross-domain fault diagnosis

In modern industrial equipment maintenance, transfer learning is a promising tool that has been widely utilized to solve the problem of the insufficient generalization ability of diagnostic models, caused by changes in working conditions. However, owing to the single knowledge transfer source and fuzzy marginal distribution matching, the ability of traditional transfer learning methods for cross-domain fault diagnosis is not ideal. In practice, collecting multi-source data from different scenarios can provide richer generalization knowledge, and fine-grained information matching of relevant subdomains can achieve more accurate knowledge transfer, which is conducive to the improvement of the cross-domain fault diagnosis performance. To this end, a multi-source subdomain adaptation transfer learning method is proposed to transfer diagnostic knowledge from multiple sources for cross-domain fault diagnosis. This approach exploits a multi-branch network structure to match the feature spatial distributions of each source and target domain separately, where the local maximum mean discrepancy is used for fine-grained local alignment of subdomain distributions within the same category of different domains. Moreover, the weighted score of a source-specific is obtained according to its distribution distance, and multiple source classifiers are combined with the corresponding weighted scores for the joint diagnosis of the device status. Extensive experiments are conducted on three rotating machinery datasets to verify the effectiveness of the proposed model for cross-domain fault diagnosis.

Terrain traversability prediction for off-road vehicles based on multi-source transfer learning

In this paper, a novel terrain traversability prediction method is proposed for new operation environments. When an off-road vehicle is operated on rough terrains or slopes made up of unconsolidated materials, it is crucial to accurately predict terrain traversability to ensure efficient operations and avoid critical mobility risks. However, the prediction of traversability in new environments is challenging, especially for possibly risky terrains, because the traverse data available for such terrains is either limited or non-existent. To address this limitation, this study proposes an adaptive terrain traversability prediction method based on multi-source transfer Gaussian process regression. The proposed method utilizes the limited data available on low-risk terrains of the target environment to enhance the prediction accuracy on untraversed, possibly higher-risk terrains by leveraging past traverse experiences on multiple types of terrain surface. The effectiveness of the proposed method is demonstrated in scenarios where vehicle slippage and power consumption are predicted using a dataset of various terrain surfaces and geometries. In addition to predicting terrain traversability as continuous values, the utility of the proposed method is demonstrated in binary risk level classification of yet to be traversed steep terrains from limited data on safer terrains.

Motor imagery EEG decoding using manifold embedded transfer learning

BackgroundBrain computer interface (BCI) utilizes brain signals to help users interact with external devices directly. EEG is one of the most commonly used techniques for brain signal acquisition in BCI. However, it is notoriously difficult to build a generic EEG recognition model due to significant non-stationarity and subject-to-subject variations, and the requirement for long time training. Transfer learning (TL) is particularly useful because it can alleviate the calibration requirement in EEG-based BCI applications by transferring the calibration information from existing subjects to new subject. To take advantage of geometric properties in Riemann manifold and joint distribution adaptation, a manifold embedded transfer learning (METL) framework was proposed for motor imagery (MI) EEG decoding. New methodFirst, the covariance matrices of the EEG trials are first aligned on the SPD manifold. Then the features are extracted from both the symmetric positive definite (SPD) manifold and Grassmann manifold. Finally, the classification model is learned by combining the structural risk minimization (SRM) of source domain and joint distribution alignment of source and target domains. ResultExperimental results on two MI EEG datasets verify the effectiveness of the proposed METL. In particular, when there are a small amount of labeled samples in the target domain, METL demonstrated a more accurate and stable classification performance than conventional methods. Comparison with existing methodsCompared with several state-of-the-art methods, METL has achieved better classification accuracy, 71.81% and 69.06% in single-to-single (STS), 83.14% and 76.00% in multi-to-single (MTS) transfer tasks, respectively. ConclusionsMETL can cope with single source domain or multi-source domains and compared with single-source transfer learning, multi-source transfer learning can improve the performance effectively due to the data expansion. It is effective enough to achieve superior performance for classification of EEG signals.

Learning for amalgamation: A multi-source transfer learning framework for sentiment classification

Transfer learning plays an essential role in Deep Learning, which can remarkably improve the performance of the target domain, whose training data is not sufficient. Our work explores beyond the common practice of transfer learning with a single pre-trained model. We focus on the task of Vietnamese sentiment classification and propose LIFA, a framework to learn a unified embedding from several pre-trained models. We further propose two more LIFA variants that encourage the pre-trained models to either cooperate or compete with one another. Studying these variants sheds light on the success of LIFA by showing that sharing knowledge among the models is more beneficial for transfer learning. Moreover, we construct the AISIA-VN-Review-F dataset, the first large-scale Vietnamese sentiment classification database. We conduct extensive experiments on the AISIA-VN-Review-F and existing benchmarks to demonstrate the efficacy of LIFA compared to other techniques. To contribute to the Vietnamese NLP research, we publish our source code and datasets to the research community upon acceptance.

Multi-source transfer learning with Graph Neural Network for excellent modelling the bioactivities of ligands targeting orphan G protein-coupled receptors.

G protein-coupled receptors (GPCRs) have been the targets for more than 40% of the currently approved drugs. Although neural networks can effectively improve the accuracy of prediction with the biological activity, the result is undesirable in the limited orphan GPCRs (oGPCRs) datasets. To this end, we proposed Multi-source Transfer Learning with Graph Neural Network, called MSTL-GNN, to bridge this gap. Firstly, there are three ideal sources of data for transfer learning, oGPCRs, experimentally validated GPCRs, and invalidated GPCRs similar to the former one. Secondly, the SIMLEs format GPCRs convert to graphics, and they can be the input of Graph Neural Network (GNN) and ensemble learning for improving prediction accuracy. Finally, our experiments show that MSTL-GNN remarkably improves the prediction of GPCRs ligand activity value compared with previous studies. On average, the two evaluation indexes we adopted, R2 and Root-mean-square deviation (RMSE). Compared with the state-of-the-art work MSTL-GNN increased up to 67.13% and 17.22%, respectively. The effectiveness of MSTL-GNN in the field of GPCR Drug discovery with limited data also paves the way for other similar application scenarios.

Multisource Heterogeneous Transfer Learning via Feature Augmentation for Ship Classification in SAR Imagery

Improving ship classification performance in synthetic aperture radar (SAR) imagery by the methods based on transfer learning (TL) is a newly emerging research topic and has great potential. The existing studies merely address the problem of transfer learning from a single source domain (AIS or ORS) to the target domain (SAR) based on the homogeneous transfer learning (HoTL) which requires all domains are represented by homogeneous features with same dimensions. Our work takes a step forward and attempts to address a more meaningful and challenging problem that transfers knowledge from multiple source domains for the purpose of exploring and exploiting complementarity cross source domains to assist ship classification in the target domain. To this end, our study develops a multi-source heterogeneous transfer learning (MS-HeTL) method which liberates the restriction of utilizing the exact same features for all domains, allows each domain represented by a more appropriate feature and thus improves the ship classification performance even further. Specifically, we first propose multi-source heterogeneous feature augmentation (MS-HFA) to effectively solve the challenges brought by feature heterogeneity and fully exploit the complementarity cross domains. Then a support vector machine (SVM) classification framework is specific-designed to be incorporated with the augmented feature representations in the common space to conduct knowledge transfer cross domains. Extensive experiments on two benchmark datasets, HR-SAR and FUSAR, show that the proposed method outperforms the existing methods, and demonstrates its effectiveness and advantages. All datasets and source codes are available at https://github.com/BUCT-RS-ML/MS-HeTL-via-MS-HFA.