Unsupervised Transfer Learning Research Articles

1D-DRSETL: a novel unsupervised transfer learning method for cross-condition fault diagnosis of rolling bearing

Transfer learning can meet the challenge of cross-condition fault diagnosis. However, the diagnostic effectiveness of transfer learning in actual applications is unsatisfactory, mainly due to the great unbalance in labeling between testing and training samples. A one-dimensional dual residual squeeze-and-excitation transfer learning network (1D-DRSETL) is proposed for an unsupervised accurate intelligent diagnosis under cross-condition in this paper for unlabeled small sample. First, a special block is designed to obtain transferable features by adaptively focusing on fault-sensitive information. Second, the joint maximum mean discrepancy is utilized to deal with the feature matching problem under cross-conditions. Then, speed up model training with AdaBelief optimizer. Finally, cross-conditions transfer diagnosis experiments are designed to demonstrate the superiority of the method based on a self-made dataset and the publicly available rolling bearings dataset. The experimental results show that the proposed method can achieve higher fault diagnosis accuracy and better robustness under cross-conditions than the contrasting methods.

Multisource Deep Transfer Learning Based on Balanced Distribution Adaptation.

The current traditional unsupervised transfer learning assumes that the sample is collected from a single domain. From the aspect of practical application, the sample from a single-source domain is often not enough. In most cases, we usually collect labeled data from multiple domains. In recent years, multisource unsupervised transfer learning with deep learning has focused on aligning in the common feature space and then seeking to minimize the distribution difference between the source and target domains, such as marginal distribution, conditional distribution, or both. Moreover, conditional distribution and marginal distribution are often treated equally, which will lead to poor performance in practical applications. The existing algorithms that consider balanced distribution are often based on a single-source domain. To solve the above-mentioned problems, we propose a multisource transfer learning algorithm based on distribution adaptation. This algorithm considers adjusting the weights of two distributions to solve the problem of distribution adaptation in multisource transfer learning. A large number of experiments have shown that our method MTLBDA has achieved significant results in popular image classification datasets such as Office-31.

Transfer learning: a friendly introduction

Infinite numbers of real-world applications use Machine Learning (ML) techniques to develop potentially the best data available for the users. Transfer learning (TL), one of the categories under ML, has received much attention from the research communities in the past few years. Traditional ML algorithms perform under the assumption that a model uses limited data distribution to train and test samples. These conventional methods predict target tasks undemanding and are applied to small data distribution. However, this issue conceivably is resolved using TL. TL is acknowledged for its connectivity among the additional testing and training samples resulting in faster output with efficient results. This paper contributes to the domain and scope of TL, citing situational use based on their periods and a few of its applications. The paper provides an in-depth focus on the techniques; Inductive TL, Transductive TL, Unsupervised TL, which consists of sample selection, and domain adaptation, followed by contributions and future directions.

A Class-Level Matching Unsupervised Transfer Learning Network for Rolling Bearing Fault Diagnosis Under Various Working Conditions

A Class-Level Matching Unsupervised Transfer Learning Network for Rolling Bearing Fault Diagnosis Under Various Working Conditions

Unsupervised transfer learning for condition monitoring of a fleet of wind turbines

The condition monitoring and health status prediction of a fleet of wind turbines are essential for the safety of wind turbines. At present, the Supervisory Control And Data Acquisition (SCADA) system has been widely used in wind turbines, which can monitor and collect various physical information and sensor information of wind turbines in real-time. Due to the fact that the amount of data obtained by SCADA systems is extremely large, developing an intelligent decision-making system based on deep learning is a very valuable research. Therefore, this paper is committed to exploring a health status prediction algorithm of wind turbines based on deep learning and SCADA systems. However, yet in actual industrial applications, it is very time-consuming and expensive to obtain a large amount of labeled data. In addition, as failures rarely occur, there is a serious sample imbalance problem in the datasets. More importantly, due to the difference in working environment and physical parameters, there are significant differences in the feature distribution of different wind turbines data, which lead to a significant drop in the performance of the deep learning model on unknown wind turbines.
 Therefore, we propose an unsupervised transfer learning algorithm based on Generative Adversarial Networks for wind turbine health status prediction (WT-GAN). WT-GAN can not only remove the domain shift between wind turbines, but also it is an unsupervised learning method. This practically means that only the unlabeled data for the target domain is required, which solves the problem of labeling data. In order to evaluate the effectiveness of WT-GAN on the condition monitoring of a fleet of wind turbines, we apply this method to one dataset about blade icing detection of wind turbines. The experimental results prove that the proposed method can predict the health of the wind turbine well. In addition, it can significantly reduce the domain shift among different wind turbines, thereby achieving excellent performance on unknown wind turbines.

A fault diagnosis method for rolling bearings based on inter-class repulsive force discriminant transfer learning

For fault diagnosis of rolling bearings, it is generally difficult or even impossible to obtain class labels of new working condition samples under actual variable working conditions, which leads to a low fault diagnosis accuracy. On account of this, we propose a novel unsupervised transfer learning method called inter-class repulsive force discriminant transfer learning (IRFDTL) in this paper. In the proposed IRFDTL, to reduce the classification error in source domain and improve the generalization ability of IRFDTL, a nonnegative extended slack matrix is creatively constructed to transform the strict binary label matrix into an extended slack label matrix. Moreover, the joint distribution discrepancy is introduced to reduce the difference between source and target domains, and the inter-class repulsive force term is innovatively designed to promote the discriminative learning effect by increasing the inter-class distance. Finally, the whole framework of IRFDTL is optimized by the alternating direction multiplier method. By using the labeled samples under historical working conditions, the IRFDTL can perform high-precision class discrimination on the testing samples under new working conditions when there are no class labels of testing samples. The proposed IRFDTL-based fault diagnosis method can achieve precise fault diagnosis of the testing samples under new working conditions, and fault diagnosis instances of rolling bearings verify the effectiveness of the proposed method.

Self-starting process monitoring based on transfer learning

Conventional self-starting control schemes can perform poorly when monitoring processes with early shifts, being limited by the number of historical observations sampled. In real applications, pre-observed data sets from other production lines are always available, prompting us to propose a scheme that monitors the target process using historical data obtained from other sources. The methodology of self-taught clustering from unsupervised transfer learning is revised to transfer knowledge from previous observations and improve out-of-control (OC) performance, especially for processes with early shifts. However, if the difference in distribution between the target process and the pre-observed data set is large, our scheme may not be the best. Simulation results and two illustrative examples demonstrate the superiority of the proposed scheme.

Should We Embed in Chemistry? A Comparison of Unsupervised Transfer Learning with PCA, UMAP, and VAE on Molecular Fingerprints.

Methods for dimensionality reduction are showing significant contributions to knowledge generation in high-dimensional modeling scenarios throughout many disciplines. By achieving a lower dimensional representation (also called embedding), fewer computing resources are needed in downstream machine learning tasks, thus leading to a faster training time, lower complexity, and statistical flexibility. In this work, we investigate the utility of three prominent unsupervised embedding techniques (principal component analysis—PCA, uniform manifold approximation and projection—UMAP, and variational autoencoders—VAEs) for solving classification tasks in the domain of toxicology. To this end, we compare these embedding techniques against a set of molecular fingerprint-based models that do not utilize additional pre-preprocessing of features. Inspired by the success of transfer learning in several fields, we further study the performance of embedders when trained on an external dataset of chemical compounds. To gain a better understanding of their characteristics, we evaluate the embedders with different embedding dimensionalities, and with different sizes of the external dataset. Our findings show that the recently popularized UMAP approach can be utilized alongside known techniques such as PCA and VAE as a pre-compression technique in the toxicology domain. Nevertheless, the generative model of VAE shows an advantage in pre-compressing the data with respect to classification accuracy.

Abstract 221: Develop a pathway level classifier to identify clinically relevant subtypes of glioblastoma

Abstract Introduction: Glioblastoma Multiforme (GBM) is a malignant Grade IV brain tumor. With standard treatment, the median survival for adults with GBM, IDH-wildtype, is approximately 11-15 months. We aim at identifying clinically relevant GBM subtypes based on cell signaling pathways information to facilitate personalized medicine. Methods: We rebuilt a MultiPLIER model for GBM using the recount2 compendium with updated V7.0 canonical pathways from the MSigDB database. The MultiPLIER is a machine learning model based on pathway-level information extractor (PLIER), a matrix factorization approach to identify specific pathways that regulate gene expression using a large public dataset and prior biological knowledge from multiple tissues and biological conditions. It has two inputs, the gene expression matrix, and the prior knowledge. PLIER constructed eigengene like latent variables (LVs) to approximate relevant pathways by decomposing gene expression data and a sparse matrix to specify prior information gene sets and pathways in each LV. The MultiPLIER is an unsupervised transfer learning framework to transfer the PLIER model to a specific dataset or disease with smaller sample sizes. Knowledge learned in an extensive collection of datasets can be transferred to a target domain to discover unseen patterns. We used 2315 gene sets, which have 12604 genes as prior biological information in the PLIER training. We produced a decomposition of 903 LVs with 200 LVs with high confidence (area under curve (AUC) of >0.7, false discovery rate (FDR) of <0.05). We then projected the TCGA GBM HT_HG-U133A dataset with 526 primary solid tumor samples and ten solid tissue normal samples to the MultiPLIER 903-dimension space to get a GBM-MultiPLIER model. Results: We used the univariate Cox model to select 169 survival-related LVs specific to GBM subtype studies (p <0.05). Then we used an unsupervised clustering method, consensus clustering (Monti et al., 2003), to discover GBM subtypes. Five subtypes were obtained with the p value 0.00009 of the log-rank test of survival analysis. We also performed Silhouette width, the statistical significance of clustering, and differential expression tests with the Bioconductor package, CancerSubtypes in R. We found the differentially expressed LVs (adjusted p-value<0.01) patterns between each subtype and normal samples. There are also 37 shared LVs among these five subtypes relative to normal samples (adjusted p-value<0.01). The subtype three, which is the best survival subtype, has one LV that has an estimated AUC (Area under the ROC Curve) 0.81, with a 95% confidence interval (0.75,0.86) when subtype 3 compared to other tumor samples. Conclusions: The GBM MultiPLIER model can reveal a consistent pathway or gene set differences across subtypes and capture subtype-specific patterns. These findings provide an opportunity to untangle the underlying biologic meaning further. Citation Format: Xuelian Xu, Zhen Sun, Ning Wen. Develop a pathway level classifier to identify clinically relevant subtypes of glioblastoma [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2021; 2021 Apr 10-15 and May 17-21. Philadelphia (PA): AACR; Cancer Res 2021;81(13_Suppl):Abstract nr 221.

Robust Predictive Maintenance for Robotics via Unsupervised Transfer Learning

Industrial robots play an increasingly important role in a growing number of fields. Since the breakdown of a single robot may have a negative impact on the entire process, predictive maintenance systems have gained importance as an essential component of robotics service offerings. The main shortcoming of such systems is that features extracted from a task typically differ significantly from the learnt model of a different task, incurring false alarms. In this paper, we propose a novel solution based on transfer learning which addresses a well-known challenge in predictive maintenance algorithms by passing the knowledge of the trained model from one task to another in order to prevent the need for retraining and to eliminate such false alarms. The deployment of the proposed algorithm on real-world datasets demonstrates that the algorithm can not only distinguish between tasks and mechanical condition change, it further yields a sharper deviation from the trained model in case of a mechanical condition change and thus detects mechanical issues with higher confidence.

Super-Resolution of Cardiac MR Cine Imaging using Conditional GANs and Unsupervised Transfer Learning.

High-resolution (HR), isotropic cardiac Magnetic Resonance (MR) cine imaging is challenging since it requires long acquisition and patient breath-hold times. Instead, 2D balanced steady-state free precession (SSFP) sequence is widely used in clinical routine. However, it produces highly-anisotropic image stacks, with large through-plane spacing that can hinder subsequent image analysis. To resolve this, we propose a novel, robust adversarial learning super-resolution (SR) algorithm based on conditional generative adversarial nets (GANs), that incorporates a state-of-the-art optical flow component to generate an auxiliary image to guide image synthesis. The approach is designed for real-world clinical scenarios and requires neither multiple low-resolution (LR) scans with multiple views, nor the corresponding HR scans, and is trained in an end-to-end unsupervised transfer learning fashion. The designed framework effectively incorporates visual properties and relevant structures of input images and can synthesise 3D isotropic, anatomically plausible cardiac MR images, consistent with the acquired slices. Experimental results show that the proposed SR method outperforms several state-of-the-art methods both qualitatively and quantitatively. We show that subsequent image analyses including ventricle segmentation, cardiac quantification, and non-rigid registration can benefit from the super-resolved, isotropic cardiac MR images, to produce more accurate quantitative results, without increasing the acquisition time. The average Dice similarity coefficient (DSC) for the left ventricular (LV) cavity and myocardium are 0.95 and 0.81, respectively, between real and synthesised slice segmentation. For non-rigid registration and motion tracking through the cardiac cycle, the proposed method improves the average DSC from 0.75 to 0.86, compared to the original resolution images.

Object detection in remote sensing images based on deep transfer learning

Object detection is a basic part in remote sensing image processing. At present, it is more common to conduct the topic based on deep learning, however the volume of remote sensing images has become a limitation. In order to solve the problem of small sample of remote sensing image, transfer learning is combined with deep learning in the research. First, the detection problem is caused by insufficient data, such as over-fitting, which is solved by model-based transfer learning. The structure of models and parameters obtained based on natural images are transferred to the detection task in remote sensing target domain. In addition, it is usually assumed that the distribution of training data and the testing data are the same in detection, but this is not the case. Therefore, how to improve the robustness of training models and widen the scope of application should be taken into consideration. In the research, Domain Adaptation Faster R-CNN (DA Faster R-CNN) algorithm is proposed for detecting aircraft in remote sensing images. Two domain adaptation structures are designed and selected as the criterion of similarity measurement between domains. Adversarial training is applied to alleviate the domain shift. Finally, the effectiveness of the algorithm is certified in the low brightness experiment. DA Faster R-CNN detection algorithm improves the accuracy of the original algorithm for low quality images. It is worth noting that the DA Faster R-CNN algorithm is a kind of unsupervised transfer learning method for remote sensing object detection.

Imitating targets from all sides: an unsupervised transfer learning method for person re-identification

Person re-identification (Re-ID) models usually present a limited performance when they are trained on one dataset and tested on another dataset due to the inter-dataset bias (e.g. completely different identities and backgrounds) and the intra-dataset difference (e.g. camera and pose changes). In other words, the absence of identity labels (who the person is) and pairwise labels (whether a pair of images belongs to the same person or not) leads to failures in unsupervised person Re-ID problem. We argue that synchronous consideration of these two aspects can improve the performance of unsupervised person Re-ID model. In this work, we introduce a Classification and Latent Commonality (CLC) method based on transfer learning for the unsupervised person Re-ID problem. Our method has three characteristics: (1) proposing an imitate model to generate an imitated target domain with estimated identity labels and create a pseudo target domain to compensate the pairwise labels across camera views; (2) formulating a dual classification loss on both the source domain and imitated target domain to learn a discriminative representation and diminish the inter-domain bias; (3) investigating latent commonality and reducing the intra-domain difference by constraining triplet loss on the source domain, imitated target domain and pairwise label target domain (composed of pseudo target domain and target domain). Extensive experiments are conducted on three widely employed benchmarks, including Market-1501, DukeMTMC-reID and MSMT17, and experimental results demonstrate that the proposed method can achieve a competitive performance against other state-of-the-art unsupervised Re-ID approaches.

Anomaly based novel multi-source unsupervised transfer learning approach for carbon emission centric GDP prediction

Historically, non-renewable fuels are predominant energy sources which drive various industrial and economic activities of a nation. Moreover, it has been well established in literature that carbon emission due to fuel consumption has proportional effect on the economic growth of a nation. There are many war-torn/isolated countries in the world whose gross domestic product (GDP) values are unavailable for certain years in the data repository of the World Bank. Since, hardly any work addresses this research gap of estimating missing GDP of such nations. This paper aims to predict the missing per capita GDP of war-torn/isolated countries using only their carbon emission data. The available data of these countries are not sufficient enough to train a robust predictive model to estimate their missing GDPs. Therefore, this paper proposes multi-source unsupervised transfer learning (UTL), which is an emerging yet under-explored area of transfer learning, to enlarge the training datasets in order to build a robust GDP prediction model. Thus, it exploits multi-source UTL to generate a set of distinct training domains from the collection of carbon emission dataset of various developing countries as well as a mixture of developed and developing countries by suitably detecting and removing the anomalies using isolation forest. To demonstrate the working of the proposed approach, extensive simulations are carried out by considering the carbon emission data sets of six war-torn/isolated countries viz., Afghanistan, Myanmar, Syria, Vietnam, Yemen and Iraq. Here, five different training domains were generated which are then empirically evaluated using three different kind of predictive regression models. The domain with best resulting accuracy is chosen to predict the missing per capita GDP of these war-torn/isolated nations. It is found that the proposed approach performed exceptionally well in increasing the size of relevant training dataset and hence provides a robust GDP prediction model. The strength and weakness of the proposed approach are also discussed by emphasizing the constraints that needs to be adhered while using the proposed UTL approach.

Unsupervised transfer learning for anomaly detection: Application to complementary operating condition transfer

In industrial applications, anomaly detectors are trained to raise alarms when measured samples deviate from the training data distribution. The samples used to train the model should, therefore, be sufficient in quantity and representative of all healthy operating conditions. However, for systems subject to changing operating conditions, acquiring such comprehensive datasets requires a long collection period.To train more robust anomaly detectors, we propose a new framework to perform unsupervised transfer learning (UTL) for one-class classification problems. It differs, thereby, from other applications of UTL in the literature which usually aim at finding a common structure between the datasets to perform either clustering or dimensionality reduction. The task of transferring and combining complementary training data in a completely unsupervised way has not been studied yet.The proposed methodology detects anomalies in operating conditions only experienced by other units in a fleet. We propose the use of adversarial deep learning to ensure the alignment of the different units’ distributions and introduce a new loss, inspired by a dimensionality reduction tool, to enforce the conservation of the inherent variability of each dataset. We use a state-of-the-art once-class approach to detect the anomalies. We demonstrate the benefit of the proposed framework using three open source datasets.

EGFR Assessment in Lung Cancer CT Images: Analysis of Local and Holistic Regions of Interest Using Deep Unsupervised Transfer Learning

Statistics have demonstrated that one of the main factors responsible for the high mortality rate related to lung cancer is the late diagnosis. Precision medicine practices have shown advances in the individualized treatment according to the genetic profile of each patient, providing better control on cancer response. Medical imaging offers valuable information with an extensive perspective of the cancer, opening opportunities to explore the imaging manifestations associated with the tumor genotype in a non-invasive way. This work aims to study the relevance of physiological features captured from Computed Tomography images, using three different 2D regions of interest to assess the Epidermal growth factor receptor ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EGFR</i> ) mutation status: nodule, lung containing the main nodule, and both lungs. A Convolutional Autoencoder was developed for the reconstruction of the input image. Thereafter, the encoder block was used as a feature extractor, stacking a classifier on top to assess the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EGFR</i> mutation status. Results showed that extending the analysis beyond the local nodule allowed the capture of more relevant information, suggesting the presence of useful biomarkers using the lung with nodule region of interest, which allowed to obtain the best prediction ability. This comparative study represents an innovative approach for gene mutations status assessment, contributing to the discussion on the extent of pathological phenomena associated with cancer development, and its contribution to more accurate Artificial Intelligence-based solutions, and constituting, to the best of our knowledge, the first deep learning approach that explores a comprehensive analysis for the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EGFR</i> mutation status classification.

Coupled co-clustering-based unsupervised transfer learning for the integrative analysis of single-cell genomic data.

Unsupervised methods, such as clustering methods, are essential to the analysis of single-cell genomic data. The most current clustering methods are designed for one data type only, such as single-cell RNA sequencing (scRNA-seq), single-cell ATAC sequencing (scATAC-seq) or sc-methylation data alone, and a few are developed for the integrative analysis of multiple data types. The integrative analysis of multimodal single-cell genomic data sets leverages the power in multiple data sets and can deepen the biological insight. In this paper, we propose a coupled co-clustering-based unsupervised transfer learning algorithm (coupleCoC) for the integrative analysis of multimodal single-cell data. Our proposed coupleCoC builds upon the information theoretic co-clustering framework. In co-clustering, both the cells and the genomic features are simultaneously clustered. Clustering similar genomic features reduces the noise in single-cell data and facilitates transfer of knowledge across single-cell datasets. We applied coupleCoC for the integrative analysis of scATAC-seq and scRNA-seq data, sc-methylation and scRNA-seq data and scRNA-seq data from mouse and human. We demonstrate that coupleCoC improves the overall clustering performance and matches the cell subpopulations across multimodal single-cell genomic datasets. Our method coupleCoC is also computationally efficient and can scale up to large datasets. Availability: The software and datasets are available at https://github.com/cuhklinlab/coupleCoC.

Laplacian regularized low-rank sparse representation transfer learning

In unsupervised transfer learning, it is extremely valuable to effectively extract knowledge from the vast amount of untagged data that exists by utilizing tagged data from other similar databases. In general, the data in the real world often resides in the low-dimensional manifold embedded in the high-dimensional environment space. However, the current subspace transfer learning methods do not consider the nonlinear geometry structure inside the data, so the local similarity information between the data may be lost in the learning process. In order to improve this respect, we propose a new subspace transfer learning algorithm, namely Laplacian Regularized Low-Rank Sparse Representation Transfer Learning (LRLRSR-TL). After introducing the low-rank representation and sparse constraints, the method incorporates Laplacian regularization term to represent the global low-dimensional structure and capture the inherent nonlinear geometry information of the data. Experimental investigation conducted based on five different cross-domain visual image datasets shows that the proposed method has outstanding performance compared with several state-of-the-art transfer learning methods.

Multi‐metric domain adaptation for unsupervised transfer learning

Unsupervised domain adaptation aims to learn a classifier for the unlabelled target domain by leveraging knowledge from a labelled source domain. This study presents a novel domain adaptation framework from global and local transfer perspectives, referred to as multi-metric domain adaptation (MMDA) for unsupervised transfer learning. At the global level, MMDA minimises the marginal and within-class distances and maximises the between-class distance between domains while maintaining the features of the source domain to improve the cross-domain adaptability. At the local level, MMDA exploits both in- and cross-domain manifold structures embedded in data samples to increase the discriminative ability. The authors learn a coupled transformation that projects the source and target domain data onto respective subspace where the statistical and geometrical divergences are reduced simultaneously. They formulate global and local adaptation methods in an optimisation problem and derive an analytic solution to the objective function. Extensive experiments demonstrate that MMDA shows improvements in classification accuracy compared with several existing state-of-the-art methods.

Multi-task deep learning for cardiac rhythm detection in wearable devices

Wearable devices enable theoretically continuous, longitudinal monitoring of physiological measurements such as step count, energy expenditure, and heart rate. Although the classification of abnormal cardiac rhythms such as atrial fibrillation from wearable devices has great potential, commercial algorithms remain proprietary and tend to focus on heart rate variability derived from green spectrum LED sensors placed on the wrist, where noise remains an unsolved problem. Here we develop DeepBeat, a multitask deep learning method to jointly assess signal quality and arrhythmia event detection in wearable photoplethysmography devices for real-time detection of atrial fibrillation. The model is trained on approximately one million simulated unlabeled physiological signals and fine-tuned on a curated dataset of over 500 K labeled signals from over 100 individuals from 3 different wearable devices. We demonstrate that, in comparison with a single-task model, our architecture using unsupervised transfer learning through convolutional denoising autoencoders dramatically improves the performance of atrial fibrillation detection from a F1 score of 0.54 to 0.96. We also include in our evaluation a prospectively derived replication cohort of ambulatory participants where the algorithm performed with high sensitivity (0.98), specificity (0.99), and F1 score (0.93). We show that two-stage training can help address the unbalanced data problem common to biomedical applications, where large-scale well-annotated datasets are hard to generate due to the expense of manual annotation, data acquisition, and participant privacy.