Proposed Transfer Learning Approach Research Articles

Transfer learning for pavement performance prediction

Accurate pavement performance prediction models are essential to ensure optimal allocation of resources in maintenance management. These models are developed using inventory and monitoring data regarding pavement structure, climate, traffic, and condition. However, numerous road agencies have limited pavement data. Due to the inexistence of historical data, data collection frequency, and/or quality issues, the amount of data available for the development of performance models is reduced. As a result, the resource allocati on process is significantly undermined. This paper proposes a transfer learning approach to develop pavement performance prediction models in limited data contexts. The proposed transfer learning approach is based on a boosting algorithm. In particular, a modified version of the popular TrAdaBoost learning algorithm was used. To test the proposed transfer learning approach, a case study was developed using data from the Long-Term Pavement Performance (LTPP) database and from the Portuguese road administration database. The results of this work show that it is possible to develop accurate performance prediction models in limited data contexts when a transfer learning approach is applied. All the models resulting from this approach outperformed baseline models, especially in what regards long-term forecasts. The results also showed that the transfer learning models perform consistently over different time frames, with minor performance losses from one-step to multi-step forecasts. The findings of this study should be of interest to road agencies facing limited data contexts and aiming to develop accurate prediction models that can improve their pavement management practice.

Image Forgery Detection Based on Deep Transfer Learning

The recent digital revolution has sparked a growing interest in applying convolutional neural networks (CNNs) and deep learning to the field of image forensics. The proposed methods aim to train algorithms for solving a range of predetermined tasks. However, training a model that has been randomly initialized requires extensive time for computation as well as an enormous pool of training data to draw from. Moreover, such a model needs to be developed and redeveloped from the ground up if there are any alterations to the feature-space distribution. In addressing these problems, the present paper proposes a novel approach to training image forgery detection models. The method applies prior knowledge that has been transferred to the new model from previous steganalysis models. Additionally, because CNN models generally perform badly when transferred to other databases, transfer learning accomplished through knowledge transfer allows the model to be easily trained for other databases. The various models are then evaluated using image forgery techniques such as shearing, rotating, and scaling images. The experimental results, which show an image manipulation detection has validation accuracy of over 94.89%, indicate that the proposed transfer learning approach successfully accelerates CNN model convergence but does not improve image quality.

Weighted Transfer Learning for Improving Motor Imagery-Based Brain–Computer Interface

One of the major limitations of motor imagery (MI)-based brain-computer interface (BCI) is its long calibration time. Due to between sessions/subjects variations in the properties of brain signals, typically, a large amount of training data needs to be collected at the beginning of each session to calibrate the parameters of the BCI system for the target user. In this paper, we propose a novel transfer learning approach on the classification domain to reduce the calibration time without sacrificing the classification accuracy of MI-BCI. Thus, when only few subject-specific trials are available for training, the estimation of the classification parameters is improved by incorporating previously recorded data from other users. For this purpose, a regularization parameter is added to the objective function of the classifier to make the classification parameters as close as possible to the classification parameters of the previous users who have feature spaces similar to that of the target subject. In this paper, a new similarity measure based on the Kullback-Leibler divergence (KL) is used to measure the similarity between two feature spaces obtained using subject-specific common spatial patterns (CSP). The proposed transfer learning approach is applied on the logistic regression classifier and evaluated using three datasets. The results showed that compared with the subject-specific classifier, the proposed weighted transfer learning classifier improved the classification results, particularly when few subject-specific trials were available for training (p < 0.05). Importantly, this improvement was more pronounced for users with medium and poor accuracy. Moreover, the statistical results showed that the proposed weighted transfer learning classifier performed significantly better than the considered comparable baseline algorithms.

Neural transfer learning for assigning diagnosis codes to EMRs

Electronic medical records (EMRs) are manually annotated by healthcare professionals and specialized medical coders with a standardized set of alphanumeric diagnosis and procedure codes, specifically from the International Classification of Diseases (ICD). Annotating EMRs with ICD codes is important for medical billing and downstream epidemiological studies. However, manually annotating EMRs is both time-consuming and error prone. In this paper, we explore the use of convolutional neural networks (CNNs) for automatic ICD coding. Because many codes occur infrequently, CNN performance is inhibited. Therefore, we propose supplementing EMR data with PubMed indexed biomedical research abstracts through neural transfer learning. Transfer learning is the process of "transferring" knowledge acquired from one task (the source task) to a different (target) task. For the source task, we train a CNN to predict medical subject headings (MeSH) using 1.6 million PubMed indexed biomedical abstracts. For the target task, we train a CNN on 71,463 real-world EMRs collected from the University of Kentucky (UKY) medical center to predict ICD diagnosis codes. We introduce a simple, yet effective, transfer learning methodology which avoids forgetting knowledge gained from the source task. Compared to our prior work using EMRs from the UKY medical center, we improve both the micro and macro F-scores by more than 8%. Likewise, compared to other transfer learning methods, our approach results in nearly 2% improvement in macro F-score. We show that transfer learning can improve CNN performance for EMR coding in the presence of data sparsity issues. Furthermore, we find that our proposed transfer learning approach outperforms other methods with respect to macro F-score. Finally, we analyze how transfer learning impacts codes with respect to code frequency. We find that we achieve greater improvement on infrequent codes compared to improvements in most frequent codes.

Deep learning approach to detect malaria from microscopic images

Malaria is an infectious disease which is caused by plasmodium parasite. Several image processing and machine learning based techniques have been employed to diagnose malaria, using its spatial features extracted from microscopic images. In this work, a novel deep neural network model is introduced for identifying infected falciparum malaria parasite using transfer learning approach. This proposed transfer learning approach can be achieved by unifying existing Visual Geometry Group (VGG) network and Support Vector Machine (SVM). Implementation of this unification is carried out by using “Train top layers and freeze out rest of the layers” strategy. Here, the pre-trained VGG facilitates the role of expert learning model and SVM as domain specific learning model. Initial ‘k’ layers of pre-trained VGG are retained and (n-k) layers are replaced with SVM. To evaluate the proposed VGG-SVM model, a malaria digital corpus has been generated by acquiring blood smear images of infected and non-infected malaria patients and compared with state-of-the-art Convolutional Neural Network (CNN) models. Malaria digital corpus images were used to analyse the performance of VGG19-SVM, resulting in classification accuracy of 93.1% in identification of infected falciparum malaria. Unification of VGG19-SVM shows superiority over the existing CNN models in all performance indicators such as accuracy, sensitivity, specificity, precision and F-Score. The obtained result shows the potential of transfer learning in the field of medical image analysis, especially malaria diagnosis.

Cross Modality Microscopy Segmentation via Adversarial Adaptation.

Deep learning techniques have been successfully applied to automatically segment and quantify cell-types in images acquired from both confocal and light sheet fluorescence microscopy. However, the training of deep learning networks requires a massive amount of manually-labeled training data, which is a very time-consuming operation. In this paper, we demonstrate an adversarial adaptation method to transfer deep network knowledge for microscopy segmentation from one imaging modality (e.g., confocal) to a new imaging modality (e.g., light sheet) for which no or very limited labeled training data is available. Promising segmentation results show that the proposed transfer learning approach is an effective way to rapidly develop segmentation solutions for new imaging methods.

Towards end-to-end speech recognition with transfer learning

A transfer learning-based end-to-end speech recognition approach is presented in two levels in our framework. Firstly, a feature extraction approach combining multilingual deep neural network (DNN) training with matrix factorization algorithm is introduced to extract high-level features. Secondly, the advantage of connectionist temporal classification (CTC) is transferred to the target attention-based model through a joint CTC-attention model composed of shallow recurrent neural networks (RNNs) on top of the proposed features. The experimental results show that the proposed transfer learning approach achieved the best performance among all end-to-end methods and could be comparable to the state-of-the-art speech recognition system for TIMIT when further jointly decoded with a RNN language model.

Structural knowledge transfer for learning Sum-Product Networks

To learn an effective Sum-Product Network (SPN) for probabilistic inference, one needs to have a substantial amount of data. In the case when the training dataset is small, SPN performance can be degraded. In this paper, we investigate how transfer learning can improve a SPN when the number of training examples is limited. In particular, we consider a structural transfer setting where (i) one does not have a source dataset but a source SPN, and (ii) there is some kind of similarity between the source SPN and the target domain. We propose a transfer learning approach called TopTrSPN, utilizing the information of the first layer clusters in the source SPN to learn the first layer of the target SPN. Our approach is motivated by transfer learning characteristics of Convolution Neural Network (CNN) as SPN can be viewed as a probabilistic, general-purpose convolution network. Moreover, since the source SPN may have some distribution differences from the target domain, we perform matching between the two by filtering out inconsistent variables. Empirical results on twenty benchmark datasets show the feasibility of our proposed transfer learning approach for SPN structure learning a target domain. Moreover, our proposed transfer learning approach shows encouraging performance when it is applied to text datasets with different set of variables.

Transfer Learning With Neural Networks for Bearing Fault Diagnosis in Changing Working Conditions

Traditional machine learning algorithms have made great achievements in data-driven fault diagnosis. However, they assume that all the data must be in the same working condition and have the same distribution and feature space. They are not applicable for real-world working conditions, which often change with time, so the data are hard to obtain. In order to utilize data in different working conditions to improve the performance, this paper presents a transfer learning approach for fault diagnosis with neural networks. First, it learns characteristics from massive source data and adjusts the parameters of neural networks accordingly. Second, the structure of neural networks alters for the change of data distribution. In the same time, some parameters are transferred from source task to target task. Finally, the new model is trained by a small amount of target data in another working condition. The Case Western Reserve University bearing data set is used to validate the performance of the proposed transfer learning approach. Experimental results show that the proposed transfer learning approach can improve the classification accuracy and reduce the training time comparing with the conventional neural network method when there are only a small amount of target data.

A transfer learning approach for network modeling

Network models have been widely used in many subject areas to characterize the interactions between physical entities. A typical problem is to identify the network for multiple related tasks that share some similarities. In this case, a transfer learning approach that can leverage the knowledge gained during the modeling of one task to help better model another task is highly desirable. This article proposes a transfer learning approach that adopts a Bayesian hierarchical model framework to characterize the relatedness between tasks and additionally uses L 1-regularization to ensure robust learning of the networks with limited sample sizes. A method based on the Expectation–Maximization (EM) algorithm is further developed to learn the networks from data. Simulation studies are performed that demonstrate the superiority of the proposed transfer learning approach over single-task learning that learns the network of each task in isolation. The proposed approach is also applied to identify brain connectivity networks associated with Alzheimer’s Disease (AD) from functional magnetic resonance image data. The findings are consistent with the AD literature.