Transfer Learning Research Research Articles

Transfer Learning in Machine Learning: A Review of Methods and Applications

Transfer learning has gained significant traction and popularity in the field of machine learning due to its wide range of potential applications. This review article offers a thorough examination of transfer learning techniques and their wide-ranging applications in several fields. This text provides a thorough evaluation of the literature, focusing on important research and the methodology used. Furthermore, a comparative table highlighting transfer learning research across several areas provides valuable insights into the wide range of applications. The inclusion criteria were centred on recent articles published within the past five years that comprehensively examined transfer learning methodologies, applications, frameworks, problems, and future directions. The review articles highlight the widespread use of transfer learning models, the effectiveness of data augmentation strategies, and the capability of transfer learning to tackle issues particular to different domains. Nevertheless, some constraints like as biases in the dataset, difficulties in interpreting the model, and problems with scalability have been recognised. These limitations provide opportunities for future research to focus on creating transfer learning algorithms that are more resilient and easier to read.

Unsupervised Domain Adaptation for Forest Fire Recognition Using Transferable Knowledge from Public Datasets

Deep neural networks (DNNs) have driven the recent advances in fire detection. However, existing methods require large-scale labeled samples to train data-hungry networks, which are difficult to collect and even more laborious to label. This paper applies unsupervised domain adaptation (UDA) to transfer knowledge from a labeled public fire dataset to another unlabeled one in practical application scenarios for the first time. Then, a transfer learning benchmark dataset called Fire-DA is built from public datasets for fire recognition. Next, the Deep Subdomain Adaptation Network (DSAN) and the Dynamic Adversarial Adaptation Network (DAAN) are experimented on Fire-DA to provide a benchmark result for future transfer learning research in fire recognition. Finally, two transfer tasks are built from Fire-DA to two public forest fire datasets, the aerial forest fire dataset FLAME and the large-scale fire dataset FD-dataset containing forest fire scenarios. Compared with traditional handcrafted feature-based methods and supervised CNNs, DSAN reaches 82.5% performance of the optimal supervised CNN on the testing set of FLAME. In addition, DSAN achieves 95.8% and 83.5% recognition accuracy on the testing set and challenging testing set of FD-dataset, which outperform the optimal supervised CNN by 0.5% and 2.6%, respectively. The experimental results demonstrate that DSAN achieves an impressive performance on FLAME and a new state of the art on FD-dataset without accessing their labels during training, a fundamental step toward unsupervised forest fire recognition for industrial applications.

Transfer learning for landslide susceptibility modeling using domain adaptation and case-based reasoning

Abstract. Transferability of knowledge from well-investigated areas to a new study region is gaining importance in landslide hazard research. Considering the time-consuming compilation of landslide inventories as a prerequisite for landslide susceptibility mapping, model transferability can be key to making hazard-related information available to stakeholders in a timely manner. In this paper, we compare and combine two important transfer-learning strategies for landslide susceptibility modeling: case-based reasoning (CBR) and domain adaptation (DA). Care-based reasoning gathers knowledge from previous similar situations (source areas) and applies it to solve a new problem (target area). Domain adaptation, which is widely used in computer vision, selects data from a source area that has a similar distribution to the target area. We assess the performances of single- and multiple-source CBR, DA, and CBR–DA strategies to train and combine landslide susceptibility models using generalized additive models (GAMs) for 10 study areas with various resolutions (1, 10, and 25 m) located in Austria, Ecuador, and Italy. The performance evaluation shows that CBR and combined CBR–DA based on our proposed similarity criterion were able to achieve performances comparable to benchmark models trained in the target area itself. Particularly the CBR strategies yielded favorable results in both single- and multi-source strategies. Although DA tended to have overall lower performances than CBR, it had promising results in scenarios where the source–target similarity was low. We recommend that future transfer-learning research for landslide susceptibility modeling can build on the similarity criterion we used, as it successfully helped to transfer landslide susceptibility models by identifying suitable source regions for model training.

Transfer Learning for Leaf Small Dataset Using Improved ResNet50 Network with Mixed Activation Functions

Taxonomic studies of leaves are one of the most effective means of correctly identifying plant species. In this paper, mixed activation function is used to improve the ResNet50 network in order to further improve the accuracy of leaf recognition. Firstly, leaf images of 15 common tree species in northern China were collected from the Urban Forestry Demonstration Base of Northeast Forestry University (45°43′–45°44′ N, 126°37′–126°38′ E, forest type was artificial forest), and a small leaf dataset was established. After that, seven commonly used activation functions were selected to improve the ResNet50 network structure, and the improved network was applied to the transfer learning research of the leaf small dataset. On this basis, five activation functions with better performance were selected for the study of mixed activation functions in deep learning. Two of these five activation functions are arbitrarily selected for combination, and a total of twenty combinations are obtained. Further, the first activation function was used in each combination to replace the first ReLU function after all addition operations in the ResNet50 network residual block structure, and another activation function was used to replace the other position ReLU functions. The experimental results show that in the transfer learning of the leaf small dataset using the ResNet50 deep residual network, the appropriate combination of mixed activation functions can increase the performance of the improved network to a certain extent. Among them, the ELU-Swish1 combination has the most significant improvement effect on the network performance, whose final effective validation accuracy reaches 98.17%. Furthermore, the comparison with GoogLeNet and VGG-16 also demonstrates the excellent performance of the improved ELU-Swish1 ResNet50 (ES-ResNet50) network architecture. Finally, tests on the other two small leaf datasets, Flavia and Swedish, also demonstrate the performance improvement of ES-ResNet50. The validation accuracy of the improved ES-Resnet 50 algorithm on these two datasets reaches 99.30% and 99.39%, respectively. All these experiments prove that the recognition performance of leaf transfer learning using the ES-ResNet50 network is indeed improved, which may be caused by the complementarity of the e-exponential gradient of ELU and Swish1 activation functions in the negative region.

Adapting to the Long Tail: A Meta-Analysis of Transfer Learning Research for Language Understanding Tasks.

Natural language understanding (NLU) has made massive progress driven by large benchmarks, but benchmarks often leave a long tail of infrequent phenomena underrepresented. We reflect on the question: Have transfer learning methods sufficiently addressed the poor performance of benchmark-trained models on the long tail? We conceptualize the long tail using macro-level dimensions (underrepresented genres, topics, etc.), and perform a qualitative meta-analysis of 100 representative papers on transfer learning research for NLU. Our analysis asks three questions: (i) Which long tail dimensions do transfer learning studies target? (ii) Which properties of adaptation methods help improve performance on the long tail? (iii) Which methodological gaps have greatest negative impact on long tail performance? Our answers highlight major avenues for future research in transfer learning for the long tail. Lastly, using our meta-analysis framework, we perform a case study comparing the performance of various adaptation methods on clinical narratives, which provides interesting insights that may enable us to make progress along these future avenues.

Transfer learning based on improved stacked autoencoder for bearing fault diagnosis

Deep transfer learning algorithm is regarded as a promising method to address the issue of rolling bearing fault diagnosis with limited labeled data. Stacked autoencoder (SAE) has been widely employed in deep transfer learning research since it is a semi-supervised algorithm. However, there are still some limitations for the transfer learning based on SAE, including the vanishing gradient problem caused by the sigmoid activation function in SAE, and low accuracy under the condition of cross-domain or limited labeled training data. In this work, an improved SAE based on convolutional shortcuts and domain fusion strategy (ISAE-CSDF) is proposed for fault diagnosis of rolling bearing. The sparse term Kullback–Leibler (KL) divergence in the original SAE is replaced with the convolutional shortcuts to prevent vanishing gradient problem and improve the feature extraction ability. The domain fusion strategy can transfer commonly shared feature information from various domains. The feasibility of ISAE-CSDF is validated on two publicly available bearing datasets and a custom-built experiment device. Results show that ISAE-CSDF outperforms the state-of-art methods in the context of different working conditions, cross-domain, and limited labeled data.

Reminder of the First Paper on Transfer Learning in Neural Networks, 1976

This paper describes a work on transfer learning in neural networks carried out in 1970s and early 1980s, which produced its first publication in 1976. In the contemporary research on transfer learning there is a belief that pioneering work on transfer learning took place in early 1990s, and this paper updates that knowledge, pointing out that the transfer learning research started more than a decade earlier. This paper reviews that 1970s research and addresses important issues relevant for the current transfer learning research. It gives a mathematical model and geometric interpretation of transfer learning, and a measure of transfer learning indicating positive, negative, and no transfer learning. It presents experimental investigation in the mentioned types of transfer learning. And it gives an application of transfer learning in pattern recognition using datasets of images.

A Comprehensive Survey on Transfer Learning

Transfer learning aims at improving the performance of target learners on target domains by transferring the knowledge contained in different but related source domains. In this way, the dependence on a large number of target-domain data can be reduced for constructing target learners. Due to the wide application prospects, transfer learning has become a popular and promising area in machine learning. Although there are already some valuable and impressive surveys on transfer learning, these surveys introduce approaches in a relatively isolated way and lack the recent advances in transfer learning. Due to the rapid expansion of the transfer learning area, it is both necessary and challenging to comprehensively review the relevant studies. This survey attempts to connect and systematize the existing transfer learning research studies, as well as to summarize and interpret the mechanisms and the strategies of transfer learning in a comprehensive way, which may help readers have a better understanding of the current research status and ideas. Unlike previous surveys, this survey article reviews more than 40 representative transfer learning approaches, especially homogeneous transfer learning approaches, from the perspectives of data and model. The applications of transfer learning are also briefly introduced. In order to show the performance of different transfer learning models, over 20 representative transfer learning models are used for experiments. The models are performed on three different data sets, that is, Amazon Reviews, Reuters-21578, and Office-31, and the experimental results demonstrate the importance of selecting appropriate transfer learning models for different applications in practice.

Low-Resource Cross-Domain Product Review Sentiment Classification Based on a CNN with an Auxiliary Large-Scale Corpus

The literature [-5]contains several reports evaluating the abilities of deep neural networks in text transfer learning. To our knowledge, however, there have been few efforts to fully realize the potential of deep neural networks in cross-domain product review sentiment classification. In this paper, we propose a two-layer convolutional neural network (CNN) for cross-domain product review sentiment classification (LM-CNN-LB). Transfer learning research into product review sentiment classification based on deep neural networks has been limited by the lack of a large-scale corpus; we sought to remedy this problem using a large-scale auxiliary cross-domain dataset collected from Amazon product reviews. Our proposed framework exhibits the dramatic transferability of deep neural networks for cross-domain product review sentiment classification and achieves state-of-the-art performance. The framework also outperforms complex engineered features used with a non-deep neural network method. The experiments demonstrate that introducing large-scale data from similar domains is an effective way to resolve the lack of training data. The LM-CNN-LB trained on the multi-source related domain dataset outperformed the one trained on a single similar domain.

Transfer learning using computational intelligence: A survey

Transfer learning aims to provide a framework to utilize previously-acquired knowledge to solve new but similar problems much more quickly and effectively. In contrast to classical machine learning methods, transfer learning methods exploit the knowledge accumulated from data in auxiliary domains to facilitate predictive modeling consisting of different data patterns in the current domain. To improve the performance of existing transfer learning methods and handle the knowledge transfer process in real-world systems, computational intelligence has recently been applied in transfer learning. This paper systematically examines computational intelligence-based transfer learning techniques and clusters related technique developments into four main categories: (a) neural network-based transfer learning; (b) Bayes-based transfer learning; (c) fuzzy transfer learning, and (d) applications of computational intelligence-based transfer learning. By providing state-of-the-art knowledge, this survey will directly support researchers and practice-based professionals to understand the developments in computational intelligence-based transfer learning research and applications.

A Survey on Transfer Learning

A major assumption in many machine learning and data mining algorithms is that the training and future data must be in the same feature space and have the same distribution. However, in many real-world applications, this assumption may not hold. For example, we sometimes have a classification task in one domain of interest, but we only have sufficient training data in another domain of interest, where the latter data may be in a different feature space or follow a different data distribution. In such cases, knowledge transfer, if done successfully, would greatly improve the performance of learning by avoiding much expensive data-labeling efforts. In recent years, transfer learning has emerged as a new learning framework to address this problem. This survey focuses on categorizing and reviewing the current progress on transfer learning for classification, regression, and clustering problems. In this survey, we discuss the relationship between transfer learning and other related machine learning techniques such as domain adaptation, multitask learning and sample selection bias, as well as covariate shift. We also explore some potential future issues in transfer learning research.