A Decade Survey of Transfer Learning (2010–2020)

For the treatment of Interstitial Lung Disease, it is crucial to have an early diagnosis. However, doctors still have a lot of controversy in the diagnosis of lung nodules even with today’s highly developed medical imaging technology. In this article, we summarized the five major challenges we face in medical image recognition and systematically listed the applications from traditional image recognition technology to deep learning in lung CT image recognition. Compared to the traditional convolutional neural network built and trained from scratch, it is beneficial to apply transfer learning to the recognition of lung nodules. Transfer learning focus on transferring knowledge from previous well-trained task to target learning task. Transferring means pretrained networks utilize fine-tuning to reduce iteration times of weight so that it can cope with the problem of lack of high quality images. Various experiments demonstrate that transfer learning performances better than traditional convolutional neural network under complicated circumstances of image recognition such as medical images. In this article, transfer learning is classified into 3 types: inductive transfer learning, transductive transfer learning and unsupervised transfer learning. The main difference between them is label quantity of target training set. Inductive transfer learning highly depends on feature engineering. Compared to it, training sets of two remaining has few labels. However, transductive transfer learning and unsupervised transfer learning are unstable while facing sophisticated cases.

Convolution neural network has long been used in the field of digital image processing and speech recognition, and has achieved great success. Before the convolutional neural network was proposed, both image processing and speech recognition were done by traditional machine learning algorithms. Although great results were achieved, it was difficult to make further breakthroughs, so CNN came into being. Currently, CNN for image processing and speech recognition are relatively mature. Both the theoretical research and the industrial application have been very successful, which has promoted CNN's leap-forward development. CNN's success of image processing and speech recognition has stimulated its research frenzy in natural language processing. The current CNN to handle natural language has been widely used, although some achievements have been made, the current effect is not very good. The purpose of this paper is to hopefully give a clearer explanation of the structure of CNN. At the same time, give a brief summary and prospect of current CNN research in image processing, speech recognition and natural language processing.

The wide 3D applications have led to increasing amount of 3D object data, and thus effective 3D object classification technique has become an urgent requirement. One important and challenging task for 3D object classification is how to formulate the 3D data correlation and exploit it. Most of the previous works focus on learning optimal pairwise distance metric for object comparison, which may lose the global correlation among 3D objects. Recently, a transductive hypergraph learning has been investigated for classification, which can jointly explore the correlation among multiple objects, including both the labeled and unlabeled data. Although these methods have shown better performance, they are still limited due to 1) a considerable amount of testing data may not be available in practice and 2) the high computational cost to test new coming data. To handle this problem, considering the multi-modal representations of 3D objects in practice, we propose an inductive multi-hypergraph learning algorithm, which targets on learning an optimal projection for the multi-modal training data. In this method, all the training data are formulated in multi-hypergraph based on the features, and the inductive learning is conducted to learn the projection matrices and the optimal multi-hypergraph combination weights simultaneously. Different from the transductive learning on hypergraph, the high cost training process is off-line, and the testing process is very efficient for the inductive learning on hypergraph. We have conducted experiments on two 3D benchmarks, i.e., the NTU and the ModelNet40 data sets, and compared the proposed algorithm with the state-of-the-art methods and traditional transductive multi-hypergraph learning methods. Experimental results have demonstrated that the proposed method can achieve effective and efficient classification performance. We also note that the proposed method is a general framework and has the potential to be applied in other applications in practice.

BackgroundThe accurate diagnosis of thyroid nodules represents a critical and frequently encountered challenge in clinical practice, necessitating enhanced precision in diagnostic methodologies. In this study, we investigate the predictive efficacy of distinguishing between benign and malignant thyroid nodules by employing traditional machine learning algorithms and a deep transfer learning model, aiming to advance the diagnostic paradigm in this field.MethodsIn this retrospective study, ITK-Snap software was utilized for image preprocessing and feature extraction from thyroid nodules. Feature screening and dimensionality reduction were conducted using the least absolute shrinkage and selection operator (LASSO) regression method. To identify the optimal model, both traditional machine learning and transfer learning approaches were employed, followed by model fusion using post-fusion techniques. The performance of the model was rigorously evaluated through the area under the curve (AUC), calibration curve analysis, and decision curve analysis (DCA).ResultsA total of 1134 images from 630 cases of thyroid nodules were included in this study, comprising 589 benign nodules and 545 malignant nodules. Through comparative analysis, the support vector machine (SVM), which demonstrated the best diagnostic performance among traditional machine learning models, and the Inception V3 convolutional neural network model, based on transfer learning, were selected for model construction. The SVM model achieved an AUC of 0.748 (95% CI: 0.684–0.811) for diagnosing malignant thyroid nodules, while the Inception V3 transfer learning model yielded an AUC of 0.763 (95% CI: 0.702–0.825). Following model fusion, the AUC improved to 0.783 (95% CI: 0.724–0.841). The difference in performance between the fusion model and the traditional machine learning model was statistically significant (p = 0.036). Decision curve analysis (DCA) further confirmed that the fusion model exhibits superior clinical utility, highlighting its potential for practical application in thyroid nodule diagnosis.ConclusionOur findings demonstrate that the fusion model, which integrates a convolutional neural network (CNN) with traditional machine learning and deep transfer learning techniques, can effectively differentiate between benign and malignant thyroid nodules through the analysis of ultrasound images. This model fusion approach significantly optimizes and enhances diagnostic performance, offering a robust and intelligent tool for the clinical detection of thyroid diseases.

Scientific machine learning is a burgeoning discipline which blends scientific computing and machine learning.Traditionally, scientific computing focuses on large-scale mechanistic models, usually differential equations, that are derived from scientific laws that simplified and explained phenomena.On the other hand, machine learning focuses on developing non-mechanistic data-driven models which require minimal knowledge and prior assumptions.The two sides have their pros and cons: differential equation models are great at extrapolating, the terms are explainable, and they can be fit with small data and few parameters.Machine learning models on the other hand require "big data" and lots of parameters but are not biased by the scientists ability to correctly identify valid laws and assumptions.However, the recent trend has been to merge the two disciplines, allowing explainable models that are data-driven, require less data than traditional machine learning, and utilize the knowledge encapsulated in centuries of scientific literature.The promise is to fuse a priori domain knowledge which doesn't fit into a "dataset", allow this knowledge to specify a general structure that prevents overfitting, reduces the number of parameters, and promotes extrapolatability, while still utilizing machine learning techniques to learn specific unknown terms in the model.This has started to be used for outcomes like automated hypothesis generation and accelerated scientific simulation.The purpose of this blog post is to introduce the reader to the tools of scientific machine learning, identify how they come together, and showcase the existing open source tools which can help one get started.We will be focusing on differentiable programming frameworks in the major languages for scientific machine learning: C++, Fortran, Julia, MATLAB, Python, and R.We will be comparing two important aspects: efficiency and composability.Efficiency will be taken in the context of scientific machine learning: by now most tools are well-optimized for the giant neural networks found in traditional machine learning, but, as will be discussed here, that does not necessarily make them efficient when deployed inside of differential equation solvers or when mixed with probabilistic programming tools.Additionally, composability is a key aspect of scientific machine learning since our toolkit is not ML in isolation.Our goal is not to do machine learning as seen in a machine learning conference (classification, NLP, etc.), and it's not to do traditional machine learning as applied to scientific data.Instead, we are putting ML models and techniques into the heart of scientific simulation tools to accelerate and enhance them.Our neural networks need to fully integrate with tools that simulate satellites and robotics simulators.They need to integrate with the packages that we use in our scientific work for verifying numerical accuracy, tracking units, estimating uncertainty, and much more.We need our neural networks to play nicely with existing packages for delay

Machine learning plays a pivotal role in modern technology, driving advancements across various domains such as healthcare, finance, and autonomous systems. Federated Learning (FL) offers a significant advantage over traditional machine learning by enabling decentralized model training without requiring data to be centralized, thereby enhancing privacy and security. With the advent of 6G networks, which promise ultra-reliable low-latency communications (URLLC) and massive machine-type communications (mMTC), FL can be significantly enhanced. 6G’s improved bandwidth and latency characteristics will enable more efficient data exchange and model updates, further enhancing the adoption of FL. However, the performance of FL can be significantly affected by data distribution, particularly in non-IID (non-Independent and Identically Distributed) scenarios, where FL tends to perform poorly. This paper proposes a novel approach to enhance FL by integrating Transfer Learning (TL) and Continual Learning (CL), named Integrated Federated Transfer and Continual Learning (IFTCL). TL can extract features from client training samples to benefit subsequent clients, while CL mitigates catastrophic forgetting caused by heterogeneous data across clients. This integration improves FL performance under varying degrees of heterogeneous data distributions simulated by Dirichlet distribution, enhancing accuracy, convergence speed, and reducing communication overhead. The proposed method’s feasibility is validated using a publicly available radar recognition dataset.

In the era of big data where data consist of images, text, audio etc., manually analysing text is a big challenge. Over a long time, research in Natural Language Processing (NLP) has relied upon rule based approach and machine learning approach but these approaches need domain expert to build only task specific NLP systems. However, the recent research in NLP has given rise to pre-trained language models like ELMo, GPT, BERT, DistilBERT etc. which have made transfer learning as state-of-the-art technique in NLP. In this paper, we present the comparison of transfer learning and traditional machine learning approach for text classification. To achieve our goal, we have fine-tuned BERT and DistilBERT for text classification to compare transfer learning approach against the traditional machine learning approach based on Term Frequency-Inverse Document Frequency (TF-IDF) algorithm for feature extraction. The performance of all models is empirically tested on suite of different datasets. The results obtained by fine-tuning pre-trained language models show significant improvement over traditional machine learning approach for text classification. This research study also promotes the use of DistilBERT over BERT as default transfer learning model for various text classification applications because it can be trained in less time with less number of parameters to give similar performance like BERT.

Deep learning has emerged as a powerful tool in various domains, revolutionising machine learning research. However, one persistent challenge is the scarcity of labelled training data, which hampers the performance and generalisation of deep learning models. To address this limitation, researchers have developed innovative methods to overcome data scarcity and enhance deep model learning capabilities. Two prevalent techniques that have gained significant attention are transfer learning and self-supervised learning. Transfer learning leverages knowledge learned from pre-training on a large-scale dataset, such as ImageNet, and applies it to a target task with limited labelled data. This approach allows models to benefit from the learned representations and effectively transfer knowledge to new tasks, resulting in improved learning performance and generalisation. On the other hand, self-supervised learning focuses on training models using pretext tasks that do not require manual annotation, allowing them to learn valuable representations from large amounts of unlabelled data. These learned representations can then be fine-tuned for downstream tasks, mitigating the need for extensive labelled data. In recent years, transfer and self-supervised learning have found applications in various fields, including medical image processing, video recognition, and natural language processing. These approaches have demonstrated remarkable achievements, enabling breakthroughs in areas such as disease diagnosis, object recognition, and language understanding. However, while these methods offer numerous advantages, they also have limitations. For example, transfer learning may face domain mismatch issues between the pre-training and target domains, while self-supervised learning requires careful design of pretext tasks to ensure meaningful representations. This review paper explores the recent applications of these pre-training methods in various fields within the past three years. It delves into the advantages and limitations of each approach, assesses the performance of models employing these techniques, and identifies potential directions for future research. By providing a comprehensive review of current pre-training methods, this article offers guidance for selecting the best technique for specific deep learning applications to address the data scarcity issue.

Recently, several deep learning models have been successfully proposed and have been applied to solve different Natural Language Processing (NLP) tasks. However, these models solve the problem based on single-task supervised learning and do not consider the correlation between the tasks. Based on this observation, in this paper, we implemented a multi-task learning model to joint learn two related NLP tasks simultaneously and conducted experiments to evaluate if learning these tasks jointly can improve the system performance compared with learning them individually. In addition, a comparison of our model with the state-of-the-art learning models, including multi-task learning, transfer learning, unsupervised learning and feature based traditional machine learning models is presented. This paper aims to 1) show the advantage of multi-task learning over single-task learning in training related NLP tasks, 2) illustrate the influence of various encoding structures to the proposed single- and multi-task learning models, and 3) compare the performance between multi-task learning and other learning models in literature on textual entailment task and semantic relatedness task.

Traditional machine learning and data mining have made tremendous progress in many knowledge-based areas, such as clustering, classification, and regression. However, the primary assumption in all of these areas is that the training and testing data should be in the same domain and have the same distribution. This assumption is difficult to achieve in real-world applications due to the limited availability of labeled data. Associated data in different domains can be used to expand the availability of prior knowledge about future target data. In recent years, transfer learning has been used to address such cross-domain learning problems by using information from data in a related domain and transferring that data to the target task. The transfer learning methodology is utilized in this work with unsupervised and supervised learning methods. For unsupervised learning, a novel transfer-learning possibilistic c-means (TLPCM) algorithm is proposed to handle the PCM clustering problem in a domain that has insufficient data. Moreover, TLPCM overcomes the problem of differing numbers of clusters between the source and target domains. The proposed algorithm employs the historical cluster centers of the source data as a reference to guide the clustering of the target data. The experimental studies presented here were thoroughly evaluated, and they demonstrate the advantages of TLPCM in both synthetic and real-world transfer datasets. For supervised learning, a transfer learning (TL) technique is used to pre-train a CNN model on posture data and then fine-tune it on the sleep stage data. We used a ballistocardiography (BCG) bed sensor to collect both posture and sleep stage data to provide a non-invasive, in-home monitoring system that tracks changes in the subjects' health over time. The quality of sleep has a significant impact on health and life. This study adopts a hierarchical and none-hierarchical classification structure to develop an automatic sleep stage classification system using ballistocardiogram (BCG) signals. A leave-one-subject-out cross-validation (LOSO-CV) procedure is used for testing classification performance in most of the experiments. Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM), and Deep Neural Networks DNNs are complementary in their modeling capabilities, while CNNs have the advantage of reducing frequency variations, LSTMs are good at temporal modeling. Polysomnography (PSG) data from a sleep lab was used as the ground truth for sleep stages, with the emphasis on three sleep stages, specifically, awake, rapid eye movement (REM), and non-REM sleep (NREM). Moreover, a transfer learning approach is employed with supervised learning to address the cross-resident training problem to predict early illness. We validate our method by conducting a retrospective study on three residents from TigerPlace, a retirement community in Columbia, MO, where apartments are fitted with wireless networks of motion and bed sensors. Predicting the early signs of illness in older adults by using a continuous, unobtrusive nursing home monitoring system has been shown to increase the quality of life and decrease care costs. Illness prediction is based on sensor data and uses algorithms such as support vector machine (SVM) and k-nearest neighbors (kNN). One of the most significant challenges related to the development of prediction algorithms for sensor networks is the use of knowledge from previous residents to predict new ones' behaviors. Each day, the presence or absence of illness was manually evaluated using nursing visit reports from a homegrown electronic medical record (EMR) system. In this work, the transfer learning SVM approach outperformed three other methods, i.e., regular SVM, one-class SVM, and one-class kNN.

A transfer learning environment is characterized by a machine learning algorithm being trained with data from one domain (the source domain) and being tested on data from a different domain (the target domain). In a transfer learning scenario, the class probability of the source domain may be different from the class probability of the target domain, which is referred to as "domain class imbalance". Domain class imbalance is different from "class imbalance". Class imbalance refers to the condition of a single domain having unequal class probabilities. In traditional machine learning, the training and testing data are drawn from a single domain. The effects of class imbalance in traditional machine learning are well studied, however, the issue of domain class imbalance in the field of transfer learning has received little research attention. This paper provides a comparative performance test of state-of-the-art transfer learning algorithms, using a wide-range of domain class imbalance combinations. A detailed discussion on the relative performances of the different algorithms, with statistical validation, is presented for the different domain class imbalance scenarios.

The Radiotherapy Incident Reporting and Analysis System (RIRAS) receives incident reports from Radiation Oncology facilities across the US Veterans Health Affairs (VHA) enterprise and Virginia Commonwealth University (VCU). In this work, we propose a computational pipeline for analysis of radiation oncology incident reports. Our pipeline uses machine learning (ML) and natural language processing (NLP) based methods to predict the severity of the incidents reported in the RIRAS platform using the textual description of the reported incidents. These incidents in RIRAS are reviewed by a radiation oncology subject matter expert (SME), who initially triages some incidents based on the salient elements in the incident report. To automate the triage process, we used the data from the VHA treatment centers and the VCU radiation oncology department. We used NLP combined with traditional ML algorithms, including support vector machine (SVM) with linear kernel, and compared it against the transfer learning approach with the universal language model fine-tuning (ULMFiT) algorithm. In RIRAS, severities are divided into four categories; A, B, C, and D, with A being the most severe to D being the least. In this work, we built models to predict High (A & B) vs. Low (C & D) severity instead of all the four categories. Models were evaluated with macro-averaged precision, recall, and F1-Score. The Traditional ML machine learning (SVM-linear) approach did well on the VHA dataset with 0.78 F1-Score but performed poorly on the VCU dataset with 0.5 F1-Score. The transfer learning approach did well on both datasets with 0.81 F1-Score on VHA dataset and 0.68 F1-Score on the VCU dataset. Overall, our methods show promise in automating the triage and severity determination process from radiotherapy incident reports.

Transfer learning for small molecule retention predictions

Machine learning techniques have long been the foundations of speech processing. Bayesian classiflcation, decision trees, unsupervised clustering, the EM algorithm, maximum entropy, etc. are all part of existing speech recognition systems. The success of statistical speech recognition has led to the rise of statistical and empirical methods in natural language processing. Indeed, many of the machine learning techniques used in language processing, from statistical part-of-speech tagging to the noisy channel model for machine translation have roots in work conducted in the speech fleld. However, advances in learning theory and algorithmic machine learning approaches in recent years have led to signiflcant changes in the direction and emphasis of the statistical and learning centered research in natural language processing and made a mark on natural language and speech processing. Approaches such as memory based learning, a range of linear classiflers such as Boosting, SVMs and SNoW and others have been successfully applied to a broad range of natural language problems, and these now inspire new research in speech retrieval and recognition. We have seen an increasingly close collaboration between voice and language processing researchers in some of the shared tasks such as spontaneous speech recognition and understanding, voice data information extraction, and machine translation. The purpose of this special issue was to invite speech and language researchers to communicate with each other, and with the machine learning community on the latest machine learning advances in their work. The call for papers was met with great enthusiasm from the speech and natural language community. Thirty six submissions were received; each paper was reviewed by at least three reviewers. Only ten papers were selected re∞ecting not only some of the best work on machine learning in the areas of natural language and spoken language processing but also what we view as a collection of papers that represent current trends in these areas of research both from the perspective of

Granular transfer learning using type-2 fuzzy HMM for text sequence recognition