Transfer Learning Framework Research Articles

Generalizable and replicable brain-based predictions of cognitive functioning across common psychiatric illness.

A primary aim of computational psychiatry is to establish predictive models linking individual differences in brain functioning with symptoms. In particular, cognitive impairments are transdiagnostic, treatment resistant, and associated with poor outcomes. Recent work suggests that thousands of participants may be necessary for the accurate and reliable prediction of cognition, questioning the utility of most patient collection efforts. Here, using a transfer learning framework, we train a model on functional neuroimaging data from the UK Biobank to predict cognitive functioning in three transdiagnostic samples (ns = 101 to 224). We demonstrate prediction performance in all three samples comparable to that reported in larger prediction studies and a boost of up to 116% relative to classical models trained directly in the smaller samples. Critically, the model generalizes across datasets, maintaining performance when trained and tested across independent samples. This work establishes that predictive models derived in large population-level datasets can boost the prediction of cognition across clinical studies.

Advanced semantic segmentation of aircraft main components based on transfer learning and data-driven approach

Abstract The implementation of Smart Airport and Airport 4.0 visions relies on the integration of automation, artificial intelligence, data science, and aviation technology to enhance passenger experiences and operational efficiency. One essential factor in the integration is the semantic segmentation of the aircraft main components (AMC) perception, which is essential to maintenance, repair, and operations in aircraft and airport operations. However, AMC segmentation has challenges from low data availability, high-quality annotation scarcity, and categorical imbalance, which are common in practical applications, including aviation. This study proposes a novel AMC segmentation solution, employing a transfer learning framework based on a sophisticated DeepLabV3 architecture optimized with a custom-designed Focal Dice Loss function. The proposed solution remarkably suppresses the categorical imbalance challenge and increases the dataset variability with manually annotated images and dynamic augmentation strategies to train a robust AMC segmentation model. The model achieved a notable intersection over union of 84.002% and an accuracy of 91.466%, significantly advancing the AMC segmentation performance. These results demonstrate the effectiveness of the proposed AMC segmentation solution in aircraft and airport operation scenarios. This study provides a pioneering solution to the AMC semantic perception problem and contributes a valuable dataset to the community, which is fundamental to future research on aircraft and airport semantic perception. Graphical abstract

Correction to: LegalATLE: an active transfer learning framework for legal triple extraction

Correction to: LegalATLE: an active transfer learning framework for legal triple extraction

Applications of cluster-based transfer learning in image and localization tasks

Transfer learning can address the issue of insufficient labels in machine learning. Using knowledge in a labeled domain (source domain) can assist in acquiring and learning knowledge in a domain (target domain) that lacks some or all labels. In this paper, we propose a new cluster-based semi-supervised transfer learning (CBSSTL) under a new assumption that samples in the target domain are unlabeled but contain cluster information. Furthermore, we propose a new transfer learning framework and a method for fine-tuning parameters. We tested and compared the proposed method with other unsupervised and semi-supervised transfer learning methods on well-known image datasets. The experimental results demonstrate the effectiveness of the proposed method. Additionally, we created a localization dataset for transfer learning. Finally, we tested and analyzed the proposed method on this dataset. Its particularly challenging nature makes it difficult for our method to work effectively.

Transfer learning for high-dimensional linear regression via the elastic net

In this paper, the high-dimensional linear regression problem is explored via the Elastic Net under the transfer learning framework. Within this framework, potentially related source datasets are leveraged to enhance estimation or prediction beyond what can be achieved solely with the target data. When transferable sources are known, an oracle transfer learning algorithm is proposed based on the Elastic Net. Additionally, the ℓ1/ℓ2 estimation error bounds for the corresponding estimator are established. When the transferable sources are unknown, a novel procedure for detecting transferable sources via the Elastic Net is also proposed, with its selection consistency demonstrated under regular conditions. This method transforms the source detection problem into a variable selection problem in high-dimensional space and always gets results that are consistent with the true outcomes. The performance of these methods is further demonstrated through a variety of numerical examples. Finally, our approach is applied to analyze several real datasets for illustrative purposes.

A Transfer Learning Framework for Deep Multi-Agent Reinforcement Learning

A Transfer Learning Framework for Deep Multi-Agent Reinforcement Learning

Automated lepidopteran pest developmental stages classification via transfer learning framework.

The maize crop is highly susceptible to damage caused by its primary pests, which poses considerable challenges in manually identifying and controlling them at various larval developmental stages. To mitigate this issue, we propose an automated classification system aimed at identifying the different larval developmental stages of 23 instars of 4 major lepidopteran pests: the Asian corn borer, Ostrinia furnacalis (Guenée; Lepidoptera: Crambidae), the fall armyworm, Spodoptera frugiperda (J.E. Smith; Lepidoptera: Noctuidae), the oriental armyworm, Mythimna separata (Walker; Lepidoptera: Noctuidae), and the tobacco cutworm, Spodoptera litura (Fabricius; Lepidoptera: Noctuidae). Employing 5 distinct Convolutional Neural Network architectures-Convnext, Densenet121, Efficientnetv2, Mobilenet, and Resnet-we aimed to automate the process of identifying these larval developmental stages. Each model underwent fine-tuning using 2 different optimizers: stochastic gradient descent with momentum and adaptive moment estimation (Adam). Among the array of models tested, Densenet121, coupled with the Adam optimizer, exhibited the highest classification accuracy, achieving an impressive 96.65%. The configuration performed well in identifying the larval development stages of all 4 pests, with precision, recall, and F1 score evaluation indicators reaching 98.71%, 98.66%, and 98.66%, respectively. Notably, the model was ultimately tested in a natural field environment, demonstrating that Adam_Densenet121 model achieved an accuracy of 90% in identifying the 23 instars of the 4 pests. The application of transfer learning methodology showcased its effectiveness in automating the identification of larval developmental stages, underscoring promising implications for precision-integrated pest management strategies in agriculture.

FGTL: Federated Graph Transfer Learning for Node Classification

Unsupervised multi-source domain transfer in federated scenario has become an emerging research direction, which can help unlabeled target domain to obtain the adapted model through source domains under privacy-preserving. However, when local data is graph, the difference of domains (or data heterogeneity) mainly originate from the difference in node attributes and subgraph structures, leading to serious model drift, which is not considered by the existing related algorithms. Currently, there are two challenges in this scenario: (1) The node representations extracted directly through conventional GNNs lack inter-domain generalized and consistent information, making it difficult to apply existing federated learning algorithms. (2) The knowledge of source domains has quality differences, which may lead to negative transfer. To address these issues, we propose a novel two-phase Federated Graph Transfer Learning (FGTL) framework. In the generalization phase, FGTL utilizes local contrastive learning and global context embedding to force node representations to capture the inter-domain generalized and consistent information, lightly alleviating model drift. In the transfer phase, FGTL utilizes consensus knowledge to force the decision bound of classifier to adapt to the target client. In addition, FGTL+ exploits model grouping to make consensus knowledge generation more efficient, further enhancing the scalability of FGTL. Extensive experiments show that FGTL significantly outperforms state-of-the-art related methods, while FGTL+ further enhances privacy protection and reduces both communication and computation overhead.

Natural language processing‐based deep transfer learning model across diverse tabular datasets for bond strength prediction of composite bars in concrete

AbstractAs conventional machine learning models often struggle with scarcity and structural variation of training data, this paper proposes a novel regression transfer learning framework called transferable tabular regressor (TransTabRegressor) to address this challenge. The TransTabRegressor integrates natural language processing (NLP) for feature encoding, transformer for enhanced feature representation, and deep learning (DL) for robust modeling, facilitating effective transfer learning across tabular datasets using reducing input parameters. By leveraging the NLP data processor, the framework embeds both parameter names and values, enabling it to recognize and adapt to different expressions of similar parameters. For instance, the bond strength of fiber‐reinforced polymer (FRP) bars embedded in ultra‐high‐performance concrete (UHPC) is critical for ensuring the integrity of FRP‐UHPC structures. While pullout tests are widely adopted for their simplicity to generate substantial data, beam tests provide a closer approximation to actual stress conditions but are more complex thus resulting in limited data size. As a verification, the framework is applied to predict the bond strength of FRP bars embedded in UHPC using limited beam test data. A pre‐trained model is first established using 479 pieces of pullout test data. Subsequently, two transfer learning models are developed by fine‐tuning on 115 pieces of beam test data, where 66 correspond to concrete splitting failure and 49 correspond to pullout failure. For comparative analysis, XGBoost and neural network models are directly trained on the beam test data. Evaluation results demonstrate that the transfer learning models achieve significantly improved prediction accuracy and generalization capability. This study significantly highlights the effectiveness of the proposed TransTabRegressor in handling data scarcity and variability in input parameters across various engineering applications.

Spatiotemporal upscaling of sparse air-sea pCO2 data via physics-informed transfer learning.

Global measurements of ocean pCO2 are critical to monitor and understand changes in the global carbon cycle. However, pCO2 observations remain sparse as they are mostly collected on opportunistic ship tracks. Several approaches, especially based on direct learning, have been used to upscale and extrapolate sparse point data to dense estimates using globally available input features. However, these estimates tend to exhibit spatially heterogeneous performance. As a result, we propose a physics-informed transfer learning workflow to generate dense pCO2 estimates that are grounded in real-world measurements and remainphysically consistent. The models are initially trained on dense input predictors against pCO2 estimates from Earth system model simulation, and then fine-tuned to sparse SOCAT observational data. Compared to the benchmark direct learning approach, our transfer learning framework shows major improvements of up to 56-92%. Furthermore, we demonstrate that using models that explicitly account for spatiotemporal structures in the data yield better validation performances by 50-68%. Our strategy thus presents a new monthly global pCO2 estimate that spans for 35 years between 1982-2017.

Deep learning network for indoor point cloud semantic segmentation with transferability

Semantic segmentation is crucial for interpreting point cloud data and plays a fundamental role in automating the creation of as-built BIM. Existing neural network models for semantic segmentation often heavily rely on the training dataset, resulting in a significant performance drop when applied to new datasets. This paper presents AttTransNet, a neural network model for automated point cloud semantic segmentation. Its attention-based pooling module improves local feature extraction from point clouds while reducing computational costs. The transfer learning framework enhances segmentation accuracy with minimal training on target datasets. Comparative experiments show that AttTransNet reduces training time by 80 % and improves segmentation accuracy by over 20 % compared with other SOTA methods. Cross-dataset experiments reveal that the transfer learning framework increases accuracy on new datasets by 150 %. By adding semantic information to point clouds, AttTransNet aids BIM modelers with direct reference, encouraging broader application of automated point cloud segmentation in the industry.

MIMO system identification and uncertainty calibration with a limited amount of data using transfer learning

Multiple-input multiple-output (MIMO) systems are fundamental in numerous advanced engineering applications, from aerospace to telecommunications, where precise system identification is critical for optimal performance. However, the identification of such systems often faces significant hurdles due to data scarcity, with existing approaches typically requiring substantial amounts of data for effective training. Addressing this challenge, this paper introduces a novel transfer learning framework designed specifically for MIMO system identification under conditions of limited data and inherent uncertainties. The proposed framework is applied to two case studies: the first in metal additive manufacturing, specifically the laser-blown powder-directed energy deposition as the source domain and the laser hot wire-directed energy deposition as the target domain, and the second involving a nonlinear case study of a continuous stirred-tank reactor (CSTR) with a temperature-dependent reaction. The results underscore the framework's effectiveness in capturing the dynamics of the target systems, including the ability to effectively model nonlinear dynamics. Comparative analyses highlight the benefits of employing dimensionless numbers in dynamic system modelling, offering reduced dimensionality, more physical meaning, and increased model accuracy. Overall, the proposed framework presents a promising approach to enhance system identification in MIMO systems with limited data and uncertainties, with potential applications across diverse domains.

LegalATLE: an active transfer learning framework for legal triple extraction

LegalATLE: an active transfer learning framework for legal triple extraction

A transfer learning framework for the assessment of unconventional resources opportunities in the Middle East

An inadequate understanding of fluid transport processes in ultra-low permeability rocks has propelled data-driven modeling as an alternative and complementary tool to create recovery forecasts that honor the available data in the field. However, newly discovered fields lack data, which makes it challenging to apply traditional data-driven approaches to decision-making. Transfer learning provides an opportunity to start the early-stage analysis of the asset before adequate data becomes available. Here, we present a transfer learning framework that allows the basin-wide assessment of new shale gas and tight oil prospects. The proposed method is developed on real-world data from several thousand horizontal hydraulically fractured wells in the Eagle Ford super-basin in South Texas. A novel aspect of our method is integrating reservoir engineering domain expertise and the relationship among geologic and petrophysical variables, drilling and completion parameters, and well productivity in the data pre-processing and feature generation steps. We consider the temporal and spatial balancing of the training data to ensure that the resulting predictive models honor the real-world practices followed during unconventional field development. Our full-cycle transfer learning workflow consists of dimensionality reduction and unsupervised clustering, supervised learning, and hyperparameter fine-tuning. We test the workflow by examining the performance of a data-driven model of the Eagle Ford Basin on potential plays in the Middle East: Tuwaiq Mountain and Hanifa. The existence of multiple types of hydrocarbons–liquid oil, condensate, and dry gas–in the Eagle Ford results in a model flexible enough to be tested on various types of assets in a new location. We address the challenge of model overfitting by using data types with a relatively low resolution, which allows generalization to different geologies with basin-wide accuracy.

An Adaptive Transfer Learning Framework for Data-Scarce HVAC Power Consumption Forecasting

An Adaptive Transfer Learning Framework for Data-Scarce HVAC Power Consumption Forecasting

Privacy-Preserving Transfer Learning Framework for Kidney Disease Detection

This paper introduces a new privacy-preserving transfer learning framework for the classification of kidney diseases. In the proposed framework, transfer learning is employed for feature extraction, and differential privacy is used to obtain noisy gradients. A variety of CNN architectures, including Xception, ResNet50, InceptionResNetV2, MobileNet, DenseNet201, InceptionV3, and VGG19 are utilized to evaluate the proposed framework. Analysis of a large dataset of 12,400 labeled kidney CT images shows that transfer learning architectures based on the proposed framework achieve excellent accuracy ratios in privacy-preserving classification. These results demonstrate the effectiveness of the proposed framework in enabling transfer learning models to classify kidney diseases while ensuring privacy. The MobileNet architecture stands out for its exceptional performance, with an impressive accuracy of 99.83% in privacy-preserving classification. Considering the findings of this study, it is evident that the proposed framework is appropriate for the early and private diagnosis of kidney diseases and promotes the achievement of promising results in this field.

Transfer learning for well logging formation evaluation using similarity weights

Machine learning has been widely applied in well logging formation evaluation studies. However, several challenges negatively impacted the generalization capabilities of machine learning models in practical implementations, such as the mismatch of data domain between training and testing datasets, imbalances among sample categories, and inadequate representation of data model. These issues have led to substantial insufficient identification for reservoir and significant deviations in subsequent evaluations. To improve the transferability of machine learning models within limited sample sets, this study proposes a weight transfer learning framework based on the similarity of the labels. The similarity weighting method includes both hard weights and soft weights. By evaluating the similarity between test and training sets of logging data, the similarity results are used to estimate the weights of training samples, thereby optimizing the model learning process. We develop a double experts' network and a bidirectional gated neural network based on hierarchical attention and multi-head attention (BiGRU-MHSA) for well logs reconstruction and lithofacies classification tasks. Oil field data results for the shale strata in the Gulong area of the Songliao Basin of China indicate that the double experts’ network model performs well in curve reconstruction tasks. However, it may not be effective in lithofacies classification tasks, while BiGRU-MHSA performs well in that area. In the study of constructing large-scale well logging processing and formation interpretation models, it is maybe more beneficial by employing different expert models for combined evaluations. In addition, although the improvement is limited, hard or soft weighting methods is better than unweighted (i.e., average-weighted) in significantly different adjacent wells. The code and data are open and available for subsequent studies on other lithofacies layers.

General framework for unsteady aerodynamic prediction of airfoils based on deep transfer learning

Analyzing the unsteady aerodynamic performance of airfoils under dynamic stall using computational fluid dynamics (CFD) is computationally intensive. Although deep learning models can quickly predict aerodynamic parameters, their generalization capability on a small-scale dataset is often poor. This paper presents a novel deep transfer learning (TL) framework that combines model-based and instance-based transfer methods, termed synergistic instance-model TL. This framework facilitates rapid predictions of unsteady aerodynamic performance for various airfoils and pitch oscillations from the small-scale dataset. The framework integrates the accelerated training speed of model-based methods with the dynamic dataset expansion benefits of instance-based approaches. Initially, a pre-trained Wasserstein-deep convolutional generative adversarial network (W-DCGAN) is developed, combining a convolutional neural network with a generative adversarial network to predict aerodynamic hysteresis loops for the SC1095 airfoil in the source domain. The framework then fine-tunes the pre-trained model and incorporates weighted source domain dataset into the small-scale target domain dataset, producing the transferred model W-DCGAN-TL. This approach significantly reduces prediction inaccuracies compared to model-based and non-TL methods when applied to the small-scale dataset. The framework's flexibility allows the use of pre-trained models and datasets from related aerodynamic problems to address issues with insufficient data. Consequently, it is expected to reduce the dependency on extensive datasets, enhance design efficiency, and minimize resource requirements.

T-phPINN: Physics-informed neural networks for solving 2D non-Fourier heat conduction equations

Non-Fourier heat conduction plays a dominant role in many extreme transient heat conduction processes, such as laser pulses and heat transfer in biological systems, but the heat wave effect makes it difficult to solve the temperature field accurately and quickly. In order to solve this problem, the first order time derivative enhanced parallel hard constraints physics-informed neural networks (T-phPINN) is proposed. T-phPINN comprises two subnetworks and incorporates a first order time derivative to capture sharp temperature changes. Two numerical cases show that the minimum relative error of T-phPINN is 0.001 % and 0.015 %, which is 1.04 % and 12.30 % of the error of conventional PINN respectively, proving the accuracy of our architecture. A transfer learning framework is established for scenarios of different parameters, the training only requires 1/6 iterations of the basic model, and close accuracy is obtained. The computational cost of T-phPINN is evaluated using the finite element method as the baseline. For the two cases, the single calculation time is 33.43 % and 51.50 % of the baseline, while the multiple calculation time under the acceleration of transfer learning is 11.59 % and 17.75 % of the baseline. This study will be helpful for solving large-scale non-Fourier heat conduction equations precisely and expeditiously.

A spatiotemporal transfer learning framework with mixture of experts for traffic flow prediction

A spatiotemporal transfer learning framework with mixture of experts for traffic flow prediction