Instance-based Transfer Learning Research Articles

Stream salinity prediction in data-scarce regions: Application of transfer learning and uncertainty quantification

Scarcity of stream salinity data poses a challenge to understanding salinity dynamics and its implications for water supply management in water-scarce salt-prone regions around the world. This paper introduces a framework for generating continuous daily stream salinity estimates using instance-based transfer learning (TL) and assessing the reliability of the synthetic salinity data through uncertainty quantification via prediction intervals (PIs). The framework was developed using two temporally distinct specific conductance (SC) datasets from the Upper Red River Basin (URRB) located in southwestern Oklahoma and Texas Panhandle, United States. The instance-based TL approach was implemented by calibrating Feedforward Neural Networks (FFNNs) on a source SC dataset of around 1200 instantaneous grab samples collected by United States Geological Survey (USGS) from 1959 to 1993. The trained FFNNs were subsequently tested on a target dataset (1998-present) of 220 instantaneous grab samples collected by the Oklahoma Water Resources Board (OWRB). The framework's generalizability was assessed in the data-rich Bird Creek watershed in Oklahoma by manipulating continuous SC data to simulate data-scarce conditions for training the models and using the complete Bird Creek dataset for model evaluation. The Lower Upper Bound Estimation (LUBE) method was used with FFNNs to estimate PIs for uncertainty quantification. Autoregressive SC prediction methods via FFNN were found to be reliable with Nash Sutcliffe Efficiency (NSE) values of 0.65 and 0.45 on in-sample and out-of-sample test data, respectively. The same modeling scenario resulted in an NSE of 0.54 for the Bird Creek data using a similar missing data ratio, whereas a higher ratio of observed data increased the accuracy (NSE = 0.84). The relatively narrow estimated PIs for the North Fork Red River in the URRB indicated satisfactory stream salinity predictions, showing an average width equivalent to 25 % of the observed range and a confidence level of 70 %.

High-beta disruption prediction study on HL-2A with instance-based transfer learning

Unmitigated disruptions pose a much more serious threat when large-scale tokamaks are operating in the high performance regime. Machine learning based disruption predictors can exhibit impressive performance. However, their effectiveness is based on a substantial amount of training data. In future reactors, obtaining a substantial amount of disruption data in high performance regimes without risking damage to the machine is highly improbable. Using machine learning to develop disruption predictors on data from the low performance regime and transfer them to the high performance regime is an effective solution for a large reactor-sized tokamak like ITER and beyond. In this study, a number of models are trained using different subsets of data from the HL-2A tokamak experiment. A SHapley Additive exPlanations (SHAP) analysis is executed on the models, revealing that there are different, even contradicting, patterns between different performance regimes. Thus, simply mixing data among different performance regimes will not yield optimal results. Based on this analysis, we propose an instance-based transfer learning technique which trains the model using a dataset generated with an optimized strategy. The strategy involves instance and feature selection based on the physics behind differences in high- and low-performance discharges, as revealed by SHAP model analysis. The TrAdaBoost technique significantly improved the model performance from 0.78 BA (balanced accuracy) to 0.86 BA with a few high-performance operation data.

Local instance-based transfer learning for reinforcement learning

Similarity-based transfer learning for reinforcement learning has garnered attention for its potential to enhance target task learning. However, it faces significant challenges in efficiency and effectiveness, primarily stemming from issues such as sparse reward, long trajectory, and strict similarity. To solve these problems, this paper proposes a local instance-based transfer learning method for reinforcement learning. Instead of relying on sparse reward and long trajectory, this approach leverages the Q value of the local trajectory to evaluate similarity, thereby significantly enhancing transfer efficiency. Furthermore, by relaxing the strictness of the similarity, three transfer policies are proposed to facilitate positive transfer. Extensive experimental results demonstrate that the effectiveness and efficiency of the proposed method in comparison with traditional similarity-based transfer learning methods.

An instance-based transfer learning model with attention mechanism for freight train travel time prediction in the China–Europe railway express

Since the inception of the China–Europe Railway Express (CRE), rail transportation has emerged as a predominant means of transporting goods across international borders. The quality and reliability of railway transport services are constrained by the precision of train travel times. However, the real-time data available for the CRE trains is rather limited due to the volume and duration of operations, which increases the difficulty of forecasting CRE train travel times. In order to enhance the accuracy of predictions, a novel two-stage transfer learning model with the Long Short-Term Memory attention (T.R2_LSTM_A) is introduced in this study. By incorporating attention mechanisms, the Long Short-Term Memory networks (LSTM) module, the two-stage TrAdaBoost.R2 algorithm, and transfer learning (TL), the model captures the inherent characteristics of time series data and attains a high level of accuracy in predictions. This approach effectively mitigates the limitations of current transfer learning methods, which are primarily caused by insufficient sample data. In the model, an instance-based TL method is utilized to address the shortcomings of insufficient data, and an attention-based LSTM is selected as a base learner to overcome the thorny issues of gradient disappearance and explosion. Additionally, LSTM Attention (LSTM_A), a novel attention mechanism, is introduced to improve the ability of the proposed model to capture complicated characteristics. The T.R2_LSTM_A model is systemically evaluated and shown to be superior to Random Forest (RF), Support Vector Regression (SVR), LSTM, convolution-based LSTM (CLSTM), AdaBoost_LSTM, LSTM with attention mechanism (LSTM_Attention), and TrAdaBoost.R2_RF (T.R2_RF) for predicting train travel time, using a real-life case study of CRE. The MSE, RMSE, MAPE, and MAE values for the T.R2_LSTM_A model are equal to 5.572, 2.361, 2.578, and 1.767, respectively. These values surpass those of other widely used prediction methods, indicating that the T.R2_LSTM_A model is well-suited for predicting train travel times in the CRE.

Novel Instance-Based Transfer Learning for Asphalt Pavement Performance Prediction

The deep learning method has been widely used in the engineering field. The availability of the training dataset is one of the most important limitations of the deep learning method. Accurate prediction of pavement performance plays a vital role in road preventive maintenance (PM) and decision-making. Pavement performance prediction based on deep learning has been widely used around the world for its accuracy, robustness, and automation. However, most of the countries in the world have not built their pavement performance historical database, which prevents preventive maintenance using the deep learning method. This study presents an innovative particle swarm optimization (PSO) algorithm-enhanced two-stage TrAdaBoost.R2 transfer learning algorithm, which could significantly increase the pavement performance prediction database. The Long-Term Pavement Performance (LTPP) database is used as the source domain data, and one of the highways in China is chosen as the target domain to predict pavement performance. The results show that the proposed PSO-Two-stage TrAdaBoost.R2 model has the highest accuracy compared with AdaBoost.R2 model and traditional regression decision tree model. The validation case study shows significant consistency between the predicted International Roughness Index (IRI) and the whole-year measurement data with an R2 of 0.7. This study demonstrates the great potential of the innovative instance-based transfer learning method in pavement performance prediction of a region’s lack of data. This study also contributes to other engineering fields that could greatly increase the universality of deep learning.

Adjustment of scRNA-seq data to improve cell-type decomposition of spatial transcriptomics.

Most sequencing-based spatial transcriptomics (ST) technologies do not achieve single-cell resolution where each captured location (spot) may contain a mixture of cells from heterogeneous cell types, and several cell-type decomposition methods have been proposed to estimate cell type proportions of each spot by integrating with single-cell RNA sequencing (scRNA-seq) data. However, these existing methods did not fully consider the effect of distribution difference between scRNA-seq and ST data for decomposition, leading to biased cell-type-specific genes derived from scRNA-seq for ST data. To address this issue, we develop an instance-based transfer learning framework to adjust scRNA-seq data by ST data to correctly match cell-type-specific gene expression. We evaluate the effect of raw and adjusted scRNA-seq data on cell-type decomposition by eight leading decomposition methods using both simulated and real datasets. Experimental results show that data adjustment can effectively reduce distribution difference and improve decomposition, thus enabling for a more precise depiction on spatial organization of cell types. We highlight the importance of data adjustment in integrative analysis of scRNA-seq with ST data and provide guidance for improved cell-type decomposition.

Resource-Constrained Multisource Instance-Based Transfer Learning.

In today's machine learning (ML), the need for vast amounts of training data has become a significant challenge. Transfer learning (TL) offers a promising solution by leveraging knowledge across different domains/tasks, effectively addressing data scarcity. However, TL encounters computational and communication challenges in resource-constrained scenarios, and negative transfer (NT) can arise from specific data distributions. This article presents a novel focus on maximizing the accuracy of instance-based TL in multisource resource-constrained environments while mitigating NT, a key concern in TL. Previous studies have overlooked the impact of resource consumption in addressing the NT problem. To address these challenges, we introduce an optimization model named multisource resource-constrained optimized TL (MSOPTL), which employs a convex combination of empirical sources and target errors while considering feasibility and resource constraints. Moreover, we enhance one of the generalization error upper bounds in domain adaptation setting by demonstrating the potential to substitute the H ∆ H divergence with the Kullback-Leibler (KL) divergence. We utilize this enhanced error upper bound as one of the feasibility constraints of MSOPTL. Our suggested model can be applied as a versatile framework for various ML methods. Our approach is extensively validated in a neural network (NN)-based classification problem, demonstrating the efficiency of MSOPTL in achieving the desired trade-offs between TL's benefits and associated costs. This advancement holds tremendous potential for enhancing edge artificial intelligence (AI) applications in resource-constrained environments.

Predictive model of pump energy consumption based on image feature extraction and instance-based transfer learning

The energy consumption of pumps for water supply accounts for a high proportion of the total global energy consumption. The analysis of the pump's energy consumption factors and the construction of prediction models play a crucial role in carbon emission reductions throughout the water supply process. However, difficulties in accurately constructing input features under complex variables and the scarcity and anomalies of monitoring data during system operation pose great challenges to accurate energy consumption prediction. In order to address this issue, a water pump energy consumption prediction method based on image feature extraction and instance transfer is proposed, using the water pump systems installed in two residential communities as the research objects. Firstly, a novel image-based correlation determination method for feature engineering was introduced and validated on a sufficient dataset. Subsequently, in the context of poor data transfer learning, a water pump energy consumption prediction model, named Two-stage TrAdaBoost.R2 (TSTR), was established by incorporating the image correlation determination method and fine-tuning the model parameters using the particle swarm optimization algorithm to enhance prediction accuracy. The experimental results demonstrated the effectiveness of the proposed image-based correlation determination method, showing a 5.49%–44.33% improvement in the coefficient of variation of root mean square error (CV-RMSE) prediction accuracy compared to classical time-series correlation determination methods. Furthermore, in the poor data transfer prediction experiment, the TSTR-based model achieved 0.90% and 20.22% higher CV-RMSE prediction accuracy compared to two comparative models. These findings confirmed the effectiveness of instance transfer learning methods in poor data scenarios and provided valuable insights for selecting input feature construction methods under complex variables.

Instance-based transfer learning method for locating loose particles inside aerospace equipment

The existing supervised-learning-based loose particle localization method requires a lot of resources to create a complete loose particle localization data set, which leads to the problems of long localization testing cycle, large data storage requirement and low resource utilization. In this study, the authors proposed an instance-based transfer learning method for locating loose particles inside aerospace equipment. It used correlations between the localization data sets corresponding to the aerospace equipment with similar structure or size, to complete the knowledge transfer from the source localization data set to the new localization data set. Specifically, based on the existing localization data set containing a large amount of localization data, when facing new aerospace equipment, only a new localization data set containing small amounts of localization data needed to be created. The TrAdaBoost algorithm is used to learn on the new localization data set, the weights of the localization data in the existing localization data set are calculated, the localization data with higher weights are filtered to add to the new localization data set, thus create the transfer training data set and train the loose particle transfer localization (abbr. LPTL) model. This effectively improves the localization efficiency and data resource utilization. In addition, based on engineering application requirements, the authors proposed the definition of equipment-level loose particle localization accuracy. Tests on aerospace power supplies and aerospace electronic single-machines show that, the appropriate weight threshold was set to 0.35, 6004 pieces of localization data were filtered from the existing localization data set. The average classification accuracy of 82.85% and the average localization accuracy of 83.33% were achieved by the LPTL model based on the parameter-optimized XGBoost ensemble classifier. In theory, this method can be extended and applied to the loose particle material identification and weight estimation research, and has high reference value.

Federated Learning for Distributed IIoT Intrusion Detection Using Transfer Approaches

Considering that low-cost and resource-constrained sensors coupled inherently could be vulnerable to growing numbers of intrusion threats, industrial Internet-of-Things (IIoT) systems are faced with severe security concerns. Data sharing for building high-performance intrusion detection models is also prohibited due to the sensitivity, privacy, and high value of IIoT data. This paper presents an anomaly-based intrusion detection system with federated learning for privacy-preserving machine learning in future IIoT networks. To tackle the urgent issue of training local models with non-independent and identically distributed (non-IID) data, we adopt instance-based transfer learning at local. Furthermore, to boost the performance of this system for IIoT intrusion detection, we propose a rank aggregation algorithm with a weighted voting approach. The proposed system achieves superior detection performance with 95.97% and 73.70% accuracy for AdaBoost and Random Forest respectively, outperforming the baseline models by 12.72% and 14.8%.

A hybrid transfer learning method for transient stability prediction considering sample imbalance

Data-driven transient stability prediction (TSP) exists with issues of model robustness and sample imbalance. An instance-based and parameter-based of hybrid transfer learning (HTL) method for TSP considering sample imbalance is proposed to address these two issues. The instance-based transfer learning is firstly utilized to select those applicable samples from the source domain system when the boundary conditions such as the network topology and the operational mode of power system change, which may significantly shorten the time for time-domain simulation (TDS) of the target domain system. Subsequently, the conditional generative adversarial network (CGAN) is employed to augment the unstable samples for obtaining a more balanced training data set. Finally, the parameter-based transfer learning is adopted to quickly update the model for TSP. Case studies conducted on a New England 10-machine 39-bus system, an IEEE 50-machine 145-bus system and a Western Electricity Coordinating Council (WECC) 29-machine 179-bus system demonstrate superior quality and diversity of the generated samples obtained by the CGAN-based data augmentation algorithm than the ones of traditional methods involving SMOTE, ADASYN, GAN and DCGAN. Furthermore, the results from numerous numerical experiments also indicate that the proposed HTL method considering sample imbalance improves the model robustness of TSP.

Full-scale spatio-temporal traffic flow estimation for city-wide networks: a transfer learning based approach

The full-scale spatio-temporal traffic flow estimation/prediction has always been a hot spot in transportation engineering. The low coverage rate of detectors in transport networks brings difficulties to the city-wide traffic flow estimation/prediction. Moreover, it is difficult for traditional analytical traffic flow models to deal with the traffic flow estimation/prediction problem over urban transport networks in a complex environment. Current data-driven methods mainly focus on road segments with detectors. An instance-based transfer learning method is proposed to estimate network-wide traffic flows including road segments without detectors. Case studies based on simulation data and empirical data collected from the open-source PeMS database are conducted to verify its effectiveness. For the traffic flow estimation of segments without detectors, the mean absolute percentage error (MAPE) is approximately 11% for both datasets, which is superior to the existing methods in the literature and reduces MAPE by two percentage points.

A transfer learning approach based on gradient boosting machine for diagnosis of Alzheimer's disease.

Early detection of Alzheimer’s disease (AD) during the Mild Cognitive Impairment (MCI) stage could enable effective intervention to slow down disease progression. Computer-aided diagnosis of AD relies on a sufficient amount of biomarker data. When this requirement is not fulfilled, transfer learning can be used to transfer knowledge from a source domain with more amount of labeled data than available in the desired target domain. In this study, an instance-based transfer learning framework is presented based on the gradient boosting machine (GBM). In GBM, a sequence of base learners is built, and each learner focuses on the errors (residuals) of the previous learner. In our transfer learning version of GBM (TrGB), a weighting mechanism based on the residuals of the base learners is defined for the source instances. Consequently, instances with different distribution than the target data will have a lower impact on the target learner. The proposed weighting scheme aims to transfer as much information as possible from the source domain while avoiding negative transfer. The target data in this study was obtained from the Mount Sinai dataset which is collected and processed in a collaborative 5-year project at the Mount Sinai Medical Center. The Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset was used as the source domain. The experimental results showed that the proposed TrGB algorithm could improve the classification accuracy by 1.5 and 4.5% for CN vs. MCI and multiclass classification, respectively, as compared to the conventional methods. Also, using the TrGB model and transferred knowledge from the CN vs. AD classification of the source domain, the average score of early MCI vs. late MCI classification improved by 5%.

Instance-based transfer learning for soil organic carbon estimation

Soil organic carbon (SOC) is a vital component for sustainable agricultural production. This research investigates the transfer learning-based neural network model to improve classical machine learning estimation of SOC values from other geochemical and physical soil parameters. The results on datasets based on LUCAS data from 2015 showed that the Instance-based transfer learning model captured the valuable information contained in different source domains (cropland and grassland) of soil samples when estimating the SOC values in arable cropland areas. The effects of using transfer learning are more pronounced in the case of different source (grassland) and target (cropland) domains. Obtained results indicate that the transfer learning (TL) approach provides better or at least equal output results compared to the classical machine learning procedure. The proposed TL methodology could be used to generate a pedotransfer function (PTF) for target domains with described samples and unknown related PTF outputs if the described samples with known related PTF outputs from a different geographic or similar land class source domain are available.

Bandgap prediction on small thermoelectric material dataset via instance-based transfer learning

Transfer learning can be applied to improve the robustness of material property prediction model when only a few data available. In this study, a novel transfer learning model for bandgap prediction of diamond-like structure chalcogenides is presented. The instance-based transfer algorithm Two-stage TrAdaBoost.R2 is used with its special built-in strategy to enable knowledge transfer from previously stored relevant data to the training dataset. The applicability of various material descriptors to this approach is tested. Features generated by MatErials Graph Network suits the transfer scenario best, with the model achieving a mean absolute error of 0.21 eV and an R2 of 0.83, which is almost the same level as other well-designed conventional models. The impact of different amounts of auxiliary data on transfer performance is also discussed. Tests show the trends in accuracy change. These confirm the potential of transfer learning for solving problems with small dataset for material machine learning.

Direct Edge-to-Edge Many-to-Many Latent Feature Transfer Learning

Smart cameras that leverage edge-computing (edge cameras) are increasingly being combined with deep neural networks to realize the Artificial Intelligence of Things (AIoT), enabling a smart life with “low-touch services,” such as unmanned stores. However, deploying many edge cameras and training their models (edge models) in unmanned stores is time consuming and labor intensive. Therefore, studies have utilized transfer learning methods, but training edge models often need powerful servers. Although we previously proposed direct edge-to-edge instance transfer, this approach did not exploit latent features, still required high bandwidth and long training time, and caused privacy leakage. Therefore, we propose direct edge-to-edge many-to-many latent feature transfer learning, which includes elite latent feature (ELF) extraction, direct edge-to-edge one-to-many latent feature transfer learning (DeOmf), and direct edge-to-edge many-to-one latent feature transfer learning (DeMof). Through ELF extraction, DeOmf allows one source edge camera to transfer latent features to multiple target edge cameras, which improves knowledge reuse and accelerates initial model training. Further, DeMof can help exploit the diversity of multiple source edge cameras for one target edge camera. The experimental results show that DeOmf improves accuracy by 6.30%, reduces training time by 32.15%, and saves 22.92% (maximum 83.33%) on transmission cost. In addition, DeMof increases accuracy by 3.42% and saves 66.99% on training time and 56.67% on transmission cost. These improvements can greatly broaden the applicability of our proposed system when comparing to instance-based transfer learning.

Instance-based transfer learning method via modified domain-adversarial neural network with influence function: Applications to design metamodeling and fault diagnosis

The availability of a large amount of high-quality data is critical to the performance of machine-learning models. It is challenging to obtain a training dataset because data collection is costly and time-consuming. However, data scarcity can be overcome and an accurate model can be obtained if data from similar models are reused. In this paper, we propose an instance-based transfer learning method to obtain a more accurate model for situations with data scarcity. The proposed method uses a modified domain-adaptation technique to generate auxiliary target-domain data from source-domain data. Subsequently, useful data are selected from the auxiliary target-domain data to preclude the negative transfer that may leverage source-domain data to reduce the learning performance in the target domain. A modified domain-adversarial neural network was used to generate auxiliary target-domain data in the context of instance-based transfer learning. Particularly, the feature extractor and domain discriminator were trained to extract the domain-invariant features from the source and target domains, whereas the target generator was trained to generate auxiliary target-domain data using the domain-invariant features. Additionally, an influence function that can measure the influence of individual training samples on the learning performance was applied to identify useful data. Three case studies were conducted to validate the proposed method: a mathematical function example, drone blade metamodeling, and bearing fault diagnosis. The results of these case studies indicate a significant improvement in neural network prediction despite data scarcity.

Improved identification of cotton cultivated areas by applying instance-based transfer learning on the time series of MODIS NDVI

Cotton is an important cash crop and strategic material in the world, as the main source of natural and renewable fiber for textiles. Accurate and timely cotton distribution maps are crucial for monitoring and managing cotton cultivation system. Remotely sensed data have been widely used in cropland mapping, whereas relatively less attention has been paid specifically to cotton mapping partly due to the difficulty in obtaining the training samples over large regions. To resolve this issue, this study introduced the instance-based transfer learning to identify the cotton cultivated areas using the remotely sensed images. The annual time series of normalized difference vegetation index (NDVI) derived from Moderate Resolution Imaging Spectroradiometer (MODIS) images was adopted as input features. The random forest (RF) and long short-term memory (LSTM) algorithms integrated with the Transfer AdaBoost (TrAdaBoost) algorithm were adopted to generate the RF-based and LSTM-based TrAdaBoost approaches, respectively. The experiments were conducted in Arkansas State of the United States, where the cropland data layer (CDL) was available and utilized as a source of auxiliary data. The results showed that both the RF-based and LSTM-based TrAdaBoost performed better than the original RF and LSTM under the condition that the training samples in the target domain were limited. The advantages of transfer learning were much greater when the percentage of training samples from the source domain equals 80%. The two transfer learning approaches were then applied to identify the cotton cultivated areas in Uzbekistan. The cotton areas detected by MODIS images in 2018 were agreed well with the statistics at the sub-national level, with the R2 values of 0.57 and 0.64, respectively. These results demonstrate the potential of the RF-based and LSTM-based TrAdaBoost approaches in generating the cotton distribution maps when there are few samples in the target domain.

Modeling of Coal Mill Process Monitoring Based on Instance-based Transfer Learning

Modeling of Coal Mill Process Monitoring Based on Instance-based Transfer Learning

A transfer Learning-Based LSTM strategy for imputing Large-Scale consecutive missing data and its application in a water quality prediction system

In recent years, water quality monitoring has been crucial to improve water resource protection and management. Under the relevant laws and regulations, environmental protection department agencies monitor lakes, streams, rivers, and other types of water bodies to assess water quality conditions. The valid and high-quality data generated from these monitoring activities help water resource managers understand the existing pollution situations, energy consumption problems and pollution control needs. However, there are inevitably many problems with water quality data in the real world due to human mistakes or system failures. One of the most frequently occurring issues is missing data. Although most existing studies have explored classic statistical methods or emerging machine/deep learning methods to fill gaps in data, these methods are not suitable for large-scale consecutive missing data problems. To address this issue, this paper proposes a novel algorithm called TrAdaBoost-LSTM, which integrates state-of-the-art deep learning theory through long short-term memory (LSTM) and instance-based transfer learning through TrAdaBoost. This model inherits the full advantages of the LSTM model and transfer learning technique, namely the powerful ability to capture the long-term dependencies among time series and the flexibility of leveraging the related knowledge from complete datasets to fill in large-scale consecutive missing data. A case study involving Dissolved Oxygen concentrations obtained from water quality monitoring stations is conducted to validate the effectiveness and superiority of the proposed method. The results show that the proposed TrAdaBoost-LSTM model not only improves the imputation accuracy by 15%~25% compared with that of alternative models based on the obtained performance indicators, but also provides potential ideas for similar future research.