Adaptive Transfer Learning Research Articles

Efficient Multivariate Time Series Anomaly Detection Through Transfer Learning for Large-Scale Software Systems

Timely anomaly detection of multivariate time series (MTS) is of vital importance for managing large-scale software systems. However, many deep learning-based MTS anomaly detection models require long-term MTS training data to achieve optimal performance, which often conflicts with the frequent pattern changes observed in software systems. Moreover, the training overhead of vast MTS in large-scale software systems is unacceptably high. To address these issues, we design OmniTransfer , a model-agnostic framework that combines weighted hierarchical agglomerative clustering with an adaptive transfer learning strategy, making many state-of-the-art (SOTA) MTS anomaly detection models efficient and effective. Extensive experiments using real-world data from a large web content service provider and a network operator show that OmniTransfer significantly reduces the model initialization time by 46.49% and the training cost by 74.51%, while maintaining high accuracy in detecting anomalies.

An Adaptive Transfer Learning Framework for Data-Scarce HVAC Power Consumption Forecasting

An Adaptive Transfer Learning Framework for Data-Scarce HVAC Power Consumption Forecasting

An Adaptive Transfer Learning Framework for Functional Classification

In this paper, we study the transfer learning problem in functional classification, aiming to improve the classification accuracy of the target data by leveraging information from related source datasets. To facilitate transfer learning, we propose a novel transferability function tailored for classification problems, enabling a more accurate evaluation of the similarity between source and target dataset distributions. Interestingly, we find that a source dataset can offer more substantial benefits under certain conditions than another dataset with an identical distribution to the target dataset. This observation renders the commonly-used debiasing step in the parameter-based transfer learning algorithm unnecessary under some circumstances to the classification problem. In particular, we propose two adaptive transfer learning algorithms based on the functional Distance Weighted Discrimination (DWD) classifier for scenarios with and without prior knowledge regarding informative sources. Furthermore, we establish the upper bound on the excess risk of the proposed classifiers, providing the statistical gain via transfer learning mathematically provable. Simulation studies are conducted to thoroughly examine the finite-sample performance of the proposed algorithms. Finally, we implement the proposed method to Beijing air-quality data, and significantly improve the prediction of the PM 2.5 level of a target station by effectively incorporating information from source datasets.

Application of Artificial Neuromolecular System in Robotic Arm Control to Assist Progressive Rehabilitation for Upper Extremity Stroke Patients

Freedom of movement of the hands is the most desired hope of stroke patients. However, stroke recovery is a long, long road for many patients. If artificial intelligence can assist human arm movement, the possibility of stroke patients returning to normal hand movement might be significantly increased. This study uses the artificial neuromolecular system (ANM system) developed in our laboratory as the core of motion control, in an attempt to learn to control the mechanical arm to produce actions similar to human rehabilitation training and the transition between different activities. This research adopts two methods. The first is hypothetical exploration, the so-called “artificial world” simulation method. The detailed approach uses the V-REP (Virtual Robot Experimentation Platform) to conduct different experimental runs to capture relevant data. Our policy is to establish an action database systematically to a certain extent. From these data, we use the ANM system with self-organization and learning capabilities to develop the relationship between these actions and establish the possibility of conversion between different activities. The second method of this study is to use the data from a hospital in Toronto, Canada. Our experimental results show that the ANM system can continuously learn for problem-solving. In addition, our three experimental results of adaptive learning, transfer learning, and cross-task learning further confirm that the ANM system can use previously learned systems to complete the delivered tasks through autonomous learning (instead of learning from scratch).

Multi-objective evolutionary algorithm based on transfer learning and neural networks: Dual operator feature fusion and weight vector adaptation

In multi-objective evolutionary algorithms, one of the focal points is finding a balance between diversity and convergence. In decomposition-based algorithms, the role of weight vectors is crucial. Despite numerous studies dedicated to these aspects, there is a scarcity of utilizing transfer learning algorithms for dual-operator feature fusion and employing neural networks for accurate partitioning of the objective space population. To address the aforementioned issues, this paper proposes the following improvements: (1) Implementing a Balanced Distribution Adaptation (BDA) transfer learning algorithm to achieve dual-operator feature fusion, resulting in a transfer population guiding the adaptive adjustment of weight vectors. (2) Integrating the BDA algorithm with multi-objective algorithms requires labeling the data, a challenge in the multi-objective evolutionary algorithm. To tackle this issue, non-dominated sorting is introduced as a bridge connecting the BDA and multi-objective evolutionary algorithms. This serves as a method to combine the advantages of decomposition-based and Pareto-dominance principle-based multi-objective algorithms. (3) To overcome the impact of traditional Euclidean distance on population sparsity, a neural network is employed to determine the population's distribution in the objective space accurately. This ensures the precise identification of individuals to be removed from the current population and the areas where additions are needed. In order to fully validate the effectiveness of the proposed algorithm, four different sets of experiments are conducted in the experimental section, where three sets of benchmarking problems are compared to a variety of algorithms that have received much attention in recent years, as well as ablation experiments.

Using Domain Adaptation and Inductive Transfer Learning to Improve Patient Outcome Prediction in the Intensive Care Unit: Retrospective Observational Study.

Accurate patient outcome prediction in the intensive care unit (ICU) can potentially lead to more effective and efficient patient care. Deep learning models are capable of learning from data to accurately predict patient outcomes, but they typically require large amounts of data and computational resources. Transfer learning (TL) can help in scenarios where data and computational resources are scarce by leveraging pretrained models. While TL has been widely used in medical imaging and natural language processing, it has been rare in electronic health record (EHR) analysis. Furthermore, domain adaptation (DA) has been the most common TL method in general, whereas inductive transfer learning (ITL) has been rare. To the best of our knowledge, DA and ITL have never been studied in-depth in the context of EHR-based ICU patient outcome prediction. This study investigated DA, as well as rarely researched ITL, in EHR-based ICU patient outcome prediction under simulated, varying levels of data scarcity. Two patient cohorts were used in this study: (1) eCritical, a multicenter ICU data from 55,689 unique admission records from 48,672 unique patients admitted to 15 medical-surgical ICUs in Alberta, Canada, between March 2013 and December 2019, and (2) Medical Information Mart for Intensive Care III, a single-center, publicly available ICU data set from Boston, Massachusetts, acquired between 2001 and 2012 containing 61,532 admission records from 46,476 patients. We compared DA and ITL models with baseline models (without TL) of fully connected neural networks, logistic regression, and lasso regression in the prediction of 30-day mortality, acute kidney injury, ICU length of stay, and hospital length of stay. Random subsets of training data, ranging from 1% to 75%, as well as the full data set, were used to compare the performances of DA and ITL with the baseline models at various levels of data scarcity. Overall, the ITL models outperformed the baseline models in 55 of 56 comparisons (all P values <.001). The DA models outperformed the baseline models in 45 of 56 comparisons (all P values <.001). ITL resulted in better performance than DA in terms of the number of times and the margin with which it outperformed the baseline models. In 11 of 16 cases (8 of 8 for ITL and 3 of 8 for DA), TL models outperformed baseline models when trained using 1% data subset. TL-based ICU patient outcome prediction models are useful in data-scarce scenarios. The results of this study can be used to estimate ICU outcome prediction performance at different levels of data scarcity, with and without TL. The publicly available pretrained models from this study can serve as building blocks in further research for the development and validation of models in other ICU cohorts and outcomes.

Adaptive iterative transfer learning for effective snapping shrimp sound detection.

This study aims to detect the bioacoustics signal in the underwater soundscape, specifically those produced by snapping shrimp, using adaptive iterative transfer learning. The proposed network is initially trained with pre-classified snapping shrimp sounds and Gaussian noise, then applied to classify and remove snapping-free noise from field data. This separated ambient noise is subsequently used for transfer learning. This process was iterated to distinguish more effectively between ambient noise and snapping shrimp sounds characteristics, resulting in improved classification. Through iterative transfer learning, significant improvements in precision and recall were observed. The application to field data confirmed that the trained network could detect signals that were difficult to identify using existing threshold classification methods. Furthermore, it was found that the rate of false detection decreased, and detection probability improved with each stage. This research demonstrates that incorporating the noise characteristics of field data into the trained network via iterative transfer learning can generate more realistic training data. The proposed network can successfully detect signals that are challenging to identify using existing threshold classification methods.

Machining quality prediction of multi-feature parts using integrated multi-source domain dynamic adaptive transfer learning

Machining quality prediction of multi-feature parts has been a challenging problem because of small dataset and inconsistent quality data distribution with respect to each machining feature. Transfer learning that leverages knowledge of one task and can be repurposed on another task seems a good solution for this purpose. However, traditional transfer learning typically has a single source domain and a target domain, which limits its applications in multi-source scenarios (e.g., multi-feature). To solve this issue, this paper proposes a novel integrated multi-source domain dynamic adaptive transfer learning (IMD-DATL) framework for machining quality prediction of multi-feature part machining systems. Specifically, a domain-sample similarity double matching multi-source domain integration method is designed to construct the integration knowledge transfer from multiple source domains to the target domain. A residual feature extraction network based on sample entropy-dynamic channel double-layer attention structure and a fine-grained transferable feature attention module are designed. These three attentions are used to improve the feature learning ability and the adaptation level to the predicted object in the three dimensions of sample, channel and data feature. Finally, multiple sets of comparative experiments in thin-walled part machining systems confirm the effectiveness and superiority of the proposed method for cross-domain quality prediction. Compared with other traditional transfer learning methods, the MAE, RMSE and Score on average of this method are increased by 5.47 %, 4.59 % and 4.84 %, respectively, compared with other multi-source domain adaptation methods, the MAE, RMSE and Score on average of this method are increased by 7.13 %, 7.37 % and 6.52 %, respectively.

A cross-dataset adaptive domain selection transfer learning framework for motor imagery-based brain-computer interfaces

Objective. In brain-computer interfaces (BCIs) that utilize motor imagery (MI), minimizing calibration time has become increasingly critical for real-world applications. Recently, transfer learning (TL) has been shown to effectively reduce the calibration time in MI-BCIs. However, variations in data distribution among subjects can significantly influence the performance of TL in MI-BCIs. Approach. We propose a cross-dataset adaptive domain selection transfer learning framework that integrates domain selection, data alignment, and an enhanced common spatial pattern (CSP) algorithm. Our approach uses a huge dataset of 109 subjects as the source domain. We begin by identifying non-BCI illiterate subjects from this huge dataset, then determine the source domain subjects most closely aligned with the target subjects using maximum mean discrepancy. After undergoing Euclidean alignment processing, features are extracted by multiple composite CSP. The final classification is carried out using the support vector machine. Main results. Our findings indicate that the proposed technique outperforms existing methods, achieving classification accuracies of 75.05% and 76.82% in two cross-dataset experiments, respectively. Significance. By reducing the need for extensive training data, yet maintaining high accuracy, our method optimizes the practical implementation of MI-BCIs.

Multi-source self-supervised domain adaptation network for VRLA battery anomaly detection of data center under non-ideal conditions

As the key component of uninterruptible power supply, the anomaly detection of valve-regulated lead-acid (VRLA) batteries is the reliable guarantee for the operation of internet data centers. Although deep neural networks (DNNs) have been investigated in battery anomaly detection, the key problem of feature learning under non-ideal conditions, i.e., data scarcity and class-imbalance, is not addressed very well. In this article, a novel DNN called multi-source self-supervised domain adaptation network (MSSSDAN) is proposed for VRLA battery anomaly detection of data center under non-ideal conditions. Firstly, multi-dimensional collaborative self-supervised learning (MDCSSL) is proposed to address the class-imbalanced problem by performing auxiliary feature learning task considering the temporal dependency of different channels. Secondly, a novel transfer learning (TL) framework, multi-source adaptive transfer learning (MSATL) is developed to solve the domain shift problem by adaptively minimizing the distribution discrepancy of different domains. Finally, the outperformance of MSSSDAN is verified on the VRLA battery dataset collected from Tencent data center. The testing results demonstrate that MSSSDAN can effectively solve the problem of battery anomaly detection under non-ideal conditions.

A novel adaptive deep transfer learning method towards thermal error modeling of electric spindles under variable conditions

Thermal error modeling (TEM) plays a vital role in maintaining the machining accuracy of electric spindles. Recently, deep learning (DL) techniques have obtained promising achievements in this area. However, DL techniques have certain limitations. The data acquired from variable working conditions present large distribution discrepancies, the DL-based model established on one working condition fails to obtain satisfactory prediction accuracy in another working condition. Moreover, existing studies focus solely on using temperature features to build prediction models, neglecting the full use of multi-sensory information. To address these issues, this paper proposes a novel adaptive deep transfer learning method towards TEM of electric spindles, which takes full advantage of the temperature, current, and power sensory information. Firstly, finite element simulation is employed to analyze the thermal characteristics of the electric spindle and determine the locations for temperature measurement points. Then, the convolutional long short-term memory network (C-LSTMN) is constructed, where the spatial features from multi-sensory information between inputs and prediction patterns are captured by convolutional layers, these features are further processed by long short-term memory network (LSTMN) to extract temporal features. Subsequently, the multi-kernel joint maximum mean discrepancy (MK-JMMD) measure is developed to minimize the distribution discrepancies between the source and target domains, thus the prediction model initially established on the source domain can be adaptive to effectively predict on the target domain. Finally, with unavailable thermal error data in the target domain, the proposed method is validated through 12 transfer tasks using datasets from four working conditions with other comparison methods. The results demonstrate that the proposed method overcomes the challenges of unavailable labeled thermal error samples and outperforms advanced methods.

Incorporating a-priori information in deep learning models for quantitative susceptibility mapping via adaptive convolution.

Quantitative susceptibility mapping (QSM) has attracted considerable interest for tissue characterization (e.g., iron and calcium accumulation, myelination, venous vasculature) in the human brain and relies on extensive data processing of gradient-echo MRI phase images. While deep learning-based field-to-susceptibility inversion has shown great potential, the acquisition parameters applied in clinical settings such as image resolution or image orientation with respect to the magnetic field have not been fully accounted for. Furthermore, the lack of comprehensive training data covering a wide range of acquisition parameters further limits the current QSM deep learning approaches. Here, we propose the integration of a priori information of imaging parameters into convolutional neural networks with our approach, adaptive convolution, that learns the mapping between the additional presented information (acquisition parameters) and the changes in the phase images associated with these varying acquisition parameters. By associating a-priori information with the network parameters itself, the optimal set of convolution weights is selected based on data-specific attributes, leading to generalizability towards changes in acquisition parameters. Moreover, we demonstrate the feasibility of pre-training on synthetic data and transfer learning to clinical brain data to achieve substantial improvements in the computation of susceptibility maps. The adaptive convolution 3D U-Net demonstrated generalizability in acquisition parameters on synthetic and in-vivo data and outperformed models lacking adaptive convolution or transfer learning. Further experiments demonstrate the impact of the side information on the adaptive model and assessed susceptibility map computation on simulated pathologic data sets and measured phase data.

Deep Transfer Learning Based Surrogate Modeling for Optimal Investment Decision of Distribution Networks

At present, the distribution network investment is faced with complicated operation strategies and various new investment elements. It's difficult to evaluate the impact of different investment measures on the final benefits due to the complicated physical modeling and potential coupling correlation, which contributes to that the traditional model-driven decision-making method may not work. This paper proposes a novel deep transfer learning based surrogate-assisted method to address the challenge. The key idea is to develop a surrogate model to replace the input-output correlation between the certain investment measure and its benefits, which can be further reformulated into mixed-integer linear constraints and embedded into the classical investment decision-making model for the target distribution network. As the model is trained through a designed two-stage adaptive transfer learning algorithm, which can extract the similar rules from other distribution networks, the dilemma of insufficient effective samples is alleviated, while a neural network constrained algorithm is designed to linear the surrogate model for simplifying the optimization. Numerical cases prove the effectiveness of the surrogate model on correlation mining and show the superiority of the surrogate-assisted method in achieving the precision investment for target distribution network.

State-of-Health Estimation of Li-Ion Batteries Using Semiparametric Adaptive Transfer Learning

For lithium-ion batteries (LIBs), precise state of health (SoH) estimation can provide direction for reasonable use, minimize battery failure rates, and extend battery life. Data-driven methods are promising for SoH estimation since they work effectively without requiring human interaction and have great nonlinear prediction capabilities. The majority of studies also found that the training data was sufficient. However, in real-world contexts, data collection is frequently time-consuming and expensive. This paper suggests using transfer learning to make the model less dependent on data for tracking battery state of health (SoH). Many approaches assume training and testing data have the same distribution. Due to distribution mismatch, a model that works for one data set may not for another. We propose an adaptive transfer learning technique relying on Gaussian Processes regression (AT-GPR) in this research. Automatically evaluating source-to-target task similarity can construct learning systems. Bayesian semi-parametric transfer kernel proposed to learn target task model. Battery cyclic aging data from two publicly available data sets in various working environments are considered to validate the suggested method. The experiments show that AT-GPR generates reliable prediction results; however, only 20% of the overall data set is made up of training data.

Deep transfer learning enables battery state of charge and state of health estimation

In the realm of lithium-ion battery state estimation, traditional data driven approaches face challenges in accurately estimating state of charge and state of health throughout the battery's life cycle under dynamic working condition, and there is still a lack of research on models that can fulfill these requirements simultaneously. To address these issues, this study proposes an adaptive convolutional gated recurrent unit with Kalman filter for state of charge estimation throughtout battery's full life cycle, leveraging transfer learning and deep learning techniques. Additionally, an adaptive convolutional gated recurrent unit with average post-processor is developed to estimate the battery state of health under dynamic working conditions, using voltage, current, temperature, state of charge, and accumulated discharge capacity as input features. Furthermore, a joint adaptive deep transfer learning model is proposed for simultaneously state of charge and state of health estimation through battery's full life cycle under dynamic working conditions. Experimental results validate the feasibility, accuracy, and robustness of the proposed models.

Fine classification of crops based on an inductive transfer learning method with compact polarimetric SAR images

ABSTRACT Compact polarimetric synthetic aperture radar (CP SAR) reduces fully polarimetric SAR system complexity and expands the imaging swath. Generally, fine classification of crop types relies on many labeled training samples. However, due to the temporal interval of crop phenology and ground environment variations over time, training samples from one dataset usually perform poorly for another. Therefore, in this study, transfer learning is introduced to crop classification to ensure classification accuracy by improving reusability of training samples. A stable and robust inductive transfer learning method, i.e. the Transfer Bagging-based Ensemble Learning (TBEL) algorithm, is proposed. The main idea is to select an adequate number of representative samples from unlabeled datasets to characterize each class in the target domain based on limited labeled samples and construct a classifier set to classify the target domain. This study investigates CP SAR data performance in transfer learning for crop classification. The proposed algorithm in the experimental study is compared with six typical methods (Subspace Alignment (SA), CORrelation ALignment (CORAL), Joint Distribution Adaptation (JDA), Balanced Distribution Adaptation (BDA), Transfer Bagging (TrBagg), and Bagging-based Ensemble Transfer Learning (BETL)). The experimental results show that the crop classification accuracy based on the TBEL algorithm is more stable, with an improved overall classification accuracy of 2–6%. Classifying the same rice harvest stage in the cross-year domain has the highest overall accuracy of 92.2%. Wheat fields in different scenes are also classified. Based on the TBEL algorithm, the overall classification accuracy improves by 1–10% compared with typical methods, with an accuracy of at least 87.6%. Furthermore, by testing the CP mode classification performance over various crops in transfer learning, we find that the circular CP mode performs better than the linear mode in most cases. This conclusion agrees with single-scene applications and was first verified in transfer learning.

Enhanced corn seed disease classification: leveraging MobileNetV2 with feature augmentation and transfer learning

In the era of advancing artificial intelligence (AI), its application in agriculture has become increasingly pivotal. This study explores the integration of AI for the discriminative classification of corn diseases, addressing the need for efficient agricultural practices. Leveraging a comprehensive dataset, the study encompasses 21,662 images categorized into four classes: Broken, Discolored, Silk cut, and Pure. The proposed model, an enhanced iteration of MobileNetV2, strategically incorporates additional layers—Average Pooling, Flatten, Dense, Dropout, and softmax—augmenting its feature extraction capabilities. Model tuning techniques, including data augmentation, adaptive learning rate, model checkpointing, dropout, and transfer learning, fortify the model's efficiency. Results showcase the proposed model's exceptional performance, achieving an accuracy of ~96% across the four classes. Precision, recall, and F1-score metrics underscore the model's proficiency, with precision values ranging from 0.949 to 0.975 and recall values from 0.957 to 0.963. In a comparative analysis with state-of-the-art (SOTA) models, the proposed model outshines counterparts in terms of precision, recall, F1-score, and accuracy. Notably, MobileNetV2, the base model for the proposed architecture, achieves the highest values, affirming its superiority in accurately classifying instances within the corn disease dataset. This study not only contributes to the growing body of AI applications in agriculture but also presents a novel and effective model for corn disease classification. The proposed model's robust performance, combined with its competitive edge against SOTA models, positions it as a promising solution for advancing precision agriculture and crop management.

Selective and Adaptive Incremental Transfer Learning with Multiple Datasets for Machine Fault Diagnosis

Selective and Adaptive Incremental Transfer Learning with Multiple Datasets for Machine Fault Diagnosis

Transfer Learning with Active Sampling for Rapid Training and Calibration in BCI-P300 Across Health States and Multi-centre Data.

Machine learning and deep learning advancements have boosted Brain-Computer Interface (BCI) performance, but their wide-scale applicability is limited due to factors like individual health, hardware variations, and cultural differences affecting neural data. Studies often focus on uniform single-site experiments in uniform settings, leading to high performance that may not translate well to real-world diversity. Deep learning models aim to enhance BCI classification accuracy, and transfer learning has been suggested to adapt models to individual neural patterns using a base model trained on others' data. This approach promises better generalizability and reduced overfitting, yet challenges remain in handling diverse and imbalanced datasets from different equipment, subjects, multiple centres in different countries, and both healthy and patient populations for effective model transfer and tuning. In a setting characterized by maximal heterogeneity, we proposed P300 wave detection in BCIs employing a convolutional neural network fitted with adaptive transfer learning based on Poison Sampling Disk (PDS) called Active Sampling (AS), which flexibly adjusts the transition from source data to the target domain. Our results reported for subject adaptive with 40% of adaptive fine-tuning that the averaged classification accuracy improved by 5.36% and standard deviation reduced by 12.22% using two distinct, internationally replicated datasets. These results outperformed in classification accuracy, computational time, and training efficiency, mainly due to the proposed Active Sampling (AS) method for transfer learning.

Research on the application of the Sleep EEG Net model based on domain adaptation transfer in the detection of driving fatigue

Fatigue detection in driving faces challenges stemming from data scarcity and difficulty in data acquisition, which poses a significant challenge to traditional fatigue detection methods. To address this issue, this study introduces a Sleep EEG Net model based on domain adaptation transfer learning. This model was pre-trained using the publicly available Sleep-EDF dataset, and domain adaptation transfer training techniques were employed to train the feature extractor of the pre-trained model, enabling cross-domain knowledge transfer. As a result, the model has been successfully applied to the task of fatigue detection in driving with only a limited amount of fatigue driving data. Experimental results demonstrate that this approach achieves a recognition accuracy of 91.5% in fatigue detection tasks. Furthermore, the model exhibits strong generalization capabilities and robustness, achieving high recognition accuracy in both simulated and real driving environments, thereby validating its effectiveness in practical applications.