On-line transfer learning for multi-fidelity data fusion with ensemble of deep neural networks

Abstract

Deep Neural Network (DNN) is widely used in engineering applications for its ability to handle problems with almost any nonlinearities. However, it is generally difficult to obtain sufficient high-fidelity (HF) sample points for expensive optimization tasks, which may affect the generalization performance of DNN and result in inaccurate predictions. To solve this problem and improve the prediction accuracy of DNN, this paper proposes an on-line transfer learning based multi-fidelity data fusion (OTL-MFDF) method including two parts. In the first part, the ensemble of DNNs is established. Firstly, a large number of low-fidelity sample points and a few HF sample points are generated, which are used as the source dataset and target dataset, respectively. Then, the Bayesian Optimization (BO) is utilized to obtain several groups of hyperparameters, based on which DNNs are pre-trained using the source dataset. Next, these pre-trained DNNs are re-trained by fine-tuning on the target dataset, and the ensemble of DNNs is established by assigning different weights to each pre-trained DNN. In the second part, the on-line learning system is developed for adaptive updating of the ensemble of DNNs. To evaluate the uncertainty error of the predicted values of DNN and determine the location of the updated HF sample point, the query-by-committee strategy based on the ensemble of DNNs is developed. The Covariance Matrix Adaptation Evolutionary Strategies is employed as the optimizer to find out the location where the maximal disagreement is achieved by the ensemble of DNNs. The design space is partitioned by the Voronoi diagram method, and then the selected point is moved to its nearest Voronoi cell boundary to avoid clustering between the updated point and the existing sample points. Three different types of test problems and an engineering example are adopted to illustrate the effectiveness of the OTL-MFDF method. Results verify the outstanding efficiency, global prediction accuracy and applicability of the OTL-MFDF method.