Effect of error metrics on optimum weight factor selection for ensemble of metamodels

Abstract

Optimization of complex engineering systems is performed using computationally expensive high fidelity computer simulations (e.g., finite element analysis). During optimization these high-fidelity simulations are performed many times, so the computational cost becomes excessive. To alleviate the computational burden, metamodels are used to mimic the behavior of these computationally expensive simulations. The prediction capability of metamodeling can be improved by combining various types of models in the form of a weighted average ensemble. The contribution of each models is usually determined such that the root mean square cross validation error (RMSE-CV) is minimized in an aim to minimize the actual root mean square error (RMSE). However, for some applications, other error metrics such as the maximum absolute error (MAXE) may be the error metric of interest. It can be argued, intuitively, that when MAXE is more important than RMSE, the weight factors in ensemble should be determined by minimizing the maximum absolute cross validation error (MAXE-CV). Interestingly, it is found that the ensemble model based on MAXE-CV minimization is less accurate than the ensemble model based on RMSE-CV minimization even if the MAXE is the metric of interest. The reason is found to be that MAXE-CV is mostly related with the geography of the DOE rather than the prediction ability of metamodels.