A gradient boosting approach with diversity promoting measures for the ensemble of surrogates in engineering

Abstract

Ensemble of surrogates has shown to be effective in the modeling of a variety of engineering problems, from aerospace to automotive and oil and gas industries, among many others. The ensemble performance though can be negatively affected by the lack of diversity (correlated errors) among the members of the ensemble, i.e., similar errors throughout the input space. During the last decade, ensemble research efforts in engineering have mostly focused on the so-called ensemble integration (combining prediction) phase with, for example, linear weighted averages of surrogate predictions based on performance (error) measures of both local and global natures. On the ensemble generation side (members of the ensemble), the emphasis has been on promoting diversity through the so-called heterogeneous ensembles, that is the use of different learners (surrogates), such as, linear regression, radial basis functions, kriging, support vector regression, and neural networks. Because of the lack of control on diversity in heterogeneous ensembles, potentially correlated error among the learners remains a critical issue. This paper presents an alternative approach based on homogeneous ensembles (single learning algorithm) that promotes diversity both implicitly and explicitly, by randomly sampling the training data and deterministically changing the data supplied to each surrogate model in the ensemble generation process. Specifically, it includes decision trees as learners, subsampling of the training set, and training on random selections of the input features, i.e., random subspace. The ensemble is sequentially constructed as an additive model where at each stage a simple surrogate is fit to the negative gradient of the loss function at the training data of the latest ensemble (gradient boosting). In addition, as a by-product of the modeling process, the proposed approach provides importance measures to preliminary rank input features. Statistically significant prediction improvements over heterogeneous ensembles were observed in a variety of well-known analytical test functions and industrial case studies in the areas of petroleum engineering-net present value of oil production in a mature reservoir and structural analysis-FSAE brake pedal von Mises stress and buckling load factor.