Ensemble Surrogate Models Research Articles

An ensemble optimizer with a stacking ensemble surrogate model for identification of groundwater contamination source

The application of the simulation-optimization method for groundwater contamination source identification (GCSI) encounters two main challenges: the substantial time cost of calling the simulation model, and the limitations on the accuracy of identification results due to the complexity, nonlinearity, and ill-posed nature of the inverse problem. To address these issues, we have innovatively developed an inversion framework based on ensemble learning strategies. This framework comprises a stacking ensemble model (SEM), which integrates three distinct machine learning models (Extremely Randomized Trees, Adaptive Boosting, and Bidirectional Gated Recurrent Unit), and an ensemble optimizer (E-GKSEEFO), which combines two newly proposed swarm intelligence optimizers (Genghis Khan Shark Optimizer and Electric Eel Foraging Optimizer). Specifically, the SEM serves as a surrogate model for the groundwater numerical simulation model. Compared to the original simulation model, it significantly reduces time cost while maintaining accuracy. The E-GKSEEFO, functioning as the search strategy for the optimization model, greatly enhances the accuracy of the optimization results. We have verified the performance of the SEM-E-GKSEEFO ensemble inversion framework through two hypothetical scenarios derived from an actual coal gangue pile. The results are as follows. (1) The SEM exhibits improved fitting performance compared to single machine learning models when dealing with high-dimensional nonlinear data from GCSI. (2) The E-GKSEEFO achieves significantly higher accuracy in the identification results of GCSI than individual optimizers. These findings affirm the effectiveness and superiority of the proposed SEM-E-GKSEEFO ensemble inversion framework.

Ensemble surrogate modeling of advective-dispersive transport with intraparticle diffusion model for column-leaching test

Column-leaching tests are a common approach for assessing the leaching behavior and resulting environmental risks of contaminated soils and waste materials, which are frequently reused for various construction purposes. The observed breakthrough curves of the contaminants are influenced by the complex dynamics of solute transport and kinetic inter-phase mass transfer. Disentangling these interactions necessitates numerical models. However, inverse modeling and sensitivity analysis can be time-consuming, especially when sorption kinetics are explicitly described by intraparticle diffusion, which requires discretizing the domain both in the flow direction along the column axis and inside the grains. To circumvent the need for such computationally intensive models, we have developed two different ensemble surrogate models. These models employ two separate ensemble methods: random forest stacking and inverse-distance weighted interpolation. Each method is applied to base surrogate models that cover different parts of the parameter space. The base surrogate models use the method of Extremely randomized Trees (ExtraTrees). The defined parameter range is based on the German standard for column-leaching tests. To optimize the base surrogate models, we utilized adaptive-sampling methods based on three distinct infill criteria: maximizing the expected improvement, staying within a certain Mahalanobis distance to the best estimate (both for exploitation), and maximizing the standard deviation (for exploration). The ensemble surrogate model demonstrates excellent performance in emulating the behavior of the original numerical model, with a relative root mean squared error of 0.09. We applied our proposed ensemble surrogate model to estimate the complete posterior parameter distribution using Simulation-Based Inference, specifically Neural Posterior Estimation, to determine the full parameter distribution conditioned on copper-leaching data from two different soils. Samples drawn from the posterior distribution align perfectly with the observed data for both the surrogate and original models.

Groundwater LNAPL Contamination Source Identification Based on Stacking Ensemble Surrogate Model

Groundwater LNAPL (Light Non-Aqueous Phase Liquid) contamination source identification (GLCSI) is essential for effective remediation and risk assessment. Addressing the GLCSI problem often involves numerous repetitive forward simulations, which are computationally expensive and time-consuming. Establishing a surrogate model for the simulation model is an effective way to overcome this challenge. However, how to obtain high-quality samples for training the surrogate model and which method should be used to develop the surrogate model with higher accuracy remain important questions to explore. To this end, this paper innovatively adopted the quasi-Monte Carlo (QMC) method to sample from the prior space of unknown variables. Then, this paper established a variety of individual machine learning surrogate models, respectively, and screened three with higher training accuracy among them as the base-learning models (BLMs). The Stacking ensemble framework was utilized to integrate the three BLMs to establish the ensemble surrogate model for the groundwater LNAPL multiphase flow numerical simulation model. Finally, a hypothetical case of groundwater LNAPL contamination was designed. After evaluating the accuracy of the Stacking ensemble surrogate model, the differential evolution Markov chain (DE-MC) algorithm was applied to jointly identify information on groundwater LNAPL contamination source and key hydrogeological parameters. The results of this study demonstrated the following: (1) Employing the QMC method to sample from the prior space resulted in more uniformly distributed and representative samples, which improved the quality of the training data. (2) The developed Stacking ensemble surrogate model had a higher accuracy than any individual surrogate model, with an average R2 of 0.995, and reduced the computational burden by 99.56% compared to the inversion process based on the simulation model. (3) The application of the DE-MC algorithm effectively solved the GLCSI problem, and the mean relative error of the identification results of unknown variables was less than 5%.

Application of a Dirichlet Distribution-Based Ensemble Surrogate Model in Aerodynamic Optimization

Surrogate models have been widely applied in the aerodynamic optimization of aircrafts, whereas the traditional individual surrogate models have the defects of low robustness and applicability. In this study, a novel ensemble surrogate model is proposed and applied in the multi-objective optimization of the airfoil. The backpropagation neural network, deep belief network, and kriging surrogate models are selected as the member surrogate models, and the Dirichlet distribution strategy is introduced to adaptively generate the weights of the member surrogate models in constructing the ensemble surrogate model. An improved multi-objective particle swarm optimization (MOPSO) framework is established by employing the α-stable distribution function to enhance the global convergence rate of the algorithm. Based on the improved MOPSO framework in which the ensemble surrogate model is embedded, the multi-objective optimization of the airfoil is conducted. The results indicate that the proposed ensemble-surrogate-model-based optimization obtains better aerodynamic performance of the airfoil under multiple operating conditions, compared to the individual-surrogate-model-based optimization.

Improving query efficiency of black-box attacks via the preference of deep learning models

Black-box query attacks are effective at compromising deep-learning models using only the model's output. These attacks typically face challenges with low attack success rates (ASRs) when limited to fewer than ten queries per example. Recent approaches have improved ASRs due to the transferability of initial perturbations, yet they still suffer from inefficient querying. Our study introduces the Gradient-Aligned Attack (GAA) to enhance ASRs with minimal perturbation by focusing on the model's preference. We define a preference property where the generated adversarial example prefers to be misclassified as the wrong category with a high initial confidence. This property is further elucidated by the gradient preference, suggesting a positive correlation between the magnitude of a coefficient in a partial derivative and the norm of the derivative itself. Utilizing this, we devise the gradient-aligned CE (GACE) loss to precisely estimate gradients by aligning these coefficients between the surrogate and victim models, with coefficients assessed by the victim model's outputs. GAA, based on the GACE loss, also aims to achieve the smallest perturbation. Our tests on ImageNet, CIFAR10, and Imagga API show that GAA can increase ASRs by 25.7% and 40.3% for untargeted and targeted attacks respectively, while only needing minimally disruptive perturbations. Furthermore, the GACE loss reduces the number of necessary queries by up to 2.5x and enhances the transferability of advanced attacks by up to 14.2%, especially when using an ensemble surrogate model. Code is available at https://github.com/HaloMoto/GradientAlignedAttack.

Optimizing hydrogen yield in sorption-enhanced steam methane reforming: A novel framework integrating chemical reaction model, ensemble learning method, and whale optimization algorithm

High hydrogen production is essential in sorption-enhanced steam methane reforming (SE-SMR). This study presents an innovative optimization framework to optimize the H2 yield of SE-SMR through a coupled reaction model, prediction surrogate model, and whale optimization algorithm. Firstly, a computational fluid dynamics chemical reaction model of SE-SMR is constructed. A database is established to consider the H2 yield of SE-SMR with temperature, pressure, velocity, and steam to carbon ratio. Then, a surrogate model is developed on the basis of eXtreme Gradient Boosting (XGBoost) to predict the H2 yield under various operating parameters. Finally, the surrogate model is exploited to calculate the fitness function of the whale optimization algorithm to construct an optimization framework for maximizing the H2 yield. The results show that the XGBoost ensemble surrogate model can predict the H2 yield of SE-SMR within 4 s with high accuracy. The optimization framework can optimize the maximum H2 yield and the corresponding operating parameters within 1 min. The optimization results are validated by applying the model and the percentage error of only 0.79 %. Moreover, under optimal operating conditions, methane is almost completely converted to hydrogen, and the dry hydrogen mole fraction could reach 95.96 %. The presented optimization framework can guide the prediction of H2 yield and the optimization of operating parameters in SE-SMR.

Ensemble learning based hierarchical surrogate model for multi-fidelity information fusion

Recently, multi-fidelity information fusion based surrogate modeling methods have made great progress in the engineering design and optimization tasks. Two main issues in this field are: (1) Since the overarching trend of the responses from the high-fidelity (HF) model is captured by the low-fidelity (LF) model, inaccurate LF model may negatively impact the final modeling results. (2) The responsiveness of multi-fidelity surrogate (MFS) models to variations in the relationship between LF and HF models leads to limited prediction performance. To this end, we propose an ensemble learning based MFS modeling method with a hierarchical framework, called EL-MFS. Specifically, to alleviate the impact of issue (1), we present an adaptive ensemble surrogate model, which aims to effectively mitigate the negative impact of inappropriate LF model selection on the HF approximation results. Furthermore, we propose to exploit the feature mapping and hierarchical framework to boost the versatility of the model as a way to mitigate the dependence of MFS performance on the relationship between HF and LF models. To assess the effectiveness of the proposed model, a sequence of numerical problems is tested, and some advanced surrogates are selected as the baseline models. Moreover, to demonstrate the potential of the proposed model in aiding intricate engineering design, one engineering case is investigated as well.

A robust ensemble surrogate model and its application in the in situ data modelling and analysis of a combine harvester

In this article, a robust ensemble model is proposed based on extended adaptive hybrid functions and fuzzy clustering. In the outlier detection stage, each sample is assigned memberships to judge whether it is an outlier or not, where the memberships are determined based on the responses of the ensemble surrogate model of each cluster. Then, the detected outliers are removed from the initial training samples, and the final prediction model is constructed based on the remaining normal samples. The results of numerical problems and the in-situ dataset from a combine harvester show that the proposed model can provide accurate detection results for outliers and accurate prediction results for new points. The sensitivity analysis based on the proposed robust ensemble model indicates that the angle of guide plate, the open rate of cleaning fan, and the height of header have a greater effect on the cleaning loss of combine harvester.

A Data Stream Ensemble Assisted Multifactorial Evolutionary Algorithm for Offline Data-Driven Dynamic Optimization.

Existing work on offline data-driven optimization mainly focuses on problems in static environments, and little attention has been paid to problems in dynamic environments. Offline data-driven optimization in dynamic environments is a challenging problem because the distribution of collected data varies over time, requiring surrogate models and optimal solutions tracking with time. This paper proposes a knowledge-transfer-based data-driven optimization algorithm to address these issues. First, an ensemble learning method is adopted to train surrogate models to leverage the knowledge of data in historical environments as well as adapt to new environments. Specifically, given data in a new environment, a model is constructed with the new data, and the preserved models of historical environments are further trained with the new data. Then, these models are considered to be base learners and combined as an ensemble surrogate model. After that, all base learners and the ensemble surrogate model are simultaneously optimized in a multitask environment for finding optimal solutions for real fitness functions. In this way, the optimization tasks in the previous environments can be used to accelerate the tracking of the optimum in the current environment. Since the ensemble model is the most accurate surrogate, we assign more individuals to the ensemble surrogate than its base learners. Empirical results on six dynamic optimization benchmark problems demonstrate the effectiveness of the proposed algorithm compared with four state-of-the-art offline data-driven optimization algorithms. Code is available at https://github.com/Peacefulyang/DSE_MFS.git.

Maximizing power density in proton exchange membrane fuel cells: An integrated optimization framework coupling multi-physics structure models, machine learning, and improved gray wolf optimizer

This study proposes an innovative optimization framework to optimize channel structure and maximize power density by coupling multi-physics structure models, machine learning, and swarm intelligence algorithms. First, proton exchange membrane fuel cells (PEMFC) imitated water-drop block channels are employed for constructing multi-physics structure models. A database of the PEMFC output performance under various structural parameters of imitated water-drop block is established. Then, a machine-learning-based surrogate model is constructed based on the adaptive boosting (AdaBoost) ensemble algorithm to predict the output performance under different channel parameters. Finally, the improved gray wolf optimizer (IGWO) fitness function is calculated using a surrogate model to establish an optimization framework for effectively optimizing the channel structure. Results show that the AdaBoost ensemble surrogate model predicts the PEMFC polarization curves with extremely high accuracy and efficiency within one second. The optimization framework is capable of swiftly predicting both the optimal channel structure and maximum power density in under two minutes. The predicted values are returned to the physical model for validation with an error of only 3.96%. Simultaneously, the optimal channel structure can effectively enhance the PEMFC performance. The proposed optimization framework will guide the performance prediction and channel structure multivariate optimization.

A novel adaptive-weight ensemble surrogate model base on distance and mixture error.

Surrogate models are commonly used as a substitute for the computation-intensive simulations in design optimization. However, building a high-accuracy surrogate model with limited samples remains a challenging task. In this paper, a novel adaptive-weight ensemble surrogate modeling method is proposed to address this challenge. Instead of using a single error metric, the proposed method takes into account the position of the prediction sample, the mixture error metric and the learning characteristics of the component surrogate models. The effectiveness of proposed ensemble models are tested on five highly nonlinear benchmark functions and a finite element model for the analysis of the frequency response of an automotive exhaust pipe. Comparative results demonstrate the effectiveness and promising potential of proposed method in achieving higher accuracy.

Adaptive stochastic configuration network ensemble for structural reliability analysis

This paper develops a new adaptive surrogate model method for structural reliability analysis, which is constructed by approximating the limit state function (LSF) using stochastic configuration network (SCN) ensemble model. The proposed ensemble model integrates three SCN base models and sets weights for each prediction result, reducing outliers and greatly improving the output accuracy and stability of the algorithm. In constructing a surrogate model for reliability analysis, a new adaptive sampling strategy is designed, including a novel learning function. The new strategy enables the most informative sample points are selected and significantly reduce the number of required sample point. Without further calls to the LSF, the structural failure probability is quickly evaluated using Monte Carlo simulation in terms of the constructed SCN ensemble surrogate model. Three benchmark problems, including a highly nonlinear problem with small failure probability and two multi-variable engineering structural problems, are used to verify the effectiveness of the proposed method.

A Pointwise-Optimal Ensemble of Surrogate Models

AbstractThe ensemble of surrogate models is commonly used to replace computationally expensive simulations due to their superior prediction accuracy and robustness compared to individual models. This paper proposes a new pointwise ensemble of surrogate models, namely, a pointwise-optimal ensemble of surrogate models (POEMs). To address the limitations of the cross-validation (CV) error in evaluating the performance of regression surrogate models, this paper introduces the compensated cross-validation error, which is more reliable in selecting better individual surrogate models and improving the accuracy of surrogate model ensembles. To overcome the limitations of CV error in calculating pointwise weight factors, this paper designs and solves an optimization problem at training points to obtain corresponding pointwise weight factors. Additionally, this paper proposes two weight calculation methods to be applied in the interpolation and extrapolation regions, respectively, to reduce the instability of ensembles caused by extrapolation. Thirty test functions are employed to investigate the appropriate hyperparameters of POEM and the Friedman test is used to verify the rationality of the α value. The thirty test functions are also used to examine the performance of POEM and compare it with state-of-the-art ensemble surrogate models. Furthermore, POEM is applied to a large-aperture mirror holder optimization case to verify its superiority. The results demonstrate that POEM presents better accuracy and robustness than individual surrogates and other compared ensembles of surrogate models.

Structural optimization design of metal rubber isolator based on an ensemble surrogate model

The finite element simulation design of products depends on the computer’s processing power; hence an adequate model with acceptable accuracy can significantly reduce the calculation cost. The optimization method based on the surrogate model is a general method to solve this problem. Among them, the sampling point selection of the surrogate model plays a crucial role in the calculation efficiency and accuracy. Therefore, according to the applicable standards of metal rubber isolators, this paper proposes an improved sampling method based on the surrogate model method and then optimizes the structure of metal rubber isolators. The metal rubber isolator's performance and lightweight design requirements are considered based on satisfying the precision and boundary conditions.

Adaptive ensemble surrogate-based optimization and analysis of forklift pallet racks

As an important part of the lifting platform of pallet forklift trucks, how to reduce the deformation of the pallet rack under working conditions while reducing the mass to ensure the safety performance of forklift trucks is the most concerning issue in the design of forklift truck structure. The pallet rack structure is complex, and optimizing simulation using traditional high-precision simulation models takes much time and effort. Therefore, this paper takes the lifting platform of an unmanned AVG forklift truck as the research object and establishes a parametric model of the pallet rack using the 3D modelling software SolidWorks and the finite element analysis software ANSYS to carry out static analysis of it. Optimization design variables are selected, a single surrogate model and ensemble surrogate model are chosen for various surrogate model techniques, a small number of sample points are used to construct a low-precision model, and adaptive infilling technology is used to improve the model accuracy, and the structure is optimized, and the results show that, while the pallet rack structure meets the requirements of light weight and strength, the mass is reduced by 1.2%, and the morphology is reduced by 17.2%. Moreover, a global sensitivity analysis of each design parameter was carried out under the guidance of the surrogate model, highlighting the most influential design variable as the height of the channel steel and establishing the design variables that should be taken into account in the structural design. This paper compares the performance of the mainstream single-surrogate model and ensemble-surrogate model as well as the adaptive infilling strategy in engineering design and proves that the surrogate model optimization method has a certain guiding significance for the structural optimization design of pallet racking.

Identification of light nonaqueous phase liquid groundwater contamination source based on empirical mode decomposition and deep learning.

The simulation optimization method was used to the identification of light nonaqueous phase liquid (LNAPL) groundwater contamination source (GCS) with the help of a hypothetical case in this study. When applying the simulation optimization method to identify GCS, it was a common technical means to establish surrogate model for the simulation model to participate in the iterative calculation to reduce the calculation load and calculation time. However, it was difficult for a single modeling method to establish surrogate model with high accuracy for the LNAPL contamination multiphase flow simulation model (MFSM). To give full play to advantages of single surrogate model and improve the accuracy of the surrogate model to the MFSM, a combination of deep belief neural network (DBNN) and long short-term memory (LSTM) neural network was used to establish artificial intelligence ensemble surrogate model (AIESM) for the MFSM. At the same time, to reduce the influence of noise in observed concentrations on the accuracy of the identification results, empirical mode decomposition (EMD) and wavelet analysis methods were used to denoise the observed concentrations, and their noise reduction effects were compared. The observed concentrations with better noise reduction effect and the observed concentrations without denoising were used to construct the objective function, and constraints of the optimization model were determined meanwhile. Then, the objective function and the constraints were integrated to build the optimization model to identify GCS and simulation model parameters. Applying the AIESM instead of the MFSM to embed in the optimization model and participate in the iterative calculation. Finally, the genetic algorithm (GA) was used to solve the optimization model to obtain the identification results of GCS and simulation model parameters. The results showed that compared with the single DBNN and LSTM surrogate models, AIESM obtained the highest accuracy and could replace the MFSM to participate in the iterative calculation, thereby reducing the calculation load and calculation time by more than 99%. Comparing with the wavelet analysis, EMD could reduce the noise in the concentrations more effectively, improved the accuracy of the approximated concentrations to the actual values, and increased the accuracy of the GCSs identification results by 1.45%.

A Scalable Digital Twin Framework Based on a Novel Adaptive Ensemble Surrogate Model

AbstractThe concept of digital twins is to have a digital model that can replicate the behavior of a physical asset in real time. However, using digital models to reflect the structural performance of physical assets usually faces high computational costs, which makes it difficult for the model to satisfy real-time requirements. As a technique to replace expensive simulations, surrogate models have great potential to solve this problem. In practice, however, the optimal individual surrogate model (ISM) applicable to a given problem usually changes as factors change, and this can be mitigated by integrating multiple ISMs. Therefore, this paper proposes a scalable digital twin framework based on a novel adaptive ensemble surrogate model. This ensemble not only provides robust approximation but also reduces the additional cost brought by the ensemble by reducing the number of ISMs participating in the ensemble through multicriterion model screening. Moreover, based on the characteristics of the finite element method, a node rearrangement method, which provides scalability for the construction of a digital model, is proposed. That is, the distribution and number of nodes can be customized to not only decrease the computational cost by reducing nodes but also obtain the information at key positions by customizing the locations of nodes. Numerical experiments are employed to verify the performance of the proposed ensemble and node rearrangement method. A telehandler is used as an example to build a scalable digital twin, which proves the feasibility and effectiveness of the framework.

Simultaneous identification of the number, location and release intensity of groundwater contamination sources based on simulation optimization and ensemble surrogate model

Abstract In most instances, the number, location and release intensity of groundwater contamination sources (GCSs) are all unknown. The 0-1 mixed integer nonlinear programming optimization model (0-1 MINPOM) used previously could only identify the location and release intensity for GCSs. Thus a 0-1 MINPOM was improved and applied to simultaneously identify the number, location and release history of GCSs. In addition, to reduce the time of calling the simulation model for iterative calculation, the deep belief neural network (DBNN), long short-term memory network (LSTM) and Kriging method were applied to establish an ensemble surrogate model of the simulation model to replace the simulation model for iterative calculation. The results show that compared with the three single surrogate models, the ensemble surrogate model has the highest approximation accuracy to the simulation model, and could save 99% of the calculation time when iterative calculation was performed in place of the simulation model. The improved 0-1 MINPOM could identify the number, location, and release history of GCSs simultaneously. The accuracy of identifying the number of GCSs was 100%, and the accuracies of identifying the locations and release history were above 95.12% and 83.51%, respectively.

A Dimension Reduction-Based Multidisciplinary Design Optimization Method for High Pressure Turbine Blades

Abstract The complex aero-thermal coupling between fluid and solid regions within high-pressure turbines makes it important to perform multidisciplinary design optimization of high-pressure turbine blades. However, most published works failed to consider the correlations between blade profiles and cooling structures that could best compromise the aerodynamic and thermal performance of high-pressure turbine blades, and the related optimization problems were so far limited to single- or bi-objective ones. The critical drawbacks of these available studies are mainly due to the reduced accuracies of the adopted methods when dealing with large numbers of design variables and objectives. To tackle these difficulties, a dimension reduction-based multidisciplinary design optimization method is proposed and validated through an aero-thermal design optimization of the NASA-C3X vane with a total of 39 design variables and five performance objectives. The main novelties of this proposed method lie in a hybrid dimension reduction of design space by means of the proper orthogonal decomposition and global sensitivity analysis methods, as well as the integration of the ensemble surrogate model and the reference vector evolutionary algorithm for optimal solutions. The results show that the prediction accuracy of the ensemble surrogate model for each performance objective is enhanced, even though the dimensionalities of design space are reduced. Complicated compromises exist among the five performance objectives under consideration. For NASA-C3X vane, the optimal design helps reduce irreversible flow losses especially wake losses while reducing the volumes with high-temperature and high-temperature gradient near the trailing edge is mainly responsible for the reduced irreversible losses due to heat transfer. The outcomes of this work are particularly relevant for the advanced design optimization methods for high pressure turbines.

Design optimization of a nuclear main steam safety valve based on an E-AHF ensemble surrogate model

Main steam safety valves are commonly used in nuclear power plants to provide final protections from overpressure events. Blowdown and dynamic stability are two critical characteristics of safety valves. However, due to the parameter sensitivity and multi-parameter features of safety valves, using traditional method to design and/or optimize them is generally difficult and/or inefficient. To overcome these problems, a surrogate model-based valve design optimization is carried out in this study, of particular interest are methods of valve surrogate modeling, valve parameters global sensitivity analysis and valve performance optimization. To construct the surrogate model, Design of Experiments (DoE) and Computational Fluid Dynamics (CFD) simulations of the safety valve were performed successively, thereby an ensemble surrogate model (E-AHF) was built for valve blowdown and stability predictions. With the developed E-AHF model, global sensitivity analysis (GSA) on the valve parameters was performed, thereby five primary parameters that affect valve performance were identified. Finally, the k-sigma method is used to conduct the robust optimization on the valve. After optimization, the valve remains stable, the minimum blowdown of the safety valve is reduced greatly from 13.30% to 2.70%, and the corresponding variance is reduced from 1.04 to 0.65 as well, confirming the feasibility and effectiveness of the optimization method proposed in this paper.