Types Of Surrogate Models Research Articles

Surrogate-assisted microscopic traffic simulation-based optimisation of routing parameters

Reactive and predictive routing algorithms have to work fast and reliably for a large number of traffic participants. Therefore, simple rules and thresholds guide the routing decisions rather than extensive data collection and machine learning. In this paper, we optimise some of the thresholds governing the behaviour of a reactive and predictive routing algorithm by using the microscopic traffic simulator TraffSim. Microscopic traffic simulation is more exact than its macroscopic counterpart and very well suited to test the efficiency of a reactive and predictive routing algorithm. Unfortunately, it is also tremendously more computationally expensive, impairing the applicability of 'conventional' heuristic optimisation techniques like genetic algorithms or evolution strategies. Extensive use of surrogate models in an optimisation procedure is a promising alternative. Several variations of the efficient global optimisation (EGO) algorithm are tested and compared. Furthermore, a new type of surrogate model geared towards the parameter optimisation is presented.

Data-driven uncertainty quantification for predictive flow and transport modeling using support vector machines

Specification of hydraulic conductivity as a model parameter in groundwater flow and transport equations is an essential step in predictive simulations. It is often infeasible in practice to characterize this model parameter at all points in space due to complex hydrogeological environments leading to significant parameter uncertainties. Quantifying these uncertainties requires the formulation and solution of an inverse problem using data corresponding to observable model responses. Several types of inverse problems may be formulated under various physical and statistical assumptions on the model parameters, model response, and the data. Solutions to most types of inverse problems require large numbers of model evaluations. In this study, we incorporate the use of surrogate models based on support vector machines to increase the number of samples used in approximating a solution to an inverse problem at a relatively low computational cost. To test the global capabilities of this type of surrogate model for quantifying uncertainties, we use a framework rooted in measure theory for constructing pullback and push-forward probability measures to study the data-to-parameter-to-prediction propagation of uncertainties under minimal statistical assumptions. Additionally, we demonstrate that it is possible to build a support vector machine using relatively low-dimensional representations of the hydraulic conductivity to propagate distributions. The numerical examples further demonstrate that we can make reliable probabilistic predictions of contaminant concentration at spatial locations.

Wind turbine site-specific load estimation using artificial neural networks calibrated by means of high-fidelity load simulations

Previous studies have suggested the use of reduced-order models calibrated by means of high-fidelity load simulations as means for computationally inexpensive wind turbine load assessments; the so far best performing surrogate modelling approach in terms of balance between accuracy and computational cost has been the polynomial chaos expansion (PCE). Regarding the growing interest in advanced machine learning applications, the potential of using Artificial Neural-Network (ANN) based surrogate models for improved simplified load assessment is investigated in this study. Different ANN model architectures have been evaluated and compared to other types of surrogate models (PCE and quadratic response surface). The results show that a feedforward neural network with two hidden layers and 11 neurons per layer, trained with the Levenberg Marquardt backpropagation algorithm is able to estimate blade root flapwise damage-equivalent loads (DEL) more accurately and faster than a PCE trained on the same data set. Further research will focus on further model improvements by applying different training techniques, as well as expanding the work with more load components.

Multiple Surrogate-Assisted Many-Objective Optimization for Computationally Expensive Engineering Design

Engineering design often involves problems with multiple conflicting performance criteria, commonly referred to as multi-objective optimization problems (MOP). MOPs are known to be particularly challenging if the number of objectives is more than three. This has motivated recent attempts to solve MOPs with more than three objectives, which are now more specifically referred to as “many-objective” optimization problems (MaOPs). Evolutionary algorithms (EAs) used to solve such problems require numerous design evaluations prior to convergence. This is not practical for engineering applications involving computationally expensive evaluations such as computational fluid dynamics and finite element analysis. While the use of surrogates has been commonly studied for single-objective optimization, there is scarce literature on its use for MOPs/MaOPs. This paper attempts to bridge this research gap by introducing a surrogate-assisted optimization algorithm for solving MOP/MaOP within a limited computing budget. The algorithm relies on principles of decomposition and adaptation of reference vectors for effective search. The flexibility of function representation is offered through the use of multiple types of surrogate models. Furthermore, to efficiently deal with constrained MaOPs, marginally infeasible solutions are promoted during initial phases of the search. The performance of the proposed algorithm is benchmarked with the state-of-the-art approaches using a range of problems with up to ten objective problems. Thereafter, a case study involving vehicle design is presented to demonstrate the utility of the approach.

A fast-prediction surrogate model for large datasets

Surrogate models approximate a function based on a set of training points and can then predict the function at new points. In engineering, kriging is widely used because it is fast to train and is generally more accurate than other types of surrogate models. However, the prediction time of kriging increases with the size of the dataset, and the training can fail if the dataset is too large or poorly spaced, which limits the accuracy that is attainable. We develop a new surrogate modeling technique—regularized minimal-energy tensor-product splines (RMTS)—that is not susceptible to training failure, and whose prediction time does not increase with the number of training points. The improved scalability with the number of training points is due to the use of tensor-product splines, where energy minimization is used to handle under-constrained problems in which there are more spline coefficients than training points. RMTS scales up to four dimensions with 10–15 spline coefficients per dimension, but scaling beyond that requires coarsening of the spline in some of the dimensions because of the computational cost of the energy minimization step. Benchmarking using a suite of one- to four-dimensional problems shows that while kriging is the most accurate option for a small number of training points, RMTS is the best alternative when a large set of data points is available or a low prediction time is desired. The best-case average root-mean-square error for the 4-D problems is close to 1% for RMTS and just under 10% for kriging.

Green’s function–based surrogate model for windfields using limited samples

There exists many applications for which wind-velocity is desired over a three-dimensional space. The vector field associated with these wind velocities is known as a “windfield” or “velocity-windfield.” The present work provides a fast method to characterize windfields. The approach uses the free-space Green’s function for potential theory as an inexpensive surrogate model in lieu of either complicated physics-based models or other types of surrogate models, both of which require volumetric discretizations for the three-dimensional case. Using the gradient of the third Green’s identity, the wind-velocity in the interior of a domain is entirely characterized by a surface discretization while still providing a three-dimensional model. The unknown densities on the surface are determined from enforcement of the interior form of the identity at arbitrary points coinciding with wind measurements taken by unmanned aerial vehicles. Numerical results support the feasibility of the method.

An optimization approach to reduce the risk of hydrate plugging during gas-dominated restart operations

In this paper, common meta-heuristic optimization techniques are applied for the potential application of reducing the risk of hydrate plugging in gas-dominated flowlines during restart operations. Based upon a hydrodynamic approach first shown in the experimental study of Leporcher et al. (2002), a transient, gas-dominated restart operation is emulated using Computational Fluid Dynamics (CFD) for a pipe section featuring a single low-spot filled with variable heights of free water. A two-phase CFD model has been constructed and validated using OpenFOAM® to simulate the flow. A modification has been made to the solver in order to capture the interfacial area between the gas and water phases at each time step, allowing an estimation of the amount of hydrate growth to be made during the transient and steady state phases of the restart operation. A genetic algorithm has then been used on different types of surrogate models with the goal of minimizing the restart gas velocity such that a defined plugging risk is minimized. Finally, an approach to apply this methodology to actual flowlines is discussed.

Multi-Objective Optimization With Multiple Spatially Distributed Surrogates

In engineering design optimization, evaluation of a single solution (design) often requires running one or more computationally expensive simulations. Surrogate assisted optimization (SAO) approaches have long been used for solving such problems, in which approximations/surrogates are used in lieu of computationally expensive simulations during the course of search. Existing SAO approaches often use the same type of approximation model to represent all objectives and constraints in all regions of the search space. The selection of a type of surrogate model over another is nontrivial and an a priori choice limits flexibility in representation. In this paper, we introduce a multi-objective evolutionary algorithm (EA) with multiple adaptive spatially distributed surrogates. Instead of a single global surrogate, local surrogates of multiple types are constructed in the neighborhood of each offspring solution and a multi-objective search is conducted using the best surrogate for each objective and constraint function. The proposed approach offers flexibility of representation by capitalizing on the benefits offered by various types of surrogates in different regions of the search space. The approach is also immune to illvalidation since approximated and truly evaluated solutions are not ranked together. The performance of the proposed surrogate assisted multi-objective algorithm (SAMO) is compared with baseline nondominated sorting genetic algorithm II (NSGA-II) and NSGA-II embedded with global and local surrogates of various types. The performance of the proposed approach is quantitatively assessed using several engineering design optimization problems. The numerical experiments demonstrate competence and consistency of SAMO.

Adaptive surrogate model-based fast path planning for spacecraft formation reconfiguration on libration point orbits

This work considers the optimization of fast path planning for spacecraft formation reconfiguration on libration point orbits. An adaptive surrogate model is proposed for solving the above optimization problem in abnormal conditions and/or emergency cases. First, a multi-layer optimization problem is established. In the outer loop of the multi-layer optimization, the nonlinear optimal control problem, taken as a dynamic optimization, is solved to obtain the optimal transfer paths of all spacecraft in the formation. In the inner loop of the multi-layer optimization, the parameter optimization problem, taken as a static optimization, is solved to obtain the states of all the spacecraft within the maximized area of the spacecraft formation. Then, an adaptive surrogate model is proposed. Because the multi-layer optimization problem is typically time-consuming and computationally expensive, the adaptive surrogate model is constructed using data drawn from the multi-layer optimization model and provides a fast approximation of the objective at new design points. Therefore, the adaptive surrogate model is feasible and cheap for spacecraft formation reconfiguration. Finally, numerical simulations show that the adaptive surrogate-based parameter optimization has advantages over the direct global optimization method. The accuracy, efficiency and robustness of the proposed adaptive surrogate model have also been compared between the different types of surrogate models.

Multifidelity importance sampling

Estimating statistics of model outputs with the Monte Carlo method often requires a large number of model evaluations. This leads to long runtimes if the model is expensive to evaluate. Importance sampling is one approach that can lead to a reduction in the number of model evaluations. Importance sampling uses a biasing distribution to sample the model more efficiently, but generating such a biasing distribution can be difficult and usually also requires model evaluations. A different strategy to speed up Monte Carlo sampling is to replace the computationally expensive high-fidelity model with a computationally cheap surrogate model; however, because the surrogate model outputs are only approximations of the high-fidelity model outputs, the estimate obtained using a surrogate model is in general biased with respect to the estimate obtained using the high-fidelity model. We introduce a multifidelity importance sampling (MFIS) method, which combines evaluations of both the high-fidelity and a surrogate model. It uses a surrogate model to facilitate the construction of the biasing distribution, but relies on a small number of evaluations of the high-fidelity model to derive an unbiased estimate of the statistics of interest. We prove that the MFIS estimate is unbiased even in the absence of accuracy guarantees on the surrogate model itself. The MFIS method can be used with any type of surrogate model, such as projection-based reduced-order models and data-fit models. Furthermore, the MFIS method is applicable to black-box models, i.e., where only inputs and the corresponding outputs of the high-fidelity and the surrogate model are available but not the details of the models themselves. We demonstrate on nonlinear and time-dependent problems that our MFIS method achieves speedups of up to several orders of magnitude compared to Monte Carlo with importance sampling that uses the high-fidelity model only.

Shape optimization of a cyclone separator using multi-objective surrogate-based optimization

Cyclones are one of the most widely used separators in many industrial applications. A low mass loading gas cyclone has two performance parameters, the Euler and Stokes numbers. These indices are highly sensitive to the geometrical design parameters which makes designing cyclones a challenging problem. This paper couples three surrogate models (Kriging, radial basis functions and support vector regression) with the efficient multi-objective optimization (EMO) algorithm to identify a Pareto front of cyclone designs with a minimal number of simulations. The EMO algorithm has been extended to select multiple samples per iteration (as opposed to one in the original formulation) and the ability to use an ensemble of surrogate models. The impact of using different surrogate model types is tested using well-known mathematical models of cyclone separators. The algorithm is applied to optimize the cyclone geometry, parametrized by seven design variables, and compared against the well-known NSGA-II algorithm. The results indicate that the Pareto set designs found using EMO outperform the designs found using NSGA-II while using significantly fewer function evaluations. This translates into substantial savings in time when computationally expensive CFD simulations are used.

Data-driven construction of Convex Region Surrogate models

With the increasing trend of solving more complex and integrated optimization problems, there is a need for developing process models that are sufficiently accurate as well as computationally efficient. In this work, we develop an algorithm for the data-driven construction of a type of surrogate model that can be formulated as a set of mixed-integer linear constraints, yet still provide good approximations of nonlinearities and nonconvexities. In such a surrogate model, which we refer to as Convex Region Surrogate (CRS), the feasible region is given by the union of convex regions in the form of polytopes, and for each region, the corresponding cost function can be approximated by a linear function. The general problem is as follows: given a set of data points in the parameter space and a scalar cost value associated with each data point, find a CRS model that approximates the feasible region and cost function indicated by the given data points. We present a two-phase algorithm to solve this problem and demonstrate its effectiveness with an extensive computational study as well as a real-world case study.

Advantages of surrogate models for architectural design optimization

AbstractClimate change, resource depletion, and worldwide urbanization feed the demand for more energy and resource-efficient buildings. Increasingly, architectural designers and consultants analyze building designs with easy-to-use simulation tools. To identify design alternatives with good performance, designers often turn to optimization methods. Randomized, metaheuristic methods such as genetic algorithms are popular in the architectural design field. However, are metaheuristics the best approach for architectural design problems that often are complex and ill defined? Metaheuristics may find solutions for well-defined problems, but they do not contribute to a better understanding of a complex design problem. This paper proposes surrogate-based optimization as a method that promotes understanding of the design problem. The surrogate method interpolates a mathematical model from data that relate design parameters to performance criteria. Designers can interact with this model to explore the approximate impact of changing design variables. We apply the radial basis function method, a specific type of surrogate model, to two architectural daylight optimization problems. These case studies, along with results from computational experiments, serve to discuss several advantages of surrogate models. First, surrogate models not only propose good solutions but also allow designers to address issues outside of the formulation of the optimization problem. Instead of accepting a solution presented by the optimization process, designers can improve their understanding of the design problem by interacting with the model. Second, a related advantage is that designers can quickly construct surrogate models from existing simulation results and other knowledge they might possess about the design problem. Designers can thus explore the impact of different evaluation criteria by constructing several models from the same set of data. They also can create models from approximate data and later refine them with more precise simulations. Third, surrogate-based methods typically find global optima orders of magnitude faster than genetic algorithms, especially when the evaluation of design variants requires time-intensive simulations.

Jet Pump Design Optimization by Multi-Surrogate Modeling

A basic approach to reduce the design and optimization time via surrogate modeling is to select a right type of surrogate model for a particular problem, where the model should have better accuracy and prediction capability. A multi-surrogate approach can protect a designer to select a wrong surrogate having high uncertainty in the optimal zone of the design space. Numerical analysis and optimization of a jet pump via multi-surrogate modeling have been reported in this work. Design variables including area ratio, mixing tube length to diameter ratio and setback ratio were introduced to increase the hydraulic efficiency of the jet pump. Reynolds-averaged Navier–Stokes equations were solved and responses were computed. Among different surrogate models, Sheppard function based surrogate shows better accuracy in data fitting while the radial basis neural network produced highest enhanced efficiency. The efficiency enhancement was due to the reduction of losses in the flow passage.

A multi-objective surrogate-based optimization of the crashworthiness of a hybrid impact absorber

This paper applies surrogate-based multi-objective optimization techniques to a crashworthiness problem in which the impact performance of a frontal crash absorber made of steel and a glass-fiber reinforced polyamide is optimized. Two well known crashworthiness indicators are considered as contrasting objective functions: the Specific Energy Absorption (SEA) and the Load Ratio (LR), whose responses are approximated by multiple types of surrogate models due to their computational cost and their noise levels. These models are quadratic and cubic polynomials, Gaussian process (kriging) and multivariate adaptive regression splines (MARS). The finite element model includes strain-rate sensitive properties, which is verified with experimental data from a drop-tower test. The thickness of the different parts, the geometry of the cross-section and the offsets of the reinforcement parts are chosen as design variables. Pareto solution is obtained after both models are verified. Results show improvements in both functions by almost 50% compared to the original design.

Using surrogate models for efficient optimization of simulated moving bed chromatography

A new approach of using computationally cheap surrogate models for efficient optimization of simulated moving bed (SMB) chromatography is presented. Two different types of surrogate models are developed to replace the detailed but expensive full-order SMB model for optimization purposes. The first type of surrogate is built through a coarse spatial discretization of the first-principles process model. The second one falls into the category of reduced-order modeling. The proper orthogonal decomposition (POD) method is employed to derive cost-efficient reduced-order models (ROMs) for the SMB process. The trust-region optimization framework is proposed to implement an efficient and reliable management of both types of surrogates. The framework restricts the amount of optimization performed with one surrogate and provides an adaptive model update mechanism during the course of optimization. The convergence to an optimum of the original optimization problem can be guaranteed with the help of this model management method. The potential of the new surrogate-based solution algorithm is evaluated by examining a separation problem characterized by nonlinear bi-Langmuir adsorption isotherms. By addressing the feed throughput maximization problem, the performance of each surrogate is compared to that of the standard full-order model based approach in terms of solution accuracy, CPU time and number of iterations. The quantitative results prove that the proposed scheme not only converges to the optimum obtained with the full-order system, but also provides significant computational advantages.

Improvements in surrogate models for process synthesis. Application to water network system design

High accuracy models can be obtained by using different types of surrogate models that accurately approximate equipment phenomenological models and can be used in synthesis problems, leading to faster and more precise solutions. Two types of surrogate models are used to approximate equipment phenomenological models: polynomial and neural network-based. In some cases, these surrogate models are not able to represent more complex equipment. An original methodology to reformulate these models using equations from shortcut equipment design is proposed. A medium-size case study involving fifteen units is presented. The synthesis problem is solved in a short computational time, leading many local solutions. Since several local optima objective function values are very close to each other, the choice of the best configuration among those found should be done qualitatively, because the differences among the objective function values are not significant if compared to the accuracy of equipment cost correlations in the literature.

Classification- and Regression-Assisted Differential Evolution for Computationally Expensive Problems

Differential Evolution (DE) has been well accepted as an effective evolutionary optimization technique. However, it usually involves a large number of fitness evaluations to obtain a satisfactory solution. This disadvantage severely restricts its application to computationally expensive problems, for which a single fitness evaluation can be highly time-consuming. In the past decade, a lot of investigations have been conducted to incorporate a surrogate model into an evolutionary algorithm (EA) to alleviate its computational burden in this scenario. However, only limited work was devoted to DE. More importantly, although various types of surrogate models, such as regression, ranking, and classification models, have been investigated separately, none of them consistently outperforms others. In this paper, we propose to construct a surrogate model by combining both regression and classification techniques. It is shown that due to the specific selection strategy of DE, a synergy can be established between these two types of models, and leads to a surrogate model that is more appropriate for DE. A novel surrogate model-assisted DE, named Classification- and Regression-Assisted DE (CRADE) is proposed on this basis. Experimental studies are carried out on a set of 16 benchmark functions, and CRADE has shown significant superiority over DE-assisted with only regression or classification models. Further comparison to three state-of-the-art DE variants, i.e., DE with global and local neighborhoods (DEGL), JADE, and composite DE (CoDE), also demonstrates the superiority of CRADE.

Mixture surrogate models based on Dempster-Shafer theory for global optimization problems

Recent research in algorithms for solving global optimization problems using response surface methodology has shown that it is in general not possible to use one surrogate model for solving different kinds of problems. In this paper the approach of applying Dempster-Shafer theory to surrogate model selection and their combination is described. Various conflict redistribution rules have been examined with respect to their influence on the results. Furthermore, the implications of the surrogate model type, i.e. using combined, single or a hybrid of both, have been studied. The suggested algorithms were applied to several well-known global optimization test problems. The results indicate that the used approach leads for all problems to a thorough exploration of the variable domain, i.e. the vicinities of global optima could be detected, and that the global minima could in most cases be approximated with high accuracy.

Evolutionary Model Type Selection for Global Surrogate Modeling

Due to the scale and computational complexity of currently used simulation codes, global surrogate (metamodels) models have become indispensable tools for exploring and understanding the design space. Due to their compact formulation they are cheap to evaluate and thus readily facilitate visualization, design space exploration, rapid prototyping, and sensitivity analysis. They can also be used as accurate building blocks in design packages or larger simulation environments. Consequently, there is great interest in techniques that facilitate the construction of such approximation models while minimizing the computational cost and maximizing model accuracy. Many surrogate model types exist (Support Vector Machines, Kriging, Neural Networks, etc.) but no type is optimal in all circumstances. Nor is there any hard theory available that can help make this choice. In this paper we present an automatic approach to the model type selection problem. We describe an adaptive global surrogate modeling environment with adaptive sampling, driven by speciated evolution. Different model types are evolved cooperatively using a Genetic Algorithm (heterogeneous evolution) and compete to approximate the iteratively selected data. In this way the optimal model type and complexity for a given data set or simulation code can be dynamically determined. Its utility and performance is demonstrated on a number of problems where it outperforms traditional sequential execution of each model type.