Expensive Function Evaluations Research Articles

A Systematic Approach for Estimating Colloidal Particle Adsorption Model Parameters

The estimation of ion- exchange chromatography model parameters is crucial to enable efficient model-assisted biopharmaceutical downstream process development. Model calibration methods can be hindered by model limitations combined with parameter correlations, leading to time-consuming repeated parameter estimations. While Steric Mass Action isotherm estimation methods exist, there is a need for a systematic approach to estimate model parameters for an emerging Colloidal Particle Adsorption (CPA) model proposed by Briskot et al. This study presents a novel strategy that addresses this challenge, offering significant improvements.Through a parameter sensitivity analysis, we identified key levers for improved CPA parameter estimation, enabling the prediction of elution behavior for low and high load densities in gradient and step elution mode. This analysis also revealed the correlation structure of parameters, allowing the establishment of a minimalized experimental data set for parameter estimation, by using one breakthrough, a high load and three low load density gradient elution experiments. Our workflow leverages a surrogate-assisted global-optimization tool, minimizing computationally expensive function evaluations during parameter fitting. Furthermore, we employed a customized objective function, specifically adapted to the model structure and sensitivity results, to enhance the solver's performance.Our strategy was tested on three model proteins with molecular weights of approximately 50, 150 and 200 kDa using a strong cation exchange Poros 50 HS resin. Our final approach enabled high throughput CPA model calibration for single components. The resulting CPA models were able to describe non-binding protein-pulses, low and high-loaded gradient elution, break through, as well as isocratic elution experiments.

Formulating approximation error as noise in surrogate-assisted multi-objective evolutionary algorithm

Many real multi-objective optimization problems with 20-50 decision variables often have only a small number of function evaluations available, because of their heavy time/money burden. Therefore, surrogate models are often utilized as alternatives for expensive function evaluations. However, the approximation error of the surrogate model is inevitable compared to the real function evaluation. The approximation error has a similar impact on the algorithm as noise, i.e., different optimization stages suffer from various impacts. Therefore, the current optimization stage can be indirectly detected via measuring the impact of the noise formulated by the approximation error. In addition, the rising dimension of the search space leads to an increase in the approximation errors of the surrogate models, which poses a huge challenge for existing surrogate-assisted multi-objective evolutionary algorithms. In this work, we propose a stage-adaptive surrogate-assisted multi-objective evolutionary algorithm to solve the medium-scale optimization problems. In the proposed algorithm, the ensemble model consisting of the latest and historical models is used as the surrogate model, on the basis of which a set of potential candidates can be discovered. Then, a stage-adaptive infill sampling strategy selects the most suitable sampling strategy by analyzing the demand of the current optimization stage on convergence, diversity, model accuracy to sample from the candidates. As for the current optimization stage, it is detected by a noise impact indicator, where the approximation errors of surrogate models are formulated as noise. The experimental results on a series of medium-scale expensive test problems demonstrate the superiority of the proposed algorithm over six state-of-the-art compared algorithms.

Surrogate-assisted evolutionary framework with an ensemble of teaching-learning and differential evolution for expensive optimization

Surrogate-Assisted Evolutionary Algorithms (SAEAs) integrate Evolutionary Algorithms (EAs) with surrogate models to reduce the actual number of expensive function evaluations and have been widely used in solving Expensive Optimization Problems (EOPs). However, because of the insufficient diversity of new solutions, SAEAs often fall into local optima and result in evolutionary stagnation when dealing with complex problems. To improve the diversity of new solutions, this paper introduces a Surrogate-Assisted evolutionary Framework with an ensemble of Teaching-learning and Differential evolution (SAF-TD), which successfully integrates the strengths of both EAs to produce more diverse and effective solutions. The main contributions are summarized as follows: First, a novel framework that effectively integrates two distinct EAs using a radial basis function surrogate model and strategic sampling is introduced. Second, a mutation strategy with a supervisory mechanism is proposed to enhance mutation effectiveness and avoid stagnation. Third, a volatility index derived from dimensional improvements is incorporated into parameter control to address fitness-dependent weaknesses. Experiments on five expensive optimization functions and the CEC2017 test suite across multiple dimensions were conducted to evaluate the performance of SAF-TD. The results demonstrate that SAF-TD is highly competitive compared with state-of-the-art SAEAs and exhibits excellent performance in solving EOPs.

A performance indicator-based evolutionary algorithm for expensive high-dimensional multi-/many-objective optimization

Surrogate-assisted multi-objective evolutionary algorithms have shown considerable potential for solving optimization problems in which only a small number of expensive function evaluations are available. However, most existing research remains restricted to low-/medium-dimensional problems, with very little attention paid to addressing problems involving decision variables with more than 100 dimensions. In this study, a performance indicator-based evolutionary algorithm (PIEA) is proposed for expensive high-dimensional multi-/many-objective optimization. A surrogate model is employed to approximate the performance indicator rather than directly predicting the objective function, thus simplifying the optimization complexity and mitigating the impact of cumulative errors. An efficient indicator-based optimization strategy emphasising the balance between exploration and exploitation is designed for surrogate-assisted evolution and infill sampling. A history-based selection strategy is implemented to select a suitable indicator from the preset pool for each optimization cycle. An empirical study was conducted on two well-known benchmark suites, and the results demonstrate the superiority of the proposed algorithm over several state-of-the-art algorithms. Moreover, we integrate this concept into a classification-based framework, which further verifies its effectiveness.

A surrogate-assisted a priori multiobjective evolutionary algorithm for constrained multiobjective optimization problems

We consider multiobjective optimization problems with at least one computationally expensive constraint function and propose a novel surrogate-assisted evolutionary algorithm that can incorporate preference information given a priori. We employ Kriging models to approximate expensive objective and constraint functions, enabling us to introduce a new selection strategy that emphasizes the generation of feasible solutions throughout the optimization process. In our innovative model management, we perform expensive function evaluations to identify feasible solutions that best reflect the decision maker’s preferences provided before the process. To assess the performance of our proposed algorithm, we utilize two distinct parameterless performance indicators and compare them against existing algorithms from the literature using various real-world engineering and benchmark problems. Furthermore, we assemble new algorithms to analyze the effects of the selection strategy and the model management on the performance of the proposed algorithm. The results show that in most cases, our algorithm has a better performance than the assembled algorithms, especially when there is a restricted budget for expensive function evaluations.

A trust-region-like algorithm for expensive multi-objective optimization

In solving expensive multi-objective optimization problems, surrogate models have been widely investigated. The existing multi-objective algorithms adopting surrogate models can be classified into two categories: surrogate-assisted evolutionary algorithms (SAEAs) and surrogate-based optimization algorithms (SBAs). However, their efficiency and convergence remain considerably inadequate. In this work, we propose a trust-region-like algorithm for dealing with expensive multi-objective problems, where only a small number of expensive objective function evaluations are allowed. The optimization process of the proposed algorithm mainly includes two stages. The first stage is the surrogate-assisted stage, in which a promising population with the consideration of convergence and diversity is obtained by reference vector-guided evolutionary algorithms (RVEA). The promising population is considered the preliminary trust-region. Moreover, a new adaptive sampling selection criterion switching between two different sampling strategies is used to further narrow the trust-region. The second stage is the surrogate-based stage, wherein the selection of the most promising individuals is carried out from the ultimate trust-region using the pseudo hypervolume-based expected improvement matrix (PEIM) criterion. Comparison results on the benchmark functions demonstrate that the proposed algorithm is competitive with five state-of-art algorithms in a limited computational budget. Finally, the proposed algorithm is used in the design optimization of a variable stiffness composite cylinder.

An Efficient Hybrid Multi-Objective Optimization Method Coupling Global Evolutionary and Local Gradient Searches for Solving Aerodynamic Optimization Problems

Aerodynamic shape optimization is frequently complicated and challenging due to the involvement of multiple objectives, large-scale decision variables, and expensive cost function evaluation. This paper presents a bilayer parallel hybrid algorithm framework coupling multi-objective local search and global evolution mechanism to improve the optimization efficiency and convergence accuracy in high-dimensional design space. Specifically, an efficient multi-objective hybrid algorithm (MOHA) and a gradient-based surrogate-assisted multi-objective hybrid algorithm (GS-MOHA) are developed under this framework. In MOHA, a novel multi-objective gradient operator is proposed to accelerate the exploration of the Pareto front, and it introduces new individuals to enhance the diversity of the population. Afterward, MOHA achieves a trade-off between exploitation and exploration by selecting elite individuals in the local search space during the evolutionary process. Furthermore, a surrogate-assisted hybrid algorithm based on the gradient-enhanced Kriging with the partial least squares(GEKPLS) approach is established to improve the engineering applicability of MOHA. The optimization results of benchmark functions demonstrate that MOHA is less constrained by dimensionality and can solve multi-objective optimization problems (MOPs) with up to 1000 decision variables. Compared to existing MOEAs, MOHA demonstrates notable enhancements in optimization efficiency and convergence accuracy, specifically achieving a remarkable 5–10 times increase in efficiency. In addition, the optimization efficiency of GS-MOHA is approximately five times that of MOEA/D-EGO and twice that of K-RVEA in the 30-dimensional test functions. Finally, the multi-objective optimization results of the airfoil shape design validate the effectiveness of the proposed algorithms and their potential for engineering applications.

A reliable region information driven kriging-assisted multiobjective rough fuzzy clustering algorithm for color image segmentation

Multiobjective clustering algorithms (MOCAs) are becoming increasingly popular with the merit of segmenting images from multiple perspectives. The performances of MOCAs highly depend on their fitness functions. However, most existing MOCAs adopt one pair of complementary fitness functions, which always measure intra-class compactness and inter-class separation, respectively. This may result in insufficient ability to recognize and mine feature structures from complex images. Moreover, information within color images, such as region information and uncertain information, can barely receive enough attention in MOCAs. To resolve these problems, we propose a reliable region information driven Kriging-assisted multiobjective rough fuzzy clustering algorithm (RRI-KMRFC). Firstly, a reliability-based region information extraction strategy (RRIES) is designed to obtain reliable image information with satisfactory regional homogeneity and abundant image details. Secondly, the derived region information is used to construct three complementary fitness functions, focusing on rough intra-class compactness, rough inter-class separation, and regional consistency, respectively. Such fitness functions can effectively identify the clustering structure, maintain contour details, and characterize uncertain information from color images. To efficiently optimize the proposed functions, an incremental Kriging-assisted evolutionary framework is presented to decrease the expensive function evaluations in which an improved infill sampling strategy is devised to assist in finding unexplored areas in the decision space. Finally, a rough fuzzy clustering validity index with reliable region information is proposed to select the optimal trade-off solution. Experiments performed on Berkeley and Weizmann images confirm the effectiveness and robustness of RRI-KMRFC.

Efficient model‐correction based reliability analysis of uncertain dynamical systems

AbstractIn this contribution, a model‐correction‐based strategy is applied for efficient reliability analysis of uncertain dynamical systems based on a low‐fidelity (LF) model whose outcomes are corrected in a probabilistic sense to represent the more realistic outcomes of the high‐fidelity (HF) model. In the model‐correction approach utilized, the LF model is calibrated to the HF model close to the so‐called most probable point (MPP) in standard normal space, which allows a more realistic assessment of the considered complex dynamical system. Since only few expensive limit state function evaluations of the HF model are required, an efficient reliability analysis is enabled. In an application example, the LF model describes an existing single span railway bridge modelled as simply supported Euler‐Bernoulli beam subjected to moving single forces representing the axle loads of a moving train. The HF modelling approach accounts for the bridge‐train interaction by modelling the passing train as mass‐spring‐damper (MSD) system, however increasing the computational effort of the limit state function evaluations. The failure probabilities evaluated with the model‐correction approach are contrasted and discussed with the failure probabilities of the sophisticated bridge‐train interaction model. It is shown that the model‐correction‐based approach provides reliable failure probability prediction of the HF model while leading to a significant reduction in computational effort.

Efficient model-correction-based reliability analysis of uncertain dynamical systems

The scope of this paper is to apply a model-correction-based strategy for efficient reliability analysis of uncertain dynamical systems based on a low-fidelity (LF) model whose outcomes are corrected in a probabilistic sense to represent the more realistic outcomes of a high-fidelity (HF) model. In the model-correction approach utilized, the LF model is calibrated to the HF model close to the so-called most probable point in standard normal space, which allows a more realistic assessment of the considered complex dynamical system. Since only few expensive limit state function evaluations of the HF model are required, an efficient reliability analysis is enabled. In an application example, the LF model describes an existing single-span railway bridge modelled as simply supported Euler–Bernoulli beam subjected to moving single forces representing the axle loads of a moving train. The HF modelling approach accounts for the bridge–train interaction by modelling the passing train as mass-spring-damper system, however increasing the computational effort of the limit state function evaluations. Failure probabilities evaluated with the model-correction approach are contrasted and discussed with failure probabilities of the sophisticated bridge–train interaction model evaluated with the first-order reliability method (FORM). It is demonstrated that the efficiency of the method depends on the correlation between the LF and the HF model. A comparison of the results of FORM and the model-correction-based approach shows that the latter provides reliable failure probability prediction of the HF model while leading to a significant reduction in computational effort.

An ANN-assisted efficient enriched finite element method via the selective enrichment of moment fitting

Enrichment techniques that employ nonconforming mesh are effective in modeling structures with discontinuities because numerical issues regarding mesh quality are avoided. However, the accurate integration of the bilinear and linear forms on the discretized domain, which is required in the standard Galerkin-based finite element method, is computationally expensive due to the complexity of the enriched basis function. In this paper, we present a fast and accurate alternative method of numerical integration using nonlinear regression enabled by a multi-perceptron feedforward neural network. The relationship between an implicitly represented geometry and the quadrature rule derived from the moment fitting method is predicted by the neural network; the neural network-based regression model circumvents complex computation and significantly reduces the overall online time by avoiding expensive function evaluations. Through the selected numerical examples, we demonstrate the efficiency and accuracy of the current method, as well as the flexibility of the trained network to be used in different contexts.

Optimal sampling-based neural networks for uncertainty quantification and stochastic optimization

In recent times, neural networks (NN) have been successfully utilized to model real-world engineering problems. However, complexities in constructing an optimal NN architecture and limited training data for expensive function evaluations still limit the wide application of NNs. In addition, the distribution of the training samples affects the network's ability to capture the behavior of true responses. Therefore, this study proposes a novel optimal-sampling-based, multi-response, NN framework with adaptive modification of the network's hyperparameters to alleviate these difficulties. The D-optimal design, which maximizes the determinant of the information matrix, is proposed as a sampling technique. The information matrix is built by using the samples and activation values of the neurons in the hidden layers. The optimal sampling is further integrated with an uniform max-min sampling to reduce the recurrence of sampling points and improve the space-filling property. Moreover, a network updating strategy is presented where the model hyperparameters are optimized using a Gaussian process-based Bayesian optimization. An additional feature of the framework is that the same optimal sampling location can be utilized for evaluation of different responses to improve the overall multi-response NN. The proposed framework was applied for uncertainty quantification and stochastic optimization, and the high accuracy and computational efficiency based on the results demonstrated the significance of this research.

A Transformation-Based Improved Kriging Method for the Black Box Problem in Reliability-Based Design Optimization

In order to overcome the drawbacks of expensive function evaluation in the practical reliability-based design optimization (RBDO) problem, researchers have proposed the black box-based RBDO method. The algorithm flow of the commonly employed RBDO method for the black box problem consists of the outer construction loop of the surrogate model of the constraint function and the inner surrogate model-based solving loop. To improve the solving ability of the black box RBDO problem, this paper proposes a transformation-based improved kriging method to increase the effectiveness of the two loops identified above. For the outer loop, a sample distribution-based learning function is suggested to improve the construction efficiency of the surrogate model of the constraint function. For the inner loop, a paired incremental sample-based limit reliability boundary construction approach is suggested to transform the RBDO problem into an equivalent deterministic design optimization problem that can be efficiently solved by classical optimization algorithms. The test results of five cases demonstrate that the proposed method can accurately construct the surrogate model of the constraint function and efficiently solve the black box RBDO problem.

Data-driven Harris Hawks constrained optimization for computationally expensive constrained problems

Aiming at the constrained optimization problem where function evaluation is time-consuming, this paper proposed a novel algorithm called data-driven Harris Hawks constrained optimization (DHHCO). In DHHCO, Kriging models are utilized to prospect potentially optimal areas by leveraging computationally expensive historical data during optimization. Three powerful strategies are, respectively, embedded into different phases of conventional Harris Hawks optimization (HHO) to generate diverse candidate sample data for exploiting around the existing sample data and exploring uncharted region. Moreover, a Kriging-based data-driven strategy composed of data-driven population construction and individual selection strategy is presented, which fully mines and utilizes the potential available information in the existing sample data. DHHCO inherits and develops HHO's offspring updating mechanism, and meanwhile exerts the prediction ability of Kriging, reduces the number of expensive function evaluations, and provides new ideas for data-driven constraint optimization. Comprehensive experiments have been conducted on 13 benchmark functions and a real-world expensive optimization problem. The experimental results suggest that the proposed DHHCO can achieve quite competitive performance compared with six representative algorithms and can find the near global optimum with 200 function evaluations for most examples. Moreover, DHHCO is applied to the structural optimization of the internal components of the real underwater vehicle, and the final satisfactory weight reduction effect is more than 18%.

Surrogate-assisted hybrid evolutionary algorithm with local estimation of distribution for expensive mixed-variable optimization problems

Some real-world design optimization problems can be formulated as expensive mixed-variable optimization problems (EMVOPs), which involve both continuous and discrete decision variables and expensive function evaluations. The main challenges for solving EMVOPs are the handling of mixed variables, limited number of function evaluations and multiple disconnected regions in the search space. In this work, we propose a novel algorithm with global and local search strategies for improving the search ability on disconnected regions, and it only consumes hundreds of function evaluations. The global module employs hybrid evolutionary operators and a Gower distance based surrogate model for handling mixed variables. The local estimation of distributions in different local regions performs in a competitive switching way to combine their advantages, and local surrogate models trained with selected samples improve the accuracy of approximated evaluations for the locally generated solutions. In the late stage, a local continuous search module is executed for refining the continuous decision variables. Verification results on the artificial benchmarks demonstrate that our proposed algorithm is competitive and works effectively. To verify the practicability of the algorithm, it is applied on a convolutional neural network hyperparameter optimization problems and obtains satisfactory results.

Resource allocation problems with expensive function evaluations

The resource allocation problem is among the classical problems in operations research, and has been studied extensively for decades. However, current solution approaches are not able to efficiently handle problems with expensive function evaluations, which can occur in a variety of applications. We study the integer resource allocation problem with expensive function evaluations, for both convex and non-convex separable cost functions. We present several solution methods, both heuristics and exact methods, that aim to limit the number of function evaluations. The methods are compared in numerical experiments using both randomly generated instances and instances from two resource allocation problems occurring in radiation therapy planning. Results show that the presented solution methods compare favorably against existing derivative free optimization solvers.

A Survey on Sustainable Surrogate-Based Optimisation

Surrogate-based optimisation (SBO) algorithms are a powerful technique that combine machine learning and optimisation to solve expensive optimisation problems. This type of problem appears when dealing with computationally expensive simulators or algorithms. By approximating the expensive part of the optimisation problem with a surrogate, the number of expensive function evaluations can be reduced. This paper defines sustainable SBO, which consists of three aspects: applying SBO to a sustainable application, reducing the number of expensive function evaluations, and considering the computational effort of the machine learning and optimisation parts of SBO. The paper reviews sustainable applications that have successfully applied SBO over the past years, and analyses the used framework, type of surrogate used, sustainable SBO aspects, and open questions. This leads to recommendations for researchers working on sustainability-related applications who want to apply SBO, as well as recommendations for SBO researchers. It is argued that transparency of the computation resources used in the SBO framework, as well as developing SBO techniques that can deal with a large number of variables and objectives, can lead to more sustainable SBO.

Optimistic NAUTILUS navigator for multiobjective optimization with costly function evaluations

We introduce novel concepts to solve multiobjective optimization problems involving (computationally) expensive function evaluations and propose a new interactive method called O-NAUTILUS. It combines ideas of trade-off free search and navigation (where a decision maker sees changes in objective function values in real time) and extends the NAUTILUS Navigator method to surrogate-assisted optimization. Importantly, it utilizes uncertainty quantification from surrogate models like Kriging or properties like Lipschitz continuity to approximate a so-called optimistic Pareto optimal set. This enables the decision maker to search in unexplored parts of the Pareto optimal set and requires a small amount of expensive function evaluations. We share the implementation of O-NAUTILUS as open source code. Thanks to its graphical user interface, a decision maker can see in real time how the preferences provided affect the direction of the search. We demonstrate the potential and benefits of O-NAUTILUS with a problem related to the design of vehicles.

Multi-Fidelity Bayesian Experimental Design to Quantify Extreme-Event Statistics

In this work, we develop a multi-fidelity Bayesian experimental design framework to efficiently quantify the extreme-event statistics of an input-to-response (ItR) system with given input probability and expensive function evaluations. The key idea here is to leverage low-fidelity samples whose responses can be computed with a cost of a certain fraction of that for high-fidelity samples, in an optimized configuration to reduce the total computational cost. To accomplish this goal, we employ a multi-fidelity Gaussian process as the surrogate model of the ItR function, and develop a new acquisition based on which the optimized next sample can be selected in terms of its location in the sample space and the fidelity level. In addition, we develop an inexpensive analytical evaluation of the acquisition and its derivative, avoiding numerical integrations that are prohibitive for high-dimensional problems. The new method is tested in a bi-fidelity context for a series of synthetic problems with varying dimensions, low-fidelity model accuracy and computational costs. Comparing with the single-fidelity method and the bi-fidelity method with a pre-defined fidelity hierarchy, our method consistently shows the best (or among the best) performance for all the test cases. Finally, we demonstrate the superiority of our method in solving an engineering problem of estimating the extreme ship motion statistics in irregular waves, using computational fluid dynamics (CFD) with two different grid resolutions as the high and low fidelity models.

Flatness-Based Reduced Hessian Method for Optimal Control of Aircraft

Numerical solutions of flatness-based reformulation of optimal control problems lead to an optimization problem with fewer variables and constraints. However, the expressions for the states and controls in terms of the flat output variables can be highly nonlinear and complicated. Hence, this may lead to expensive function evaluations within the optimization problem, degradation of convexity, and issues with convergence. Thus, a flatness-based reformulation of the optimal control problems may not be a viable alternative for computational guidance and control. Alternatively, a novel flatness-based reduced Hessian sequential quadratic programming algorithm is developed in this paper to solve optimization problems for a six-degree-of-freedom aircraft. An analytical null space basis is derived from the linearized differential constraints, which leads to a reduced-dimensional quadratic program with the discretized flat outputs as decision variables. Unlike flatness-based reformulation of the optimal control problem, the null space/reduced Hessian method does not introduce additional nonlinearity and preserves the convexity of the optimization problem. A nonlinear model predictive control problem is solved to demonstrate the reduced Hessian algorithm. The algorithm is validated and Monte Carlo trials are performed to assess the effectiveness of the approach. Computational studies show that the current approach is faster than methods that directly exploit sparsity up to a factor of five.