Expensive Function Evaluations Research Articles

A Novel Intelligent Multiobjective Simulated Annealing Algorithm for Designing Robust PID Controllers

This paper proposes an intelligent multiobjective simulated annealing algorithm (IMOSA) and its application to an optimal proportional integral derivative (PID) controller design problem. A well-designed PID-type controller should satisfy the following objectives: 1) disturbance attenuation; 2) robust stability; and 3) accurate setpoint tracking. The optimal PID controller design problem is a large-scale multiobjective optimization problem characterized by the following: 1) nonlinear multimodal search space; 2) large-scale search space; 3) three tight constraints; 4) multiple objectives; and 5) expensive objective function evaluations. In contrast to existing multiobjective algorithms of simulated annealing, the high performance in IMOSA arises mainly from a novel multiobjective generation mechanism using a Pareto-based scoring function without using heuristics. The multiobjective generation mechanism operates on a high-score nondominated solution using a systematic reasoning method based on an orthogonal experimental design, which exploits its neighborhood to economically generate a set of well-distributed nondominated solutions by considering individual and overall objectives. IMOSA is evaluated by using a practical design example of a super-maneuverable fighter aircraft system. An efficient existing multiobjective algorithm, the improved strength Pareto evolutionary algorithm, is also applied to the same example for comparison. Simulation results demonstrate high performance of the IMOSA-based method in designing robust PID controllers.

ORBIT: Optimization by Radial Basis Function Interpolation in Trust-Regions

We present a new derivative-free algorithm, ORBIT, for unconstrained local optimization of computationally expensive functions. A trust-region framework using interpolating Radial Basis Function (RBF) models is employed. The RBF models considered often allow ORBIT to interpolate nonlinear functions using fewer function evaluations than the polynomial models considered by present techniques. Approximation guarantees are obtained by ensuring that a subset of the interpolation points is sufficiently poised for linear interpolation. The RBF property of conditional positive definiteness yields a natural method for adding additional points. We present numerical results on test problems to motivate the use of ORBIT when only a relatively small number of expensive function evaluations are available. Results on two very different application problems, calibration of a watershed model and optimization of a PDE-based bioremediation plan, are also encouraging and support ORBIT's effectiveness on blackbox functions for which no special mathematical structure is known or available.

On the use of metamodel-assisted, multi-objective evolutionary algorithms

This article is concerned with the optimal use of metamodels in the context of multi-objective evolutionary algorithms which are based on computationally expensive function evaluations. The goal is to capture Pareto fronts of optimal solutions with the minimum computational cost. In each generation during the evolution, the metamodels act as filters that distinguish the most promising individuals, which will solely undergo exact and costly evaluations. By means of the so-called inexact pre-evaluation phase, based on continuously updated local metamodels, most of the non-promising individuals are put aside without aggravating the overall cost. The gain achieved through this technique is amazing in single-objective problems. However, with more than one objective, noticeable performance degradation occurs. This article scrutinizes the role of metamodels in multi-objective evolutionary algorithms and proposes ways to overcome expected weaknesses and improve their performance. Minimization of mathematical functions as well as aerodynamic shape optimization problems are used for demonstration purposes.

Algorithm 856

APPSPACK is software for solving unconstrained and bound-constrained optimization problems. It implements an asynchronous parallel pattern search method that has been specifically designed for problems characterized by expensive function evaluations. Using APPSPACK to solve optimization problems has several advantages: No derivative information is needed; the procedure for evaluating the objective function can be executed via a separate program or script; the code can be run serially or in parallel, regardless of whether the function evaluation itself is parallel; and the software is freely available. We describe the underlying algorithm, data structures, and features of APPSPACK version 4.0, as well as how to use and customize the software.

An optimization methodology of alkaline–surfactant–polymer flooding processes using field scale numerical simulation and multiple surrogates

After conventional waterflood processes the residual oil in the reservoir remains as a discontinuous phase in the form of oil drops trapped by capillary forces and is likely to be around 70% of the original oil in place (OOIP). The EOR method so-called alkaline–surfactant–polymer (ASP) flooding has proved to be effective in reducing the oil residual saturation in laboratory experiments and field projects through the reduction of interfacial tension and mobility ratio between oil and water phases. A critical step to make ASP floodings more effective is to find the optimal values of design variables that will maximize a given performance measure (e.g., net present value, cumulative oil recovery) considering a heterogeneous and multiphase petroleum reservoir. Previously reported works using reservoir numerical simulation have been limited to sensitivity analyses at core and field scale levels because the formal optimization problem includes computationally expensive objective function evaluations (field scale numerical simulations). This work presents a surrogate-based optimization methodology to overcome this shortcoming. The proposed approach estimates the optimal values for a set of design variables (e.g., slug size and concentration of the chemical agents) to maximize the cumulative oil recovery from a heterogeneous and multiphase petroleum reservoir subject to an ASP flooding. The surrogate-based optimization approach has been shown to be useful in the optimization of computationally expensive simulation-based models in the aerospace, automotive, and oil industries. In this work, we improve upon this approach along two directions: (i) using multiple surrogates for optimization, and (ii) incorporating an adaptive weighted average model of the individual surrogates. The cited approach involves the coupled execution of a global optimization algorithm and fast surrogates (i.e., based on Polynomial Regression, Kriging, Radial Basis Functions and a Weighted Average Model) constructed from field scale numerical simulation data. The global optimization program implements the DIRECT algorithm and the reservoir numerical simulations are conducted using the UTCHEM program from the University of Texas at Austin. The effectiveness and efficiency of the proposed methodology is demonstrated using a field scale case study.

Constrained optimization involving expensive function evaluations: A sequential approach

This paper presents a new sequential method for constrained nonlinear optimization problems. The principal characteristics of these problems are very time consuming function evaluations and the absence of derivative information. Such problems are common in design optimization, where time consuming function evaluations are carried out by simulation tools (e.g., FEM, CFD). Classical optimization methods, based on derivatives, are not applicable because often derivative information is not available and is too expensive to approximate through finite differencing. The algorithm first creates an experimental design. In the design points the underlying functions are evaluated. Local linear approximations of the real model are obtained with help of weighted regression techniques. The approximating model is then optimized within a trust region to find the best feasible objective improving point. This trust region moves along the most promising direction, which is determined on the basis of the evaluated objective values and constraint violations combined in a filter criterion. If the geometry of the points that determine the local approximations becomes bad, i.e. the points are located in such a way that they result in a bad approximation of the actual model, then we evaluate a geometry improving instead of an objective improving point. In each iteration a new local linear approximation is built, and either a new point is evaluated (objective or geometry improving) or the trust region is decreased. Convergence of the algorithm is guided by the size of this trust region. The focus of the approach is on getting good solutions with a limited number of function evaluations.

Parallel global optimization with the particle swarm algorithm

Present day engineering optimization problems often impose large computational demands, resulting in long solution times even on a modern high-end processor. To obtain enhanced computational throughput and global search capability, we detail the coarse-grained parallelization of an increasingly popular global search method, the particle swarm optimization (PSO) algorithm. Parallel PSO performance was evaluated using two categories of optimization problems possessing multiple local minima-large-scale analytical test problems with computationally cheap function evaluations and medium-scale biomechanical system identification problems with computationally expensive function evaluations. For load-balanced analytical test problems formulated using 128 design variables, speedup was close to ideal and parallel efficiency above 95% for up to 32 nodes on a Beowulf cluster. In contrast, for load-imbalanced biomechanical system identification problems with 12 design variables, speedup plateaued and parallel efficiency decreased almost linearly with increasing number of nodes. The primary factor affecting parallel performance was the synchronization requirement of the parallel algorithm, which dictated that each iteration must wait for completion of the slowest fitness evaluation. When the analytical problems were solved using a fixed number of swarm iterations, a single population of 128 particles produced a better convergence rate than did multiple independent runs performed using sub-populations (8 runs with 16 particles, 4 runs with 32 particles, or 2 runs with 64 particles). These results suggest that (1) parallel PSO exhibits excellent parallel performance under load-balanced conditions, (2) an asynchronous implementation would be valuable for real-life problems subject to load imbalance, and (3) larger population sizes should be considered when multiple processors are available.

Space exploration and global optimization for computationally intensive design problems: a rough set based approach

Modern engineering design problems often involve computation-intensive analysis and simulation processes. Design optimization based on such processes is desired to be efficient, informative and transparent. This work proposes a rough set based approach that can identify multiple sub-regions in a design space, within which all of the design points are expected to have a performance value equal to or less than a given level. The rough set method is applied iteratively on a growing sample set. A novel termination criterion is also developed to ensure a modest number of total expensive function evaluations to identify these sub-regions and search for the global optimum. The significance of the proposed method is twofold. First, it provides an intuitive method to establish the mapping from the performance space to the design space, i.e. given a performance level, its corresponding design region(s) can be identified. Such a mapping could be potentially used to explore and visualize the entire design space. Second, it can be naturally extended to a global optimization method. It also bears potential for more broad application to problems such as metamodeling-based design and robust design optimization. The proposed method was tested with a number of test problems and compared with a few well-known global optimization algorithms.

Convergence of Hybrid Space Mapping Algorithms

The space mapping technique is intended for optimization of engineering models which involve very expensive function evaluations. It may be considered a preprocessing method which often provides a very efficient initial phase of an optimization procedure. However, the ultimate rate of convergence may be poor, or the method may even fail to converge to a stationary point. We consider a convex combination of the space mapping technique with a classical optimization technique. The function to be optimized has the form H ○ f where H : R m → R is convex and f : R n → R m is smooth. Experience indicates that the combined method maintains the initial efficiency of the space mapping technique. We prove that the global convergence property of the classical technique is also maintained: The combined method provides convergence to the set of stationary points of H ○ f.

2D shape optimization of turbine blades

AbstractIn this paper, we present methods to optimize the efficiency of turbine blades. The profile of a turbine blade is parameterized by curves, its parameters form the variables of a nonlinear optimization problem. Because of the very expensive function evaluation, gradient based optimization methods are preferred. In order to provide gradient information for the optimization method, the fluid sensitivities have to be considered. Theoretical properties and implementation aspects for the sensitivity equation are discussed. Constraints guarantee the requirements for casting, machining, and cooling of the blades.

Pattern Search Methods for User-Provided Points: Application to Molecular Geometry Problems

This paper deals with the application of pattern search methods to the numerical solution of a class of molecular geometry problems with important applications in molecular physics and chemistry. The goal is to find a configuration of a cluster or a molecule with minimum total energy. The minimization problems in this class of molecular geometry problems have no constraints, and the objective function is smooth. The difficulties arise from the existence of several local minima and, especially, from the expensive function evaluation (total energy) and the possible nonavailability of first-order derivatives. We introduce a pattern search approach that attempts to exploit the physical nature of the problem by using energy lowering geometrical transformations and to take advantage of parallelism without the use of derivatives. Numerical results for a particular instance of this new class of pattern search methods are presented, showing the promise of our approach. The new pattern search methods can be used in any other context where there is a user-provided scheme to generate points leading to a potential objective function decrease.

Space Mapping: Models, Sensitivities, and Trust-Regions Methods

The goal of this paper is to organize some of the mathematical and algorithmic aspects of the space-mapping technique for continuous optimization with expensive function evaluations. First, we consider the mapping from the fine space to the coarse space when the models are vector-valued functions and when the space-mapping (nonlinear) least-squares residual is nonzero. We show how the sensitivities of the space mapping can be used to deal with space-mapping surrogates of the fine model. We derive a framework where it is possible to design globally convergent trust-region methods to minimize such fine-model surrogates. We consider also a different perspective of space mapping and apply it, for sake of simplicity, to the situation where the models are scalar functions. The space mapping is defined in a way where it is reasonable to assume that it is point-to-point. We prove that the surrogate model built by composition of the space mapping and the coarse model is a regular function. We also discuss trust-region methods in this context.

A Knowledge-Based Approach To Response Surface Modelling in Multifidelity Optimization

This paper is concerned with approximations for expensive function evaluation – the expensive functions arising in an engineering design context. The problem of reducing the computational cost of generating sufficient learning samples is addressed. Several approaches of using a priori knowledge to achieve computational economy are presented. In all these, the results of a cheap model are treated as knowledge to be incorporated in the training process. Several approaches are described here: in particular, we focus on neural based systems. This approach is then developed as a new knowledge-based kriging model which is shown to be as accurate as neural based alternatives while being much easier to train. Examples from the domain of structural optimization are given to demonstrate the approach.

An Introduction to the Space Mapping Technique

The space mapping technique is intended for optimization of engineering models which involve very expensive function evaluations. It is assumed that two different models of the same physical system are available: Besides the expensive model of primary interest (denoted the fine model), access to a cheaper (coarse) model is assumed which may be less accurate. The main idea of the space mapping technique is to use the coarse model to gain information about the fine model, and to apply this in the search for an optimal solution of the latter. Thus the technique iteratively establishes a mapping between the parameters of the two models which relate similar model responses. Having this mapping, most of the model evaluations can be directed to the fast coarse model. In many cases this technique quickly provides an approximate optimal solution to the fine model that is suffi- ciently accurate for engineering purposes. Thus the space mapping technique may be considered a preprocessing technique that perhaps must be succeeded by use of classical optimization techniques. We present an automatic scheme which integrates the space mapping and classical techniques.

A Combined Convex Approximation—Interior Point Approach for Large Scale Nonlinear Programming

The method of moving asymptotes (MMA) and its globally convergent extension SCP (sequential convex programming) are known to work well in the context of structural optimization. The two main reasons are that the approximation scheme used for the objective function and the constraints fits very well to these applications and that at an iteration point a local optimization model is used such that additional expensive function and gradient evaluations of the original problem are avoided. The subproblems that occur in both methods are special nonlinear convex programs and have traditionally been solved using a dual approach. This is now replaced by an interior point approach. The latter one is more suitable for large problems because sparsity properties of the original problem can be preserved and the separability property of the approximation functions is exploited. The effectiveness of the new method is demonstrated by a few examples dealing with problems of structural optimization.

A Study on the Construction of Response Surfaces for Design Optimization

Gradient-based optimization methods are inefficient in applications which require expensive function evaluations, and useless in applications where objective and/or constraint functions are 'noisy' due to modeling and cumulative numerical inaccuracy since gradient evaluation results cannot be reliable. Moreover, it is difficult to be integrated with commercial analysis software, and they cannot be employed when only experimental analysis results are available. In this research an optimization program based on a response surface method has been developed to overcome the aforementioned difficulties. Various methods for design of experiments and new proposed approximation models are implemented in the program. The effectiveness of the optimization program is tested on several test problems and results are discussed.

Runge–Kutta–Nyström‐type parallel block predictor–corrector methods

This paper describes the construction of block predictor–corrector methods based on Runge–Kutta–Nystrom correctors. Our approach is to apply the predictor–corrector method not only with stepsize h, but, in addition (and simultaneously) with stepsizes a i h, i = 1 ...,r. In this way, at each step, a whole block of approximations to the exact solution at off‐step points is computed. In the next step, these approximations are used to obtain a high‐order predictor formula using Lagrange or Hermite interpolation. Since the block approximations at the off‐step points can be computed in parallel, the sequential costs of these block predictor–corrector methods are comparable with those of a conventional predictor–corrector method. Furthermore, by using Runge–Kutta–Nystrom corrector methods, the computation of the approximation at each off‐step point is also highly parallel. Numerical comparisons on a shared memory computer show the efficiency of the methods for problems with expensive function evaluations.

Response surface approximations for structural optimization

Response surface methodology can be used to construct global and midrange approximations to functions in structural optimization. Since structural optimization requires expensive function evaluations, it is important to construct accurate function approximations so that rapid convergence may be achieved. In this paper techniques to find the region of interest containing the optimal design, and techniques for finding more accurate approximations are reviewed and investigated. Aspects considered are experimental design techniques, the selection of the ‘best’ regression equation, intermediate response functions and the location and size of the region of interest. Standard examples in structural optimization are used to show that the accuracy is largely dependent on the choice of the approximating function with its associated subregion size, while the selection of a larger number of points is not necessarily cost-effective. In a further attempt to improve efficiency, different regression models were investigated. The results indicate that the use of the two methods investigated does not significantly improve the results. Finding an accurate global approximation is challenging, and sufficient accuracy could only be achieved in the example problems by considering a smaller region of the design space. © 1998 John Wiley & Sons, Ltd.

AQUIFER PARAMETER ESTIMATION USING GENETIC ALGORITHMS AND NEURAL NETWORKS

Computational tools such as genetic algorithms and neural networks are becoming increasingly popular in scientific applications involving mathematical modeling. These tools emulate natural biological processes in an attempt to build more robust and efficient mathematical models. The present study explores the applicability of genetic algorithms and neural networks for aquifer parameter estimation, in an optimization framework. Although optimization models based on genetic algorithms are more robust than conventional nonlinear programming techniques, they often necessitate many computationally expensive function evaluations. On the other hand, genetic algorithms can also tolerate approximate function evaluations. The present study employs artificial neural networks that provide quick but reasonably accurate function evaluation, in conjunction with genetic algorithms. Such an optimization framework makes the resulting calibration model highly robust and efficient. Applicability of the proposed model is demonstrated on a hypothetical aquifer using synthetic test data. Through an extensive sensitivity analysis, the present study reiterates that a low probability of mutation (0.02-0.03) and a moderately high probability of crossover (0.6-0.7) are essential for good convergence of a genetic optimization model.

Annealing in a genetic algorithm for task-dynamic recovery from speech acoustics.

Genetic algorithms, as they have been used for recovery of task-dynamic parameters [R. S. McGowan, ‘‘Recovering articulatory movement from formant frequency trajectories using task dynamics and a genetic algorithm,’’ Speech Commun. 14, 19–49 (1994)], can take some time to converge, which means that expensive function evaluations are necessary to obtain optimum solutions. In previous experiments, each task dynamic- target was always given 6 bits in the chromosome. In the tests reported here, the number of bits in the chromosome string representing target positions was made a function of generation, with the range of the targets remaining constant to see whether an annealing process with ever increasing resolution would speed convergence to a good fitness maximum. Results show that annealing can be helpful in speeding convergence, although not all annealing schedules help. This procedure is related to simulated annealing where the temperature of the search is decreased as the search progresses. It is also related to codebook look-up used in the inverse problem, where vocal tract shapes that produce speech close to that of the data are accessed to initialize a subsequent optimization procedure. [Work supported by NIH Grant DC-01247 to Haskins Laboratories.]