Multi-fidelity Optimization Research Articles

Bayesian Optimization Using Simulation-Based Multiple Information Sources over Combinatorial Structures

Bayesian optimization due to its flexibility and sample efficiency has become a standard approach for simulation optimization. To reduce this problem, one can resort to cheaper surrogates of the objective function. Examples are ubiquitous, from protein engineering or material science to tuning machine learning algorithms, where one could use a subset of the full training set or even a smaller related dataset. Cheap information sources in the optimization scheme have been studied in the literature as the multi-fidelity optimization problem. Of course, cheaper sources may hold some promise toward tractability, but cheaper models offer an incomplete model inducing unknown bias and epistemic uncertainty. In this manuscript, we are concerned with the discrete case, where fx,wi is the value of the performance measure associated with the environmental condition wi and p(wi) represents the relevance of the condition wi (i.e., the probability of occurrence or the fraction of time this condition occurs). The main contribution of this paper is the proposal of a Gaussian-based framework, called augmented Gaussian process (AGP), based on sparsification, originally proposed for continuous functions and its generalization in this paper to stochastic optimization using different risk profiles for combinatorial optimization. The AGP leverages sample and cost-efficient Bayesian optimization (BO) of multiple information sources and supports a new acquisition function to select the new source–location pair considering the cost of the source and the (location-dependent) model discrepancy. An extensive set of computational results supports risk-aware optimization based on CVaR (conditional value-at-risk). Computational experiments confirm the actual performance of the MISO-AGP method and the hyperparameter optimization on benchmark functions and real-world problems.

Development of a multi-fidelity optimisation strategy based on hybrid methods

A multi-fidelity optimisation strategy has been developed in the present work, and its performance is illustrated through a series of test cases. The strategy is based on hybrid methods such that two genetic optimisation algorithms are employed, each coupled to a different fidelity level with transfer of information between them. The aim is that the low fidelity model, being less accurate but with a lower computational cost, performs a comprehensive search along the design space guiding the high fidelity model to the optimum region. This strategy has been shown to reduce the computational time of an optimisation through analytical test cases as well as numerical cases. The analytical cases have been used to tune the parameters that define the multi-fidelity strategy, while the numerical cases are employed to apply the method to engineering problems, focusing on the aerodynamic performance of an airfoil. The speed-up shows a certain dependency to the models relation, both regarding their similarity level as well as the relative computational cost. For cases exhibiting a significant dissimilarity between models, wherein the low fidelity model is notably inaccurate, the attained speed-up diminishes, and numerous instances demonstrate an absence of speed-up. However, for most cases, even with poor model similarity the optimisations are accelerated by an order of 2, while values up to 3–5 were found for higher similarity levels. Hence, the developed strategy shows a relevant decrease of computational cost of an optimisation procedure although its performance is affected by the models relative accuracy.

A novel multi-fidelity optimization framework for high-altitude propellers

A novel optimization methodology is presented for optimizing high-altitude propellers operating in the lower stratosphere. The proposed methodology combines a Bayesian optimization approach for most design variables and Vortex Theory for the optimization of twist distribution and rotational velocity. For the Bayesian optimization part of the problem, a multi-fidelity approach is employed with three levels of modeling fidelity. The considered fidelity levels, from lower to higher, are: Vortex Theory, and 3D Reynolds-Averaged Navier-Stokes (RANS) with the use of γ−Reθ transition model, converged with first-order upwind, and second-order upwind for the momentum equations. Additionally, the optimization problem involves two objectives: maximizing the cruise efficiency and minimizing the total volume of the propeller. To enhance the performance of multi-fidelity, multi-objective (MFMO) acquisition functions, a novel algorithm is proposed. The proposed optimization methodology demonstrates its effectiveness in generating several high-performing propeller designs and proves to be more efficient than an equivalent problem formulation relying solely on Bayesian optimization. Post-processing of optimization data reveals new design variable bounds that correspond to more efficient blade shapes. These findings provide insights into some important geometric features of high-altitude propellers and establish guidelines for future optimization efforts within the specified thrust and power consumption ranges. Finally, simulations of the best-performing designs across a range of advance ratios and altitudes confirm their high performance in various operating conditions.

A Framework for Developing Systematic Testbeds for Multi-Fidelity Optimization Techniques

Abstract Multi-fidelity (MF) models abound in simulation-based engineering fields. Many MF strategies have been proposed to improve the efficiency in engineering processes, especially in design optimization. When it comes to assessing the performance of MF optimization techniques, existing practice usually relies on test cases involving contrived MF models of seemingly random math functions, due to limited access to real-world MF models. While it is acceptable to use contrived MF models, these models are often manually written up rather than created in a systematic manner. This gives rise to the potential pitfall that the test MF models may be not representative of general scenarios. We propose a framework to generate test MF models systematically and characterize tested MF optimization methods' performances comprehensively. In our framework, the MF models are generated based on given high-fidelity (HF) model and come with two parameters to control their fidelity levels and allow model randomization. In our testing process, MF case problems are systematically formulated using our model creation method. Running the given MF optimization technique on these problems produces what we call “savings curve” that characterizes the method's performance similarly to how ROC curves characterize machine learning classifiers. Our test results also allow plotting “optimality curves” that serve similar functionality to savings curves in certain types of problems. The flexibility of our MF model creation facilitates the development of testing processes for general MF problem scenarios, and our framework can be easily extended to other MF applications than optimization.

Efficient adversarial training with multi-fidelity optimization for robust neural network

Adversarial examples (AEs) pose a significant threat to the security and reliability of deep neural networks. Adversarial training (AT) is one of the effective defense methods, involving the integration of a number of generated AEs into the training process to enhance model robustness. However, the computational cost associated with AE generation is unbearable, particularly for large-scale tasks. In pursuit of fast AT, many algorithms generate AEs by adopting a simple attack strategy, but they often sacrifice the quality of AEs and suffer from catastrophic overfitting, resulting in suboptimal model robustness. To address these issues, our approach incorporates multi-fidelity optimization, which employs a dynamic attack strategy to generate AEs with varying fidelity within a suitable range. Furthermore, we introduce a surrogate-assisted fidelity estimation module at the beginning of our proposed algorithm, allowing for the adaptive determination of the fidelity range tailored to specific tasks. Comparative experiments with seven state-of-the-art algorithms on three networks and three datasets demonstrate that the proposed algorithm obtains a competitive robust accuracy but spends only 50% of the training time of the projected gradient descent algorithm.

Leveraging trust for joint multi-objective and multi-fidelity optimization

In the pursuit of efficient optimization of expensive-to-evaluate systems, this paper investigates a novel approach to Bayesian multi-objective and multi-fidelity (MOMF) optimization. Traditional optimization methods, while effective, often encounter prohibitively high costs in multi-dimensional optimizations of one or more objectives. Multi-fidelity approaches offer potential remedies by utilizing multiple, less costly information sources, such as low-resolution approximations in numerical simulations. However, integrating these two strategies presents a significant challenge. We propose the innovative use of a trust metric to facilitate the joint optimization of multiple objectives and data sources. Our methodology introduces a modified multi-objective (MO) optimization policy incorporating the trust gain per evaluation cost as one of the objectives of a Pareto optimization problem. This modification enables simultaneous MOMF optimization, which proves effective in establishing the Pareto set and front at a fraction of the cost. Two specific methods of MOMF optimization are presented and compared: a holistic approach selecting both the input parameters and the fidelity parameter jointly, and a sequential approach for benchmarking. Through benchmarks on synthetic test functions, our novel approach is shown to yield significant cost reductions—up to an order of magnitude compared to pure MO optimization. Furthermore, we find that joint optimization of the trust and objective domains outperforms sequentially addressing them. We validate our findings with the specific use case of optimizing particle-in-cell simulations of laser-plasma acceleration, highlighting the practical potential of our method in the Pareto optimization of highly expensive black-box functions. Implementation of the methods in existing Bayesian optimization frameworks is straightforward, with immediate extensions e.g. to batch optimization possible. Given their ability to handle various continuous or discrete fidelity dimensions, these techniques have wide-ranging applicability in tackling simulation challenges across various scientific computing fields such as plasma physics and fluid dynamics.

Optimising the Configuration of the CMS GPU Reconstruction

Particle track reconstruction for high energy physics experiments like CMS is computationally demanding but can benefit from GPU acceleration if properly tuned. This work develops an autotuning framework to automatically optimise the throughput of GPU-accelerated CUDA kernels in CMSSW. The proposed system navigates the complex parameter space by generating configurations, benchmarking performance, and leveraging multi-fidelity optimisation from simplified applications. The autotuned launch parameters improved CMSSW tracking throughput over the default settings by finding optimised, GPU-specific configurations. The successful application of autotuning to CMSSW demonstrates both performance portability across diverse accelerators and the potential of the methodology to optimise other HEP codebases.

Practical ship afterbody optimization by multifidelity techniques

ABSTRACT This paper discusses multifidelity methods for CFD-based optimization of ship afterbody designs; aimed at a fast application to a variety of practical cases. Surrogate-based global optimization is used, using a multi-objective genetic algorithm. The surrogates are derived by combining few high-fidelity computations, by free-surface RANS codes, with many low-fidelity computations. For rather slender vessels for which the wave resistance variations over the design space are dominant, a free-surface potential flow code is found very effective as a low-fidelity solver, permitting a large reduction of the number of RANS computations and the associated cost. Examples are shown for model 5415 and a fast displacement vessel. For cases with variation in both viscous and wave resistance, an alternative method is used combining coarse and fine-grid RANS computations; the coarse-grid ones being about 20 times cheaper. Application of this coarse/fine grid multifidelity optimization to a containership and a motor yacht shows its effectivity.

Scalable Bayesian optimization with randomized prior networks

Several fundamental problems in science and engineering consist of global optimization tasks involving unknown high-dimensional (black-box) functions that map a set of controllable variables to the outcomes of an expensive experiment. Bayesian Optimization (BO) techniques are known to be effective in tackling global optimization problems using a relatively small number objective function evaluations, but their performance suffers when dealing with high-dimensional outputs. To overcome the major challenge of dimensionality, here we propose a deep learning framework for BO and sequential decision making based on bootstrapped ensembles of neural architectures with randomized priors. Using appropriate architecture choices, we show that the proposed framework can approximate functional relationships between design variables and quantities of interest, even in cases where the latter take values in high-dimensional vector spaces or even infinite-dimensional function spaces. In the context of BO, we augmented the proposed probabilistic surrogates with re-parameterized Monte Carlo approximations of multiple-point (parallel) acquisition functions, as well as methodological extensions for accommodating black-box constraints and multi-fidelity information sources. We test the proposed framework against state-of-the-art methods for BO and demonstrate superior performance across several challenging tasks with high-dimensional outputs, including a constrained multi-fidelity optimization task involving shape optimization of rotor blades in turbo-machinery.

An adaptive multi-fidelity optimization framework based on co-Kriging surrogate models and stochastic sampling with application to coastal aquifer management

Surrogate modelling has been used successfully to alleviate the computational burden that results from high-fidelity numerical models of seawater intrusion in simulation-optimization routines. Nevertheless, little attention has been given to multi-fidelity modelling methods to address cases where only limited runs with computationally expensive seawater intrusion models are considered affordable imposing a limiting factor for single-fidelity surrogate-based optimization as well. In this work, a new adaptive multi-fidelity optimization framework is proposed based on co-Kriging surrogate models considering two model fidelities of seawater intrusion. The methodology is tailored to the needs of solving pumping optimization problems with computationally expensive constraint functions and utilizes only small high-fidelity training datasets. Results from both hypothetical and real-world optimization problems demonstrate the efficiency and practicality of the proposed framework to provide a steep improvement of the objective function while it outperforms a comprehensive single-fidelity surrogate-based optimization method. The method can also be used to locate optimal solutions in the region of the global optimum when larger high-fidelity training datasets are available.

Multi-objective multi-fidelity optimisation for position-constrained human-robot collaborative disassembly planning

ABSTRACT Human-robot collaborative disassembly lines are widely used by remanufacturing companies to disassemble end-of-life (EOL) products. When disassembling large-sized EOL products, each workstation on a disassembly line is generally divided into multiple operating positions, so that different operators can disassemble the same product at their respective positions at the same time, thereby greatly improving efficiency. This paper focuses on a position-constrained human-robot collaborative disassembly planning (PC-HRCDP) problem for the above-mentioned lines, including three subproblems of disassembly sequence planning, disassembly line balancing and robot path planning. A multi-objective mixed integer programming model for PC-HRCDP is developed to solve small-scale instances. Furthermore, a multi-objective multi-fidelity optimisation (MO-MFO) algorithm is proposed to solve large-scale instances. Comprehensive experiments are conducted based on 10 problem instances generated in this study. Experimental results show that the proposed MO-MFO is better than a high-fidelity optimisation algorithm in terms of running time. In addition, benefiting from the strategy of MO-MFO to allocate the limited high-fidelity computational budget to solutions in the two stages of multi-objective optimisation and optimal sampling, MO-MFO is significantly better than the existing representative multi-fidelity optimisation algorithms in terms of the hyper-volume and the inverted generational distance.

Multi-fidelity optimization of metal sheets concerning manufacturability in deep-drawing processes

Multi-fidelity optimization, which complements an expensive high-fidelity function with cheaper low-fidelity functions, has been successfully applied in many fields of structural optimization. In the present work, an exemplary cross-die deep-drawing optimization problem is investigated to compare different objective functions and to assess the performance of a multi-fidelity efficient global optimization technique. To that end, hierarchical kriging is combined with an infill criterion called variable-fidelity expected improvement. Findings depend significantly on the choice of objective function, highlighting the importance of careful consideration when defining an objective function. We show that one function based on the share of bad elements in a forming limit diagram is not well suited to optimize the example problem. In contrast, two other definitions of objective functions, the average sheet thickness reduction and an averaged limit violation in the forming limit diagram, confirm the potential of a multi-fidelity approach. They significantly reduce computational cost at comparable result quality or even improve result quality compared to a single-fidelity optimization.

Integrated Framework for Constrained Optimization of Horizontal/Deviated Well Placement and Control for Geological CO2 Storage

Summary A general framework for optimizing the locations and time-varying injection rates of a set of monobore wells for geological carbon storage is presented and applied. Two objective functions, minimization of mobile CO2 fraction at the end of the operation and maximization of storage efficiency, are considered separately in single-objective optimizations and in combination for biobjective optimization. Appropriate linear and nonlinear constraints, involving the geometry of the well configuration, injection rates, and injected mass (for pressure management), are specified. Two derivative-free algorithms, particle swarm optimization (PSO) and differential evolution (DE), are applied and assessed. The various constraints are treated using a preprocessing repair procedure, penalty functions, and a filter method. The framework uses multifidelity (MF) optimization, in which increasing levels of grid resolution are applied during the course of the optimization run. For single-objective optimizations, the MF approach is compared with high-resolution optimization. This treatment is shown to outperform high-resolution PSO and DE optimization in terms of both solution quality and computational requirements. The MF DE optimization case provides the best (feasible) solution, with a 0.090 mobile CO2 fraction at 200 years, which represents a 68% improvement over a heuristic base-case. For the second objective function, MF PSO provides a design that results in a storage efficiency of 0.074, which is about double the base-case value. The well configurations are much different for the two objective functions, with wells more closely spaced, resulting in a single merged plume, for the storage efficiency maximization case. For the mobile CO2 minimization case, by contrast, wells are separated and pulsed, which facilitates dissolution and residual trapping. Biobjective optimization is then performed, again using the MF approach, with a model based on an actual storage operation now under development. The resulting well configurations and CO2 plumes for selected Pareto-optimal solutions are presented.

An Iterative Two-Stage Multifidelity Optimization Algorithm for Computationally Expensive Problems

Engineering design optimization often involves use of numerical simulations to assess the performance of candidate designs. The simulations for computing high-fidelity (HF) performance estimates, such as finite element analysis or computational fluid dynamics, are typically computationally expensive. In some cases, it may also be possible to run an alternate or cheaper version of the simulation (through, e.g., use of a coarse mesh) to yield a low-fidelity (LF) performance estimate. Multi-fidelity optimization refers to the class of methods that aim to manage LF and HF evaluations efficiently to optimize computationally expensive problems within a limited computing budget. Among the prominent existing multi-fidelity approaches, some of them depend on a sufficiently dense a priori sampling; while others use unidirectional information exchange from LF to HF; both of which lead to a possibility of spending evaluation budget on unpromising search regions. This paper proposes an improved multi-fidelity approach using an iterative, two-stage scheme (MFITS). It uses the collective information from previously evaluated designs to determine a sampling neighbourhood for LF evaluations. These samples are in-turn used for building a co-kriging surrogate model that is then searched globally to identify a good candidate for HF evaluation. By restricting the LF sampling neighbourhood, the computational budget can be used more efficiently, as the search is focused on regions that have historically produced good quality solutions. Numerical experiments and benchmarking are conducted on two suites of test problems and two practical design optimization problems to demonstrate the efficacy of MFITS.

Nonintrusive Aerodynamic Shape Optimisation with a POD-DEIM Based Trust Region Method

This work presents a strategy to build reduced-order models suitable for aerodynamic shape optimisation, resulting in a multifidelity optimisation framework. A reduced-order model (ROM) based on a discrete empirical interpolation (DEIM) method is employed in lieu of computational fluid dynamics (CFD) solvers for fast, nonlinear, aerodynamic modelling. The DEIM builds a set of interpolation points that allows it to reconstruct the flow fields from sets of basis obtained by proper orthogonal decomposition of a matrix of snapshots. The aerodynamic reduced-order model is completed by introducing a nonlinear mapping function between surface deformation and the DEIM interpolation points. The optimisation problem is managed by a trust region algorithm linking the multiple-fidelity solvers, with each subproblem solved using a gradient-based algorithm. The design space is initially restricted; as the optimisation trajectory evolves, new samples enrich the ROM. The proposed methodology is evaluated using a series of transonic viscous test cases based on wing configurations. Results show that for cases with a moderate number of design variables, the approach proposed is competitive with state-of-the-art gradient-based methods; in addition, the use of trust region methodology mitigates the likelihood of the optimiser converging to, shallower, local minima.

SigOpt Mulch: An intelligent system for AutoML of gradient boosted trees

Gradient boosted trees (GBTs) are ubiquitous models used by researchers, machine learning (ML) practitioners, and data scientists because of their robust performance, interpretable behavior, and ease-of-use. One critical challenge in training GBTs is the tuning of their hyperparameters. In practice, selecting these hyperparameters is often done manually. Recently, the ML community has advocated for tuning hyperparameters through black-box optimization and developed state-of-the-art systems to do so. However, applying such systems to tune GBTs suffers from two drawbacks. First, these systems are not model-aware, rather they are designed to apply to a generic model; this leaves significant optimization performance on the table. Second, using these systems requires domain knowledge such as the choice of hyperparameter search space, which is an antithesis to the automatic experimentation that black-box optimization aims to provide. In this paper, we present SigOpt Mulch, a model-aware hyperparameter tuning system specifically designed for automated tuning of GBTs that provides two improvements over existing systems. First, Mulch leverages powerful techniques in metalearning and multifidelity optimization to perform model-aware hyperparameter optimization. Second, it automates the process of learning performant hyperparameters by making intelligent decisions about the optimization search space, thus reducing the need for user domain knowledge. These innovations allow Mulch to identify good GBT hyperparameters far more efficiently—and in a more seamless and user-friendly way—than existing black-box hyperparameter tuning systems.

Multi-fidelity optimization of a quiet propeller based on deep deterministic policy gradient and transfer learning

In a propeller blade optimization, both aerodynamic and aeroacoustic performance were considered simultaneously. A multi-fidelity sampling scheme was adopted by Transfer Learning (TL) to improve the overall optimization efficiency. A Deep Neural Network (DNN) was selected to map the non-linear relationship between the blade parameters and the aerodynamic/aeroacoustic performance, with the optimization being achieved by implementing a deep reinforcement learning algorithm, namely, Deep Deterministic Policy Gradient (DDPG), upon which a Multi-fidelity DNN based surrogate model (TL-MFDNN) was introduced with Transfer Learning between pre-trained and retrained processes. It was found that, by comparing the TL-MFDNN surrogate model based optimization with DDPG optimization using direct CFD simulation, the overall computing cost can be saved up to 77.3% and the optimized propeller has maximum noise reduction of up to 1.69 dB, with a negligible penalty on propulsive performance.

Using physical property surrogate models to perform accelerated multi-fidelity optimization of force field parameters.

Accurate representations of van der Waals dispersion-repulsion interactions play an important role in high-quality molecular dynamics simulations. Training the force field parameters used in the Lennard Jones (LJ) potential typically used to represent these interactions is challenging, generally requiring adjustment based on simulations of macroscopic physical properties. The large computational expense of these simulations, especially when many parameters must be trained simultaneously, limits the size of training data set and number of optimization steps that can be taken, often requiring modelers to perform optimizations within a local parameter region. To allow for more global LJ parameter optimization against large training sets, we introduce a multi-fidelity optimization technique which uses Gaussian process surrogate modeling to build inexpensive models of physical properties as a function of LJ parameters. This approach allows for fast evaluation of approximate objective functions, greatly accelerating searches over parameter space and enabling the use of optimization algorithms capable of searching more globally. In this study, we use an iterative framework which performs global optimization with differential evolution at the surrogate level, followed by validation at the simulation level and surrogate refinement. Using this technique on two previously studied training sets, containing up to 195 physical property targets, we refit a subset of the LJ parameters for the OpenFF 1.0.0 (Parsley) force field. We demonstrate that this multi-fidelity technique can find improved parameter sets compared to a purely simulation-based optimization by searching more broadly and escaping local minima. Additionally, this technique often finds significantly different parameter minima that have comparably accurate performance. In most cases, these parameter sets are transferable to other similar molecules in a test set. Our multi-fidelity technique provides a platform for rapid, more global optimization of molecular models against physical properties, as well as a number of opportunities for further refinement of the technique.

Automatic Detection of Software Defects based on Machine Learning

Defects in software are one of the critical problems in software engineering community because they provide inaccurate results and negatively affect the quality and reliability of the software. These defects must be detected in the early stages of software development. Researchers had used Software Defect Detection (SDD) techniques to allow predicting module fault-proneness. By implementing the hyperparameter optimization techniques and exploiting data imbalances in predicting defects, this paper proposes and develops an SDD model with high performance and generalization capability. To classify defects in software modules, machine learning algorithms and ensemble learning techniques are used on the balanced datasets. The balanced datasets are obtained through using a hybrid of synthetic minority oversample (SMOTE) and Support Vector Machine (SVM). To obtain the optimal hyperparameters needed for the used classifiers and for the dataset balanced algorithms, Non-dominated Sorting Genetic Algorithm II (NDSGA-II) is used. To reduce the time and save other used resources, Hyperband technique, which is a multi-fidelity optimization, is used in NDSGA-II. A 10-fold Cross Validation (CV) is applied to overcome the overfitting and underfitting problems. The accuracy, recall, F-measure, and ROC AUC metrics are used to evaluate the SDD model. The results show that the proposed model predicts defects more accurately than the compared studies.

Multi-fidelity simulation optimization for production releasing in re-entrant mixed-flow shops

This research focuses on production releasing and routing allocation problems in re-entrant mixed-flow shops. Since re-entrant mixed flow shops are complex and dynamic, many studies evaluate release plans by developing discrete event simulation models and selecting the optimal solution according to the estimation results. However, a high-accurate discrete event simulation model requires a lot of computation time. In this research, we develop an effective multi-fidelity optimization method to address product release planning problems for re-entrant mixed-flow shops. The proposed method combines the advantages of rapid evaluation of analytical models and accurate evaluation of simulation models. It conducts iterative optimization using a low-fidelity mathematical estimation model to find good solutions and searches for the optimal solution via a high-fidelity simulation estimation model. Computational results of large-scale production releasing and routing allocation problems illustrate that the proposed approach is good at addressing large-scale problems in re-entrant mixed-flow shops.