Black-box Simulation Research Articles

High-Dimensional Bayesian Optimization Using Both Random and Supervised Embeddings

Bayesian optimization (BO) is one of the most powerful strategies to solve computationally expensive-to-evaluate blackbox optimization problems. However, BO methods are conventionally used for optimization problems of small dimension because of the curse of dimensionality. In this paper, a high-dimensionnal optimization method incorporating linear embedding subspaces of small dimension is proposed to efficiently perform the optimization. An adaptive learning strategy for these linear embeddings is carried out in conjunction with the optimization. The resulting BO method, named efficient global optimization coupled with random and supervised embedding (EGORSE), combines in an adaptive way both random and supervised linear embeddings. EGORSE has been compared to state-of-the-art algorithms and tested on academic examples with a number of design variables ranging from 10 to 600. The obtained results show the high potential of EGORSE to solve high-dimensional blackbox optimization problems, in terms of both CPU time and the limited number of calls to the expensive blackbox simulation.

Learning Hybrid Dynamics Models with Simulator-Informed Latent States

Dynamics model learning deals with the task of inferring unknown dynamics from measurement data and predicting the future behavior of the system. A typical approach to address this problem is to train recurrent models. However, predictions with these models are often not physically meaningful. Further, they suffer from deteriorated behavior over time due to accumulating errors. Often, simulators building on first principles are available being physically meaningful by design. However, modeling simplifications typically cause inaccuracies in these models. Consequently, hybrid modeling is an emerging trend that aims to combine the best of both worlds. In this paper, we propose a new approach to hybrid modeling, where we inform the latent states of a learned model via a black-box simulator. This allows to control the predictions via the simulator preventing them from accumulating errors. This is especially challenging since, in contrast to previous approaches, access to the simulator's latent states is not available. We tackle the task by leveraging observers, a well-known concept from control theory, inferring unknown latent states from observations and dynamics over time. In our learning-based setting, we jointly learn the dynamics and an observer that infers the latent states via the simulator. Thus, the simulator constantly corrects the latent states, compensating for modeling mismatch caused by learning. To maintain flexibility, we train an RNN-based residuum for the latent states that cannot be informed by the simulator.

Likelihood-free inference in state-space models with unknown dynamics

Likelihood-free inference (LFI) has been successfully applied to state-space models, where the likelihood of observations is not available but synthetic observations generated by a black-box simulator can be used for inference instead. However, much of the research up to now has been restricted to cases in which a model of state transition dynamics can be formulated in advance and the simulation budget is unrestricted. These methods fail to address the problem of state inference when simulations are computationally expensive and the Markovian state transition dynamics are undefined. The approach proposed in this manuscript enables LFI of states with a limited number of simulations by estimating the transition dynamics and using state predictions as proposals for simulations. In the experiments with non-stationary user models, the proposed method demonstrates significant improvement in accuracy for both state inference and prediction, where a multi-output Gaussian process is used for LFI of states and a Bayesian neural network as a surrogate model of transition dynamics.

Feasible set estimation under functional uncertainty by Gaussian Process modelling

In this paper we deal with the estimation of a feasible set defined by an inequality constraint on the output of a time-consuming black-box simulator. We focus on the setting where the black-box simulator takes as inputs both a set of scalar controlled variables and a functional uncontrolled variable. We then place ourselves in a probabilistic framework, modelling the functional uncontrolled variable by a random process. The inequality constraint is formulated as the expectation of the output of the simulator conditional on the values taken by the set of controlled variables. We propose an original method to solve the above feasibility problem with a reduced number of evaluations of the costly simulator. A Gaussian Process model of the simulator is learned in the joint space of controlled and uncontrolled input variables, on the basis of a set of simulations which is enriched through a sequential procedure. This procedure aims to reduce the estimation error of the feasible set by evaluating the simulator on new points chosen sequentially in the joint input space according to specific enrichment criteria. It involves as a preliminary step the reduction of the dimension of the uncontrolled input space. A variation of this strategy is also proposed, which increases adaptively the dimension of the reduced space, leading to an improvement in terms of number of calls to the simulator. The procedure we propose is compared with other sampling procedures and another modelling approach on analytical examples. Finally our methodology is implemented on an automotive industrial application. For this application, the feasible set to be recovered is the set of values of controlled variables of a gas after-treatment device, leading to the respect of pollutant emission standards of a vehicle under driving profile uncertainties.

Achieving Efficiency in Black-Box Simulation of Distribution Tails with Self-Structuring Importance Samplers

Scalable and efficient importance sampling for managing tail risks As the models employed in the realm of risk analytics and optimization become increasingly sophisticated, it is crucial that risk management tools, such as variance reduction techniques, that are typically designed for stylized models on a case by case basis evolve to scale well and gain broader applicability. In the paper titled “Achieving efficiency in black-box simulation of distribution tails with self-structuring importance samplers,” the authors take a step toward this goal by introducing a novel importance sampling scheme for estimating tail risks of objectives modeled with a diverse range of tools, including linear programs, integer linear programs, feature maps specified with neural networks, etc. Instead of explicitly tailoring the change of distribution for each specific model, as conventionally done, the paper identifies an elementary transformation of the samples. This transformation, when applied alike across a wide variety of models, yields a near-optimal reduction in variance for estimating/optimizing over tail expectations.

Optimization over decision trees: a case study for the design of stable direct-current electricity networks

In many real-world mixed-integer optimization problems from engineering, the side constraints can be subdivided into two categories: constraints which describe a certain logic to model a feasible allocation of resources (such as a maximal number of available assets, working time requirements, maintenance requirements, contractual obligations, etc.), and constraints which model physical processes and the related quantities (such as current, pressure, temperature, etc.). While the first type of constraints can often easily be stated in terms of a mixed-integer program (MIP), the second part may involve the incorporation of complex non-linearities, partial differential equations or even a black-box simulation of the involved physical process. In this work, we propose the integration of a trained tree-based classifier—a decision-tree or a random forest, into a mixed-integer optimization model as a possible remedy. We assume that the classifier has been trained on data points produced by a detailed simulation of a given complex process to represent the functional relationship between the involved physical quantities. We then derive MIP-representable reformulations of the trained classifier such that the resulting model can be solved using state-of-the-art solvers. At the hand of several use cases in terms of possible optimization goals, we show the broad applicability of our framework that is easily extendable to other tasks beyond engineering. In a detailed real-world computational study for the design of stable direct-current power networks, we demonstrate that our approach yields high-quality solutions in reasonable computation times.

Revisiting PGD Attacks for Stability Analysis of High-Dimensional Nonlinear Systems and Perception-Based Control

Many existing region-of-attraction (ROA) analysis tools find difficulty in addressing feedback systems with large-scale neural network (NN) policies and/or high-dimensional sensing modalities such as cameras. In this letter, we tailor the projected gradient descent (PGD) attack method as a general-purpose ROA analysis tool for high-dimensional nonlinear systems and end-to-end perception-based control. We show that the ROA analysis can be approximated as a constrained maximization problem such that PGD-based iterative methods can be directly applied. In the model-based setting, we show that the PGD updates can be efficiently performed using back-propagation. In the model-free setting, we propose a finite-difference PGD estimate which is general and only requires a black-box simulator for generating the trajectories of the closed-loop system given any initial state. Finally, we demonstrate the scalability and generality of our analysis tool on several numerical examples with large state dimensions or complex image observations.

Multistage optimization of a petroleum production system with material balance model

In this paper, we propose a mathematical formulation for the optimal management over time of an oil production network as a multistage optimization problem. The proposed model differs from common practice where the reservoir of the oil production network is approximated by decline curves or by black-box simulators. We model the reservoir as a controlled (non-linear) dynamical system by using material balance equations, under the standard assumptions that the fluids follow a black-oil model and that the reservoir has a tank-like behavior. The state of the dynamical system has five dimensions: the total volume of respectively oil, gas, and water in the reservoir; the total pore volume; and the reservoir pressure. We use a dynamic programming algorithm to numerically solve the multistage optimization problem on two specific instances of the general optimization problem where the state dimension can be reduced from dimension five to dimension one or two. More precisely, the first numerical application consists in optimizing the production of a dry gas reservoir which is subdivided in two tanks and which leads to a two-dimensional state (one dimension per tank), whereas the second numerical application tackles an oil reservoir with water injection which leads to a two-dimensional state. The two applications illustrate that our approach handles interconnected tanks (in the dry gas case) and that our approach allows optimization beyond first recovery of oil (in the oil with water injection case). We also provide numerical and theoretical comparisons with decline curves in the dry gas application.

Normalizing flows for likelihood-free inference with fusion simulations

Fluid-based scrape-off layer transport codes, such as UEDGE, are heavily utilized in tokamak analysis and design, but typically require user-specified anomalous transport coefficients to match experiments. Determining the uniqueness of these parameters and the uncertainties in them to match experiments can provide valuable insights to fusion scientists. We leverage recent work in the area of likelihood-free inference (‘simulation-based inference’) to train a neural network, which enables accurate statistical inference of the anomalous transport coefficients given experimental plasma profile input. UEDGE is treated as a black-box simulator and runs multiple times with anomalous transport coefficients sampled from priors, and the neural network is trained on these simulations to emulate the posterior. The neural network is trained as a normalizing flow model for density estimation, allowing it to accurately represent complicated, high-dimensional distribution functions. With a fixed simulation budget, we compare a single-round procedure to a multi-round approach that guides the training simulations toward a specific target observation. We discuss the future possibilities for use of amortized models, which train on a wide range of simulations and enable fast statistical inference for results during experiments.

Decoupled Data-Based Approach for Learning to Control Nonlinear Dynamical Systems

This article addresses the problem of learning the optimal control policy for a nonlinear stochastic dynamical. This problem is subject to the “curse of dimensionality” associated with the dynamic programming method. This article proposes a novel decoupled data-based control (D2C) algorithm that addresses this problem using a decoupled, “open-loop–closed-loop,” approach. First, an open-loop deterministic trajectory optimization problem is solved using a black-box simulation model of the dynamical system. Then, closed-loop control is developed around this open-loop trajectory by linearization of the dynamics about this nominal trajectory. By virtue of linearization, a linear quadratic regulator based algorithm can be used for this closed-loop control. We show that the performance of D2C algorithm is approximately optimal. Moreover, simulation performance suggests a significant reduction in training time compared to other state-of-the-art algorithms.

A comparison of mixed-variables Bayesian optimization approaches

Most real optimization problems are defined over a mixed search space where the variables are both discrete and continuous. In engineering applications, the objective function is typically calculated with a numerically costly black-box simulation. General mixed and costly optimization problems are therefore of a great practical interest, yet their resolution remains in a large part an open scientific question. In this article, costly mixed problems are approached through Gaussian processes where the discrete variables are relaxed into continuous latent variables. The continuous space is more easily harvested by classical Bayesian optimization techniques than a mixed space would. Discrete variables are recovered either subsequently to the continuous optimization, or simultaneously with an additional continuous-discrete compatibility constraint that is handled with augmented Lagrangians. Several possible implementations of such Bayesian mixed optimizers are compared. In particular, the reformulation of the problem with continuous latent variables is put in competition with searches working directly in the mixed space. Among the algorithms involving latent variables and an augmented Lagrangian, a particular attention is devoted to the Lagrange multipliers for which a local and a global estimation techniques are studied. The comparisons are based on the repeated optimization of three analytical functions and a beam design problem.

Framework for embedding black-box simulation into mathematical programming via kriging surrogate model applied to natural gas liquefaction process optimization

This paper presents a framework to solve the constrained black-box simulation optimization problem that arises from the optimal energy-efficient design of single-mixed refrigerant natural gas liquefaction process using reliable process simulator. Kriging surrogate model is used to introduce simple, computationally inexpensive, and effective algebraic formulations with reliable derivatives to the black-box objective and constraints functions. The algebraic surrogate optimization problem is embedded into a nonlinear programming (NLP) model in General Algebraic Modeling System (GAMS). The NLP problem is solved using efficient multi-start gradient-based optimization with CONOPT local solver to determine a candidate of decision variables for which the true functions are calculated in the rigorous simulation. The single-mixed refrigerant process is analyzed considering one-to-three-stage expansion and phase separation to assess potential energy savings. The present approach results show that more expansion stages can provide energy savings from 12.02 to 14.70 % comparing two-stage and three-stage expansion system with single-stage. This optimization framework is more effective and consistent than Particle Swarm Optimization and Genetic Algorithm given the same budget of simulation evaluations for the considered simulation optimization problems, resulting in 13.57 to 53.26 % of energy savings.

Reinforcement Learning Provides a Flexible Approach for Realistic Supply Chain Safety Stock Optimisation

Although safety stock optimisation has been studied for more than 60 years, most companies still use simplistic means to calculate necessary safety stock levels, partly due to the mismatch between existing analytical methods’ emphases on deriving provably optimal solutions and companies’ preferences to sacrifice optimal results in favour of more realistic problem settings. A newly emerging method from the field of Artificial Intelligence (AI), namely Reinforcement Learning (RL), offers promise in finding optimal solutions while accommodating more realistic problem features. Unlike analytical-based models, RL treats the problem as a black-box simulation environment mitigating against the problem of oversimplifying reality. As such, assumptions on stock keeping policy can be relaxed and a higher number of problem variables can be accommodated. While RL has been popular in other domains, its applications in safety stock optimisation remain scarce. In this paper we investigate three RL methods, namely, Q-Learning, Advantage Actor-Critic and Multi-agent Advantage Actor-Critic for optimising safety stock in a linear chain of independent agents. We find that RL can simultaneously optimise both safety stock level and order quantity parameters of an inventory policy, unlike classical safety stock optimisation models where only safety stock level is optimised while order quantity is predetermined based on simple rules. This allows RL to model more complex supply chain procurement behaviour. However, RL takes longer time to arrive at solutions, necessitating future research on identifying and improving trade-offs between the use of AI and mathematical models are needed.

Learning and planning in partially observable environments without prior domain knowledge

Planning in stochastic and partially observable environments is a central problem in artificial intelligence. To address this issue, an accurate model or a black-box simulator of the environment is usually needed in the literature. Although some recent approaches have been proposed for learning optimal behaviors under model uncertainty, prior knowledge about the environment is still required to guarantee the performance of the proposed algorithms. With the benefits of the Predictive State Representations (PSRs) approach for state representation and model prediction, in this paper, we introduce an approach for planning under partial observability with no prior domain knowledge, where an offline PSR model is firstly learned and then combined with online Monte-Carlo tree search for planning under model uncertainty. Furthermore, we also showed that with the proposed framework, the PSR models learned via other techniques, e.g., the online PSR model learning approach, can be integrated straightforwardly. By comparing with the state-of-the-art approach of planning under model uncertainty, we demonstrated the effectiveness of the proposed approaches along with the proof of their convergence.

Flexible Motion Optimization with Modulated Assistive Forces

Animated motions should be simple to direct while also being plausible. We present a flexible keyframe-based character animation system that generates plausible simulated motions for both physically-feasible and physically-infeasible motion specifications. We introduce a novel control parameterization, optimizing over internal actions, external assistive-force modulation, and keyframe timing. Our method allows for emergent behaviors between keyframes, does not require advance knowledge of contacts or exact motion timing, supports the creation of physically impossible motions, and allows for near-interactive motion creation. The use of a shooting method allows for the use of any black-box simulator. We present results for a variety of 2D and 3D characters and motions, using sparse and dense keyframes. We compare our control parameterization scheme against other possible approaches for incorporating external assistive forces.

Objective performance assessment on trainees of a VATS simulation program: A prospective single center study

Abstract Objective A prospective single center study to assess the objective impact on motion performance of a VATS simulation program on thoracic surgery trainees. Methods We developed a 6-month VATS simulation training program including exercises of progressive complexity on 3 different black box simulators: a 2D and 3D lobectomy model (Stupnik®) and a 3D perfused lobectomy model (Crabtree®). Between November 2019 and 2020, all consecutive thoracic surgery residents (study group) were prospectively enrolled in this weekly training program that was supervised by a board certified thoracic surgeon. We compared an objective performance evaluation of the study group before and after the training program by assessing movement parameters (distance in cm, time in sec) and absence of shock/extreme motion (%) on 3 simple standardized thoracoscopic exercises (peg placement on a board, rope insertion in loops and precision circle cutting) using the Simball®. Also, we determined the objective performance 6 months apart of 5 final year medical students (unexperienced controls) that were not trained. Results There were 7 residents (2 female and 5 male, median age: 29 [range: 26-34] years) who completed the 6-month VATS simulation training program. Five residents were in their first year while two had >3 year experience. The study group's objective performance improved significantly for all three movement parameters in all standardized exercises (Figure 1) after the training program. The objective performance of the unexperienced control group was comparable to the study group before training, but it remained unchanged at 6 months (p > 0.05). When comparing unexperienced and advanced residents, we observed that the training program had more impact on improving the performance for unexperienced residents (p < 0.05). Conclusion This study suggests that the implementation of a VATS simulation training program improves the objective performance of trainees compared to controls. Such programs could be interesting adjuncts for residents.

Finding and Certifying (Near-)Optimal Strategies in Black-Box Extensive-Form Games

Often---for example in war games, strategy video games, and financial simulations---the game is given to us only as a black-box simulator in which we can play it. In these settings, since the game may have unknown nature action distributions (from which we can only obtain samples) and/or be too large to expand fully, it can be difficult to compute strategies with guarantees on exploitability. Recent work (Zhang and Sandholm 2020) resulted in a notion of certificate for extensive-form games that allows exploitability guarantees while not expanding the full game tree. However, that work assumed that the black box could sample or expand arbitrary nodes of the game tree at any time, and that a series of exact game solves (via, for example, linear programming) can be conducted to compute the certificate. Each of those two assumptions severely restricts the practical applicability of that method. In this work, we relax both of the assumptions. We show that high-probability certificates can be obtained with a black box that can do nothing more than play through games, using only a regret minimizer as a subroutine. As a bonus, we obtain an equilibrium-finding algorithm with $\tilde O(1/\sqrt{T})$ convergence rate in the extensive-form game setting that does not rely on a sampling strategy with lower-bounded reach probabilities (which MCCFR assumes). We demonstrate experimentally that, in the black-box setting, our methods are able to provide nontrivial exploitability guarantees while expanding only a small fraction of the game tree.

Evolution Strategies for Approximate Solution of Bayesian Games

We address the problem of solving complex Bayesian games, characterized by high-dimensional type and action spaces, many (> 2) players, and general-sum payoffs. Our approach applies to symmetric one-shot Bayesian games, with no given analytic structure. We represent agent strategies in parametric form as neural networks, and apply natural evolution strategies (NES) [wierstra2014natural] for deep model optimization. For pure equilibrium computation, we formulate the problem as bi-level optimization, and employ NES in an iterative algorithm to implement both inner-loop best response optimization and outer-loop regret minimization. In simple games including first- and second-price auctions, it is capable of recovering known analytic solutions. For mixed equilibrium computation, we adopt an incremental strategy generation framework, with NES as strategy generator producing a finite sequence of approximate best-response strategies. We then calculate equilibria over this finite strategy set via a model-based optimization process. Both our pure and mixed equilibrium computation methods employ NES to efficiently search for strategies over the functional space, given only black-box simulation access to noisy payoff samples. We experimentally demonstrate the efficacy of all methods on two simultaneous sealed-bid auction games with distinct type distributions, and observe that the solutions exhibit qualitatively different behavior in these two environments.

Nonlinear SDRE based adaptive fuzzy control approach for age-specific drug delivery in mixed chemotherapy and immunotherapy

Determination of a near-optimal personalized drug delivery scenario in mixed chemotherapy and immunotherapy based on the age of cancer patients was presented in this paper. For this purpose, a mathematical model for cancer dynamics is considered in the form of ordinary differential equations (ODE) which contains cancer cells, tumor cells, and chemotherapy drug intervention. This model is modified by adding immunotherapy intervention effects to alter cancer dynamics. We consider two patients with known and unknown model parameters which are mentioned as a reference and unknown patients respectively. The nonlinear composite adaptive controller is introduced by compounding State Dependent Riccati Equation (SDRE) and Model Reference Adaptive Control (MRAC) techniques for achieving the drug delivery of an unknown patient. The drug delivery scenario for a reference patient with a known mathematical model and parameters is determined via the SDRE technique and then for any unknown patient, the personalized mixed therapy protocol is achieved using the treatment regimen of the reference patient as a reference model in MRAC. To reduce the side effects of chemotherapy drugs, we determine drug maximum dose limits using fuzzy control by considering the age of cancer patients. We regarded patients in four age groups of child, young, middle-aged, and old. In the proposed methodology, unknown patients are considered as a black-box simulator and the mathematical model parameters of the patient are not essential for the design of drug administration protocol and we only need patients’ age. After chemotherapy, the treatment is completed by immunotherapy to avoid cancer relapse. The effect of reference and unknown patient ages in obtaining drug delivery protocol is deeply surveyed. Numerical simulations confirm the effectiveness, robustness, and flexibility of the proposed strategy for patients of different ages. The Proposed algorithm can be an effective tool to aid oncologists in prescribing age-specific optimal treatment protocols for cancer patients.

Adaptive control designs for control-based continuation of periodic orbits in a class of uncertain linear systems

This paper proposes two novel adaptive control designs for the feedback signals used in the control-based continuation paradigm to track families of periodic orbits of periodically excited dynamical systems, including black box simulation models and physical experiments. The proposed control designs rely on modifications to the classical model reference adaptive control framework and the more recent $${\mathscr {L}}_1$$ adaptive control architecture, in which an additional low-pass filter is used to ensure guaranteed transient performance and robustness to time delays in the control input even in the limit of arbitrarily large adaptive gains. In contrast to the proportional control formulations that have been used in the literature on control-based continuation, the proposed control designs achieve stable performance with a minimum of parameter tuning. In the context of a class of linear systems with matched uncertainties, the paper demonstrates the successful integration of adaptive control feedback in control-based continuation. Specifically, the control designs are shown to ensure that the control input stabilizes the sought periodic orbits of the uncontrolled system and vanishes along these orbits, provided that an a priori unknown reference input is chosen appropriately. Numerical results obtained using the coco software package demonstrate how the combination of a nonlinear solver (Newton’s method) with the pseudo-arclength parameter continuation scheme can be used to trace the correct choice for the reference input under variations in an excitation parameter.