Robust Optimal Control Problem Research Articles

Robustness of optimal quantum annealing protocols

Abstract Noise in quantum computing devices poses a key challenge in their realization. In this paper, we study the robustness of optimal quantum annealing protocols against coherent control errors, which are multiplicative Hamlitonian errors causing detrimental effects on current quantum devices. We show that the norm of the Hamiltonian quantifies the robustness against these errors, motivating the introduction of an additional regularization term in the cost function. We analyze the optimality conditions of the resulting robust quantum optimal control problem based on Pontryagin’s maximum principle, showing that robust protocols admit larger smooth annealing sections. This suggests that quantum annealing admits improved robustness in comparison to bang-bang solutions such as the quantum approximate optimization algorithm. Finally, we perform numerical simulations to verify our analytical results and demonstrate the improved robustness of the proposed approach.

Various duality models associated with some constrained robust nonlinear optimal control problems

This paper deals with various duality results for a new class of constrained robust nonlinear control problems. Concretely, we introduce a minimisation problem associated with a vector of functionals of multiple integral type. In addition, the considered multi-objective controlled optimisation problem is governed by several mixed constraints generated by partial derivatives of second order and uncertain data. For this new class of control problems, we establish and characterise Wolfe, Mond-Weir and mixed type robust dual models. To this aim, by using the notion of convex multiple integral functional, generated by a controlled second-order Lagrangian with uncertain data, and the concept of robust weak efficient solution to the problem under study, we build a new mathematical context to state and prove the duality theorems.

Robust Policy Iteration of Uncertain Interconnected Systems With Imperfect Data

This paper investigates the robust optimal control problem of a class of continuous-time, partially linear, interconnected systems. In addition to the dynamic uncertainties resulted from the interconnected dynamic system, unknown bounded disturbances are taken into account throughout the learning process, wherein the system’s dynamics and the disturbances are assumed unknown. These challenges lead the collected online data to be imperfect. In this scenario, traditional data-driven control techniques, such as adaptive dynamic programming (ADP) and robust ADP, encounter a challenge in approximating the optimal control policy precisely due to imperfect data and computational errors. In this paper, a novel data-driven robust policy iteration method is proposed to simultaneously solve the robust optimal control problems. Without relying on the knowledge of the system’s dynamics, the external disturbances or the complete state, the implementation of the proposed method only needs to access the input and partial state information. Based on the small-gain theorem, the notions of strong unboundedness observability and input-to-output stability, it is guaranteed that the learned robust optimal control gain is stabilizing and that the solution of the closed-loop system is uniformly ultimately bounded despite the existence of dynamic uncertainties and unknown external disturbances. The simulation results reveal the efficiency and practicality of the proposed data-driven control method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by the use of reinforcement learning to improve the quality of designing adaptive optimal controllers for engineering applications. Adaptive dynamic programming methods, in particular policy iteration (PI), are widely used in solving optimal control problems. However, due to the iterative nature of PI, the approximated optimal control policy may be inaccurate and imprecise. Especially, when using imperfect system’s measurements instead of the modelling information. This can result in causing the learned control policy to deviate from the actual optimal policy. This becomes more challenging in the existence of dynamic uncertainties and unknown external disturbances which corrupt the measurements and result in imperfect data. This work investigates the conditions on the uncertainties such that the proposed novel data-driven PI algorithm is robust to system’s uncertainties, unknown external disturbances and imperfect measurements. The approximated robust optimal control policy performs robustly in the existences of imperfect data and uncertainties, and at the same time is close enough to the optimal control policy.

Off‐policy reinforcement learning algorithm for robust optimal control of uncertain nonlinear systems

AbstractIn this article an optimal control scheme is proposed to solve robust control problem for matched and unmatched system. In the proposed optimal approach the value functions are designed such that the obtained optimal control law guarantees asymptotic stability of the uncertain nonlinear system. Since the proposed robust optimal control problem is not straightforward to solve, an off‐policy reinforcement‐learning algorithm based on neural networks approximation is developed to obtain robust optimal control law iteratively. The robust control law for matched uncertain systems can be achieved via proposed off‐policy learning algorithm without requiring exact knowledge of system's dynamics. The advantages of the proposed robust optimal controller are verified by comparative simulations on an uncertain model of a car suspension system and a mathematical nonlinear model.

Time-varying gain extended state observer-based adaptive optimal control for disturbed unmanned helicopter

In this paper, the robust adaptive optimal tracking control problem is addressed for the disturbed unmanned helicopter based on the time-varying gain extended state observer (TVGESO) and adaptive dynamic programming (ADP) methods. Firstly, a novel TVGESO is developed to tackle the unknown disturbance, which can overcome the drawback of initial peaking phenomenon in the traditional linear ESO method. Meanwhile, compared with the nonlinear ESO, the proposed TVGESO possesses easier and rigorous stability analysis process. Subsequently, the optimal tracking control issue for the original unmanned helicopter system is transformed into an optimization stabilization problem. By means of the ADP and neural network techniques, the feedforward controller and optimal feedback controller are skillfully designed. Compared with the conventional backstepping approach, the designed anti-disturbance optimal controller can make the unmanned helicopter accomplish the tracking task with less energy. Finally, simulation comparisons demonstrate the validity of the developed control scheme.

Immrax: A Parallelizable and Differentiable Toolbox for Interval Analysis and Mixed Monotone Reachability in JAX

We present an implementation of interval analysis and mixed monotone interval reachability analysis as function transforms in Python, fully composable with the computational framework JAX. The resulting toolbox inherits several key features from JAX, including computational efficiency through Just-In-Time Compilation, GPU acceleration for quick parallelized computations, and Automatic Differentiability We demonstrate the toolbox’s performance on several case studies, including a reachability problem on a vehicle model controlled by a neural network, and a robust closed-loop optimal control problem for a swinging pendulum.

Semi-Infinite Programs for Robust Control and Optimization: Efficient Solutions and Extensions to Existence Constraints

Discrete-time robust optimal control problems generally take a min-max structure over continuous variable spaces, which can be difficult to solve in practice. In this paper, we extend the class of such problems that can be solved through a previously proposed local reduction method to consider those with existence constraints on the uncountable variables. We also consider the possibility of non-unique trajectories that satisfy equality and inequality constraints. Crucially, we show that the problems of interest can be cast into a standard semi-infinite program and demonstrate how to generate optimal uncertainty scenario sets in order to obtain numerical solutions. We also include examples on model predictive control for obstacle avoidance with logical conditions, control with input saturation affected by uncertainty, and optimal parameter estimation to highlight the need for the proposed extension. Our method solves each of the examples considered, producing violation-free and locally optimal solutions.

Quantum Robust Optimal Control for Linear Complex Quantum Systems With Uncertainties

The aim of this note is to study quantum robust optimal control problem for a class of linear quantum systems with uncertainties. It is shown that such problem can be converted into a mixed linear quadratic Gaussian (LQG) and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$H^{\infty }$</tex-math></inline-formula> quantum control problem. A physical model described by complex quantum stochastic differential equations (CQSDEs) is presented for the quantum system with uncertainties. The quantum system described by CQSDEs is called a complex quantum system in terms of annihilation and creation operators [1]. For an uncertain quantum system, we build a connection with a scaled linear system without uncertainty. The desired quantum robust optimal control results are derived based on the connection. We provide one numerical procedure for quantum robust optimal controller synthesis and then provide an example. The resulting controller presents robustness and optimal performance.

Robust Constrained Multi-Objective Guidance of Supersonic Transport Landing Using Evolutionary Algorithm and Polynomial Chaos

Landing of supersonic transport (SST) suffers from a large uncertainty due to its highly sensitive aerodynamic properties in the subsonic domain, as well as the wind gusts around runways. At the vehicle design stage, a landing trajectory optimization under wind uncertainty in a multi-objective solution space is desired to explore the possible trade-off in its key flight performance metrics. The proposed algorithm solves this robust constrained multi-objective optimal control problem by integrating non-intrusive polynomial chaos expansion into a constrained evolutionary algorithm. The computationally tractable optimization is made possible through the conversion of a probabilistic problem into an equivalent deterministic representation while maintaining a form of the multi-objective problem. The generated guidance trajectories achieve a significant reduction of the uncertainty in their terminal states with a marginal modification in the control history of the deterministic solutions, validating the importance of the consideration of robustness in trajectory optimization.

Automatic scenario generation for efficient solution of robust optimal control problems

AbstractExisting methods for nonlinear robust control often use scenario‐based approaches to formulate the control problem as large nonlinear optimization problems. The optimization problems are challenging to solve due to their size, especially if the control problems include time‐varying uncertainty. This paper draws from local reduction methods used in semi‐infinite optimization to solve robust optimal control problems with parametric and time‐varying uncertainty. By iteratively adding interim worst‐case scenarios to the problem, methods based on local reduction provide a way to manage the total number of scenarios. We show that the local reduction method for optimal control problems consists of solving a series of simplified optimal control problems to find worst‐case constraint violations. In particular, we present examples where local reduction methods find worst‐case scenarios that are not on the boundary of the uncertainty set. We also provide bounds on the error if local solvers are used. The proposed approach is illustrated with two case studies with parametric and additive time‐varying uncertainty. In the first case study, the number of scenarios obtained from local reduction is 101, smaller than in the case when all extreme scenarios are considered. In the second case study, the number of scenarios obtained from the local reduction is two compared to 512 extreme scenarios. Our approach was able to satisfy the constraints both for parametric uncertainty and time‐varying disturbances, whereas approaches from literature either violated the constraints or became computationally expensive.

Robust optimal control of nonlinear fractional systems

Parameter uncertainty is inevitably found in almost every system and, if is ignored, it could jeopardize the effectiveness of control method. Motivated by this issue, we consider a robust optimal control problem governed by a nonlinear fractional system with uncertain parameters, where the system sensitivity with respect to the uncertain parameters is explicitly included in the cost functional. For this problem, we first prove that the system sensitivity can be expressed as the solution of an auxiliary fractional system. Then, we propose a numerical scheme for discretizing both the original and auxiliary fractional systems, resulting in a finite-dimensional optimization problem. Furthermore, a numerical solution algorithm based on gradients of the cost functional is developed for the resulting optimization problem. Finally, numerical results for two example problems are given to demonstrate the validity and applicability of the proposed algorithm.

Robust equilibrium strategies for time-inconsistent stochastic optimal control problems with applications

In this paper, a new robust time-inconsistent stochastic optimal control problem is investigated under a general jump-diffusion model. Such a problem can be described as a sup-sup-inf problem consisting of the optimal control and the optimal selection of a probability measure that reflects the optimistic and pessimistic sentiments of the agent. Under a game-theoretic framework, the definition of the equilibrium control-measure strategy is given for the robust time-inconsistent stochastic optimal control problem and then an extended Hamilton–Jacobi–Bellman–Isaacs (HJBI) system and a verification theorem are derived for characterizing the equilibrium control-measure strategy and the corresponding equilibrium value function. Moreover, some financial optimization problems and numerical experiments are provided to illustrate the applicability of our newly derived results.

Safe, learning-based MPC for highway driving under lane-change uncertainty: A distributionally robust approach

We present a case study applying learning-based distributionally robust model predictive control to highway motion planning under stochastic uncertainty of the lane change behavior of surrounding road users. The dynamics of road users are modeled using Markov jump systems, in which the switching variable describes the desired lane of the vehicle under consideration and the continuous state describes the pose and velocity of the vehicles. We assume the switching probabilities of the underlying Markov chain to be unknown. As the vehicle is observed and thus, samples from the Markov chain are drawn, the transition probabilities are estimated along with an ambiguity set which accounts for misestimations of these probabilities. Correspondingly, a distributionally robust optimal control problem is formulated over a scenario tree, and solved in receding horizon. As a result, a motion planning procedure is obtained which through observation of the target vehicle gradually becomes less conservative while avoiding overconfidence in estimates obtained from small sample sizes. We present an extensive numerical case study, comparing the effects of several different design aspects on the controller performance and safety.

Markov decision processes under model uncertainty

AbstractWe introduce a general framework for Markov decision problems under model uncertainty in a discrete‐time infinite horizon setting. By providing a dynamic programming principle, we obtain a local‐to‐global paradigm, namely solving a local, that is, a one time‐step robust optimization problem leads to an optimizer of the global (i.e., infinite time‐steps) robust stochastic optimal control problem, as well as to a corresponding worst‐case measure. Moreover, we apply this framework to portfolio optimization involving data of the . We present two different types of ambiguity sets; one is fully data‐driven given by a Wasserstein‐ball around the empirical measure, the second one is described by a parametric set of multivariate normal distributions, where the corresponding uncertainty sets of the parameters are estimated from the data. It turns out that in scenarios where the market is volatile or bearish, the optimal portfolio strategies from the corresponding robust optimization problem outperforms the ones without model uncertainty, showcasing the importance of taking model uncertainty into account.

Stochastic discontinuous Galerkin methods for robust deterministic control of convection-diffusion equations with uncertain coefficients

We investigate a numerical behavior of robust deterministic optimal control problem subject to a convection-diffusion equation containing uncertain inputs. Stochastic Galerkin approach, turning the original optimization problem containing uncertainties into a large system of deterministic problems, is applied to discretize the stochastic domain, while a discontinuous Galerkin method is preferred for the spatial discretization due to its better convergence behavior for optimization problems governed by convection dominated PDEs. Error analysis is done for the state and adjoint variables in the energy norm, while the estimate of deterministic control is obtained in the L2-norm. Large matrix system emerging from the stochastic Galerkin method is addressed by the low-rank version of GMRES method, which reduces both the computational complexity and the memory requirements by employing Kronecker-product structure of the obtained linear system. Benchmark examples with and without control constraints are presented to illustrate the efficiency of the proposed methodology.

Robust Optimal Impulsive Blood Glucose Control Exploiting a Direct Searching Algorithm

In spite of numerous developments in blood glucose control based on the artificial pancreas, multiple daily injections (MDIs) are still the most applicable method in insulin-dependent diabetes mellitus (IDDM) subjects. These patients measure their blood glucose level (BGL) regularly and inject insulin in multiple dosages. In the present study, an efficient, robust optimal impulsive control approach is introduced to attain a safe range of blood glucose considering everyday life factors, including exercise, food intake, and circadian rhythms. This article assumes the dosages of the MDIs as control design parameters and addresses an appropriate approach for optimal dosage determination. The BGL regulation problem by the MDI program is characterized as a robust optimal impulsive control problem with uncertainties, and it is required to minimize the BGL error as the objective function. To achieve this, the polytopic form of the nonlinear glucose-insulin system is derived, and the feasible region for MDI is determined using a direct searching algorithm. Then, the optimal MDI is found by a sequential quadratic programming (SQP) approach. Regarding several uncertainties and the impulsive nature of insulin injection dosages, the proposed approach efficiently solves the BGL regulation problem. The proposed approach is evaluated by a simulated model of the blood-glucose insulin system in two regular and overfeeding scenarios with (mild, moderate) and without physical activity. The results show that the proposed method reduces the amount of injected insulin from 14% to 20% compared to a state-of-the-art approach and, simultaneously, reduces the severe hyperglycemia risk from 15% to 25% for overeating in three injection scenarios.

Robust approximate optimal control for air‐breathing hypersonic vehicle

AbstractThis paper investigates the robust approximate optimal control problem for air‐breathing hypersonic vehicle (AHV). By input‐output linearzation and introducing auxiliary variables, the high‐order nonlinear AHV dynamics are transformed to a second‐order feedback decoupling model. Then a virtual command is designed by back‐stepping strategy to guarantee the practical finite‐time convergence of tracking error. In order to drive the AHV with uncertainties to follow the virtual command, a robust approximate optimal controller based on adaptive dynamic programming (ADP) is proposed. Different from the conventional robust method, both the robustness and the input cost are taken into consideration in the proposed controller, which can optimize the control actions and hence avoid the chattering. Besides, the proposed controller adopts a single critic network design instead of actor‐critic structure, which ensures the stability of the system autonomously and simplifies the implementation procedure. The simulation results show the effectiveness of the proposed scheme.

Automatic Scenario Generation for Robust Optimal Control Problems*

Existing methods for nonlinear robust control often use scenario-based approaches to formulate the control problem as nonlinear optimization problems. Increasing the number of scenarios improves robustness, while increasing the size of the optimization problems. Mitigating the size of the problem by reducing the number of scenarios requires knowledge about how the uncertainty affects the system. This paper draws from local reduction methods used in semi-infinite optimization to solve robust optimal control problems with parametric uncertainty. We show that nonlinear robust optimal control problems are equivalent to semi-infinite optimization problems and can be solved by local reduction. By iteratively adding interim globally worst-case scenarios to the problem, methods based on local reduction provide a way to manage the total number of scenarios. In particular, we show that local reduction methods find worst case scenarios that are not on the boundary of the uncertainty set. The proposed approach is illustrated with a case study with both parametric and additive time-varying uncertainty. The number of scenarios obtained from local reduction is 101, smaller than in the case when all 214+3x192 boundary scenarios are considered. A validation with randomly drawn scenarios shows that our proposed approach reduces the number of scenarios and ensures robustness even if local solvers are used.

Robust Optimal Control With Inexact State Measurements and Adjustable Uncertainty Sets

The efficacy of robust optimal control with adjustable uncertainty sets is verified in several domains under the perfect state information setting. This paper investigates constrained robust optimal control for linear systems with linear cost functions subject to uncertain disturbances and state measurement errors that are both residing in adjustable uncertainty sets. We first show that the class of affine feedback policies of state measurements are equivalent to the class of affine feedback policies of estimated disturbances in terms of their conservativeness. Then, we formulate and solve a robust optimal control problem with adjustable uncertainty sets by considering the disturbance feedback policies. In contrast to the conventional robust optimal control, where uncertainty sets are fixed and known a priori, the uncertainty sets themselves are regarded as decision variables in our design. In particular, given the metrics for evaluating the optimal size/shape of the polyhedral uncertainty sets, a bilinear optimization problem is formulated to decide the optimal size/shape of uncertainty sets and a corresponding optimal control policy to robustly guarantee that the system will respect its constraints for all admissible uncertainties. In addition, we introduce a convex approximation for the proposed scheme to provide a computationally efficient inner approximation of the original problem. The proposed scheme is illustrated by numerical simulation of a building temperature control problem to demonstrate its effectiveness.

Robust Optimal Well Control using an Adaptive Multigrid Reinforcement Learning Framework

Reinforcement learning (RL) is a promising tool for solving robust optimal well control problems where the model parameters are highly uncertain and the system is partially observable in practice. However, the RL of robust control policies often relies on performing a large number of simulations. This could easily become computationally intractable for cases with computationally intensive simulations. To address this bottleneck, an adaptive multigrid RL framework is introduced which is inspired by principles of geometric multigrid methods used in iterative numerical algorithms. RL control policies are initially learned using computationally efficient low-fidelity simulations with coarse grid discretization of the underlying partial differential equations (PDEs). Subsequently, the simulation fidelity is increased in an adaptive manner towards the highest fidelity simulation that corresponds to the finest discretization of the model domain. The proposed framework is demonstrated using a state-of-the-art, model-free policy-based RL algorithm, namely the proximal policy optimization algorithm. Results are shown for two case studies of robust optimal well control problems, which are inspired from SPE-10 model 2 benchmark case studies. Prominent gains in computational efficiency are observed using the proposed framework, saving around 60-70% of the computational cost of its single fine-grid counterpart.