Robust Optimal Control Problem Research Articles

Preconditioners for robust optimal control problems under uncertainty

AbstractThe discretization of robust quadratic optimal control problems under uncertainty using the finite element method and the stochastic collocation method leads to large saddle‐point systems, which are fully coupled across the random realizations. Despite its relevance for numerous engineering problems, the solution of such systems is notoriously challenging. In this manuscript, we study efficient preconditioners for all‐at‐once approaches using both an algebraic and an operator preconditioning framework. We show in particular that for values of the regularization parameter not too small, the saddle‐point system can be efficiently solved by preconditioning in parallel all the state and adjoint equations. For small values of the regularization parameter, robustness can be recovered by the additional solution of a small linear system, which however couples all realizations. A mean approximation and a Chebyshev semi‐iterative method are proposed to solve this reduced system. We consider a random elliptic partial differential equation whose diffusion coefficient is modeled as an almost surely continuous and positive random field, though not necessarily uniformly bounded and coercive. We further provide estimates of the dependence of the spectrum of the preconditioned system matrix on the statistical properties of the random field and on the discretization of the probability space. Such estimates involve either the first or second moment of the random variables and , where is the spatial domain. The theoretical results are confirmed by numerical experiments, and implementation details are further addressed.

Stochastic optimal well control in subsurface reservoirs using reinforcement learning

We present a case study of model-free reinforcement learning (RL) framework to solve stochastic optimal control for a predefined parameter uncertainty distribution and partially observable system. We focus on robust optimal well control problem which is a subject of intensive research activities in the field of subsurface reservoir management. For this problem, the system is partially observed since the data is only available at well locations. Furthermore, the model parameters are highly uncertain due to sparsity of available field data. In principle, RL algorithms are capable of learning optimal action policies – a map from states to actions – to maximize a numerical reward signal. In deep RL, this mapping from state to action is parameterized using a deep neural network. In the RL formulation of the robust optimal well control problem, the states are represented by saturation and pressure values at well locations while the actions represent the valve openings controlling the flow through wells. The numerical reward refers to the total sweep efficiency and the uncertain model parameter is the subsurface permeability field. The model parameter uncertainties are handled by introducing a domain randomization scheme that exploits cluster analysis on its uncertainty distribution. We present numerical results using two state-of-the-art RL algorithms, proximal policy optimization (PPO) and advantage actor–critic (A2C), on two subsurface flow test cases representing two distinct uncertainty distributions of permeability field. The results were benchmarked against optimization results obtained using differential evolution algorithm. Furthermore, we demonstrate the robustness of the proposed use of RL by evaluating the learned control policy on unseen samples drawn from the parameter uncertainty distribution that were not used during the training process.

Robust optimal control of a network-based SIVS epidemic model with time delay

In this paper, we investigate robust optimal control of a network-based SIVS epidemic system with an uncertain time delay, which contains time-dependent and degree-dependent vaccination controls. The dynamic behaviours of the delayed system with time-independent controls are discussed. We propose a robust optimal control problem, in which the accumulated density of infected individuals and control costs, and the sensitivity to the uncertain time delay are all involved. Since the objective contains a non-standard sensitivity term, it is difficult to solve this problem using conventional optimization techniques. By introducing an auxiliary initial value system, we transform this problem into a standard form. Furthermore, a numerical scheme which combines the control parametrization scheme with the interior point algorithm is constructed. Finally, numerical simulations with different time delay and weight factor are carried out. We find that, in the case of the basic reproduction number being smaller than one, the optimal control schemes can effectively shorten the disease duration for larger time delays; in face of the disease pandemic, greater control efforts should be made in time at the beginning of disease outbreak, especially for no time delay; in either case, smaller weight factors contribute to minor vibrations of optimal controls and lesser disease survivals. These results can provide a new insight into controlling diseases.

Robust Discrete-Time Pontryagin Maximum Principle on Matrix Lie Groups

This article considers a discrete-time robust optimal control problem on matrix Lie groups. The underlying system is assumed to be perturbed by exogenous unmeasured bounded disturbances, and the control problem is posed as a min–max optimal control wherein the disturbance is the adversary and tries to maximize a cost that the control tries to minimize. Assuming the existence of a saddle point in the problem, we present a version of the Pontryagin maximum principle (PMP) that encapsulates first-order necessary conditions that the optimal control and disturbance trajectories must satisfy. This PMP features a saddle point condition on the Hamiltonian and a set of backward difference equations for the adjoint dynamics. We also present a special case of our result on Euclidean spaces. We conclude with applying the PMP to robust version of single axis rotation of a rigid body.

Robust Actor–Critic Learning for Continuous-Time Nonlinear Systems With Unmodeled Dynamics

This article considers the robust optimal control problem for a class of nonlinear systems in the presence of unmodeled dynamics. An adaptive optimal controller is designed using the online actor–critic learning and is robustified against unmodeled dynamics. To deal with unmodeled dynamics, an auxiliary signal with the system state as its input signal is designed to capture the input-to-state stability. In addition to the critic network for value function approximation, a novel robustifying term is developed and introduced into the actor network to ensure robustness during the learning process. It is shown that both the actor and the critic weights learning converge to their optimal values while guaranteeing the boundedness of all the signals in the closed loop. Simulation examples are conducted to verify the efficacy of the presented scheme.

Robust optimal control of compartmental models in epidemiology: Application to the COVID-19 pandemic

In this paper, a spectral approach is used to formulate and solve robust optimal control problems for compartmental epidemic models, allowing the uncertainty propagation through the optimal control model to be represented by a polynomial expansion of its stochastic state variables. More specifically, a statistical moment-based polynomial chaos expansion is employed. The spectral expansion of the stochastic state variables allows the computation of their main statistics to be carried out, resulting in a compact and efficient representation of the variability of the optimal control model with respect to its random parameters. The proposed robust formulation provides the designers of the optimal control strategy of the epidemic model the capability to increase the predictability of the results by simply adding upper bounds on the variability of the state variables. Moreover, this approach yields a way to efficiently estimate the probability distributions of the stochastic state variables and conduct a global sensitivity analysis. To show the practical implementation of the proposed approach, a mathematical model of COVID-19 transmission is considered. The numerical results show that the spectral approach proposed to formulate and solve robust optimal control problems for compartmental epidemic models provides healthcare systems with a valuable tool to mitigate and control the impact of infectious diseases.

Robust bi-objective optimal control of tungiasis diseases

Tungiasis, a neglected seasonal disease, leads to long-term injury and life threats to humans in developing countries. In this paper, we investigate the optimal control of tungiasis disease with an uncertain parameter, where both epidemic prevention and economic concerns are considered. Based on the life cycle of jiggers and propagation process of the disease, a human-jigger parasite model with control schemes for humans and jiggers is established. The effect of controls on the control reproduction number is discussed. A robust bi-objective optimal control problem is proposed, in which the accumulated number of infected humans and control cost, and their sensitivities to the uncertain parameter are all in the vector objective. Since the objective vector contains non-standard sensitivity terms, it is difficult to solve this problem using conventional optimization techniques. By introducing an auxiliary initial value system, we transform this problem into a standard form. Furthermore, a numerical method which combines a modified normal boundary intersection scheme with the interior point scheme is constructed. Finally, numerical simulations for different seasons are carried out using actual data of humans and jiggers in Nigeria. From all results, we conclude that enhancing the jigger adulticiding effort can significantly reduce the control reproduction number; the intervention control measures should be carried out in time, especially in the dry season; the obtained Pareto points can provide decision-makers with a trade-off between the two goals and choose an appropriate policy to implement.

Robust optimal control of connected and automated vehicle platoons through improved particle swarm optimization

This paper addresses the robust optimal control problem for connected and automated vehicle platoons subject simultaneously to uncertain parasitic actuation lag and input delays. First, by adopting the constant time headway policy (CTHP) for spacing, the intervehicular spacing error propagation transfer function is derived between adjacent vehicles downstream under a decentralized control law. Second, the Hurwitz stability region for the derived spacing error propagation transfer function is obtained in a unified stability analysis framework for the time headway and controller gains by utilizing the signature conditions for real polynomials. The robust optimal control problem is formulated as a min–max optimization problem, and then an improved multi-agent based particle swarm optimization algorithm is developed to seek the optimal values of the design parameters in the derived stability region such that a weighted objective function is minimized. Finally, simulation experiments are performed on a numerical example to illustrate the effectiveness of the proposed design approach.

Robust mean field social control problems with applications in analysis of opinion dynamics

This paper investigates the social optimality of linear quadratic mean field control systems with unmodelled dynamics. The objective of agents is to optimise the social cost, which is the sum of costs of all agents. By variational analysis and direct decoupling methods, the social optimal control problem is analysed, and two equivalent auxiliary robust optimal control problems are obtained for a representative agent. By solving the auxiliary problem with consistent mean field approximations, a set of decentralised strategies is designed, and its asymptotic social optimality is further proved. Next, the results are applied into the study of opinion dynamics in social networks. The evolution of opinions is analysed over finite and infinite horizons, respectively. All opinions are shown to reach agreement with the average opinion in a probabilistic sense. Finally, local interactions among multiple sub-populations are examined via graphon theory.

Robust optimal control for a batch nonlinear enzyme-catalytic switched time-delayed process with noisy output measurements

In this paper, we consider a nonlinear switched time-delayed (NSTD) system with an unknown time-varying function describing the batch culture. The output measurements are noisy. According to the actual fermentation process, this time-varying function appears in the form of a piecewise-linear function with unknown kinetic parameters and switching times. The quantitative definition of biological robustness is given to overcome the difficulty of accurately measuring intracellular material concentrations. Our main goal is to estimate these unknown quantities by using noisy output measurements and biological robustness. This estimation problem is formulated as a robust optimal control problem (ROCP) governed by the NSTD system subject to continuous state inequality constraints. The ROCP is approximated as a sequence of nonlinear programming subproblems by using some techniques. Due to the highly complex nature of these subproblems, we propose a hybrid parallel algorithm, based on Nelder–Mead method, simulated annealing and the gradients of the constraint functions, for solving these subproblems. The paper concludes with simulation results.

Data-driven robust model predictive control framework for stem water potential regulation and irrigation in water management

In this work, we propose a data-driven robust model predictive control (DDRMPC) framework that utilizes stem water potential (SWP) as a basis for effective irrigation control of high value-added crops. By linearizing and discretizing a nonlinear dynamic model of water dynamics, we develop a state-space model that predicts the dynamic state of SWP. In the model, soil, root, and stem are the three compartments to describe current water status of the system. In addition, evapotranspiration and precipitation are the driving force and the water inlet, respectively. A robust optimal control problem is formulated to maintain SWP above a safe level to avoid detrimental effects on crops. To describe the uncertainty within prediction errors of evapotranspiration and precipitation, a data-driven approach is adopted, which achieves a desirable tradeoff between constraint satisfaction and water saving. Meanwhile, it is shown that the proposed DDRMPC ensures both feasibility and stability. A case study based on almond tree is carried out to showcase the effectiveness of the DDRMPC strategy relative to on–off control, certainty equivalent MPC and robust MPC. In particular, the control of tree stem water potential through DDRMPC can reduce the water consumption by 7.9% compared with on–off control while maintaining zero probability of constraint violation.

Zero-Order Robust Nonlinear Model Predictive Control with Ellipsoidal Uncertainty Sets

In this paper, we propose an efficient zero-order algorithm that can be used to compute an approximate solution to robust optimal control problems (OCP) and robustified nonconvex programs in general. In particular, we focus on robustified OCPs that make use of ellipsoidal uncertainty sets and show that, with the proposed zero-order method, we can efficiently obtain suboptimal, but robustly feasible solutions. The main idea lies in leveraging an inexact sequential quadratic programming (SQP) algorithm in which an advantageous sparsity structure is enforced. The obtained sparsity allows one to eliminate the variables associated with the propagation of the ellipsoidal uncertainty sets and to solve a reduced problem with the same dimensionality and sparsity structure of a nominal OCP. The inexact algorithm can drastically reduce the computational complexity of the SQP iterations (e.g., in the case where a structure exploiting interior-point method is used to solve the underlying quadratic programs (QPs), from O(N ⋅ (n x 6 + n u 3 )) to (n x 6 + n u 3 )). Moreover, standard embedded QP solvers for nominal problems can be leveraged to solve the reduced QP.

Learning Control Supported by Dynamic Event Communication Applying to Industrial Systems

For the practical control system, the controller is normally implemented on a digital platform with a time-triggered scheme. This scheme maybe produces redundant control and resources wasting, and hence, an event-triggered scheme is gradually favored. In this article, the robust learning control scheme is proposed aiming at a class of disturbed control systems, in which the system information is processed by a novel dynamic event communication. First, the robust optimal control problem with external disturbances is redescribed as a zero-sum differential game, and with integral reinforcement learning, a model-independent weight tuning law is devised for a critic neural network. Then, in order to further reduce the computational burden, an additional dynamic variable is put forward to incorporate the past triggering information. The application of a single-link joint arm system demonstrates that the proposed scheme can guarantee learning performance and robust control effect, along with larger triggering intervals. Finally, the load frequency control problem of single-area power system is studied. On one hand, the comparative results of five control schemes reveal that the dynamic event scheme can achieve the better frequency response at the lowest information transmission rate. On the other hand, the advantages of the proposed method are illustrated by comparing with other three event-triggered schemes.

A framework for robust quadratic optimal control with parametric dynamic model uncertainty using polynomial chaos

AbstractWe propose a framework tailored to robust optimal control (OC) problems subject to parametric model uncertainty of system dynamics. First, we identify a generic class of robust objective kernels that are based on the class of deterministic quadratic objectives. It is demonstrated how such kernels can be expressed as a function of the stochastic moments of the state and how the objective terms relate to the robustness and performance of the optimal solution. Second, we engage the generalized polynomial chaos (gPC) framework to propagate uncertainty along the state trajectory. Integrating both frameworks makes way to reformulate the problem as a deterministic OC problem in function of the gPC expansion coefficients that can be solved using existing methods. We apply the framework to solve the problem of robust optimal startup behavior of a nonlinear mechanical drivetrain that is subject to a bifurcation in its dynamics.

Robust optimal control problem with multiple characteristic time points in the objective for a batch nonlinear time-varying process using parallel global optimization

In this paper, we consider a nonlinear time-varying dynamical (NTVD) system with uncertain system parameters assigned to their nominal values in batch culture of glycerol bioconversion to 1,3-propanediol induced by Klebsiella pneumoniae. Some important properties of the NTVD system are discussed. Our goal is to choose a time-varying function for the NTVD system. Thus, an optimal control problem (OCP) governed by the NTVD system and subject to continuous state inequality constraints arising from engineering specifications is proposed, where the time-varying function is the control function to be chosen such that system cost (the relative error between experimental data and the simulated output of the system) and system robustness (robustness of the system with respect to uncertain system parameters) is optimized. Based on the actual fermentation process, the time-varying function is specified by a four-piecewise linear function with unknown kinetic parameters and switching instants. The resulting OCP is approximated as a sequence of nonlinear mathematical programming subproblems by the time-scaling transformation, the constraint transcription and the locally smoothing approximation techniques. A parallel global optimization algorithm, based on a novel combination of limited information particle swarm optimization and local search strategy, is then developed to solve these subproblems. Numerical results show the effectiveness and applicability of our proposed algorithm.

Robust Optimal Formation Control of Heterogeneous Multi-Agent System via Reinforcement Learning

In this paper, a distributed robust optimal formation control problem is studied based on reinforcement learning for the heterogeneous multi-agent system with partial unknown system parameters. The formation system is subjected to equivalent disturbances containing parameter uncertainties and external disturbances. The proposed robust optimal controller consists of a nominal controller and a robust compensator. For the nominal controller, the reinforcement learning algorithm is proposed to obtain the optimal control input. For the robust compensator, the reinforcement learning algorithm is firstly used to identify the unknown dynamic parameters and then the robust compensator is designed to restrain the equivalent disturbances in the formation system. The robustness properties of the global multi-agent system are proven. A simulation of heterogeneous rotorcrafts is provided to verify the effectiveness of the proposed method.

Neural network based integral sliding mode optimal flight control of near space hypersonic vehicle

In this paper, based on the integral sliding mode method and adaptive dynamic programming (ADP) algorithm, a robust optimal tracking control scheme is presented for near space hypersonic vehicle (NSHV) system in the presence of unknown modeling error, external disturbance, and input saturation. Firstly, combining neural network, auxiliary system and integral sliding mode methods, an adaptive integral sliding mode control (AISMC) law is designed to guarantee system trajectories tend to a defined integral sliding surface and the effects of modeling uncertainty, external disturbance, and control input saturation are eliminated. Then, the robust optimal tracking control problem of original system is converted into the optimal control problem of a nominal system, and an ADP method with single critic network is utilized to acquire the corresponding optimal controller. Furthermore, Lyapunov analysis method shows that the overall control input which contains AISMC law and optimal controller can ensure all the signals in closed-loop system are stable in the sense of uniform ultimate boundedness (UUB). Finally, simulation results about attitude flight control of NSHV are given to verify the effectiveness of the proposed control scheme.

Backward-forward reachable set splitting for state-constrained differential games

This paper is about a set-based computing method for solving a general class of two-player zero-sum Stackelberg differential games. We assume that the game is modeled by a set of coupled nonlinear differential equations, which can be influenced by the control inputs of the players. Here, each of the players has to satisfy their respective state and control constraints or loses the game. The main contribution is a backward-forward reachable set splitting scheme, which can be used to derive numerically tractable conservative approximations of such two player games. In detail, we introduce a novel class of differential inequalities that can be used to find convex outer approximations of these backward and forward reachable sets. This approach is worked out in detail for ellipsoidal set parameterizations. Our numerical examples illustrate not only the effectiveness of the approach, but also the subtle differences between standard robust optimal control problems and more general constrained two-player zero-sum Stackelberg differential games.

Pricing and hedging in incomplete markets with model uncertainty

We search for a trading strategy and the associated robust price of unhedgeable assets in incomplete markets under the acknowledgement of model uncertainty. Our set-up is that we postulate the management of a firm that wants to maximise the expected surplus by choosing an optimal investment strategy. Furthermore, we assume that the firm is concerned about model misspecification. This robust optimal control problem under model uncertainty leads to (i) risk-neutral pricing for the traded risky assets, and (ii) adjusting the drift of the nontraded risk drivers in a conservative direction. The direction depends on the firm’s long or short position, and the adjustment that ensures a robust strategy leads to what is known as “actuarial” or “prudential” pricing. Our results extend to a multivariate setting. We prove existence and uniqueness of the robust price in an incomplete market via the link between the semilinear partial differential equation and backward stochastic differential equations for viscosity and classical solutions.

Robust optimal reinsurance and investment strategies for an AAI with multiple risks

This paper studies the robust optimal reinsurance and investment problem for an ambiguity averse insurer (abbr. AAI). The AAI sells insurance contracts and has access to proportional reinsurance business. The AAI can invest in a financial market consisting of four assets: one risk-free asset, one bond, one inflation protected bond and one stock, and has different levels of ambiguity aversions towards the risks. The goal of the AAI is to seek the robust optimal reinsurance and investment strategies under the worst case scenario. Here, the nominal interest rate is characterized by the Vasicek model; the inflation index is introduced according to the Fisher’s equation; and the stock price is driven by the Heston’s stochastic volatility model. The explicit forms of the robust optimal strategies and value function are derived by introducing an auxiliary robust optimal control problem and stochastic dynamic programming method. In the end of this paper, a detailed sensitivity analysis is presented to show the effects of market parameters on the robust optimal reinsurance policy, the robust optimal investment strategy and the utility loss when ignoring ambiguity.