Nonlinear Stochastic Optimal Control Research Articles

Deep L1 Stochastic Optimal Control Policies for Planetary Soft Landing

In this paper, a novel deep-learning-based solution is introduced to the Powered Descent Guidance problem, grounded in principles of nonlinear Stochastic Optimal Control (SOC) and Feynman–Kac theory. Our algorithm solves the problem by framing it as an L1-SOC problem for minimum fuel consumption. Additionally, it can handle practically useful control constraints and nonlinear dynamics and enforces state constraints as soft constraints. This is achieved by building off of recent work on deep forward-backward stochastic differential equations and differentiable neural-network layers for nonconvex optimization based on stochastic search. In contrast to previous approaches, our algorithm does not require convexification of the constraints or linearization of the dynamics and is empirically shown to be robust to stochastic disturbances and the initial conditions of the spacecraft. After training offline, our control policy can be activated once the spacecraft is within a prespecified radius of the landing zone and at a prespecified altitude, in other words, the base of an inverted cone with the tip at the landing zone. We demonstrate empirically that our controller can successfully and safely land all trajectories initialized in the vicinity of the base of this cone as well as with randomization in the starting velocities and spacecraft mass while minimizing fuel consumption.

Nonlinear dynamics and stochastic optimal control of a Huanglongbing epidemic model with resistance

Citrus Huanglongbing (HLB) epidemic diseases is a destructive threat to the citrus industry worldwide. In this paper, a stochastic epidemic model is proposed for the transmission of Huanglongbing in scenarios between Asian citrus psyllid and the citrus trees host. The influence of insecticide resistance is considered and the dynamical properties of the new model are mathematically analyzed. It is proved that the model is well-posed by obtaining the existence of a single global positive solution. Further, when R 2 > 1, weak persistence in the mean is obtained by developing a suitable Lyapunov function and sufficient criteria R 1 < 1 is derived for the extinction of the disease. Moreover, the analysis of stochastic optimality is used, and a proper strategy is studied to provide a theoretical basis for the prevention and control of HLB. Numerical simulations and examples are carried out to verify the main theoretical results.

Sequential convex programming for non-linear stochastic optimal control

This work introduces a sequential convex programming framework for non-linear, finitedimensional stochastic optimal control, where uncertainties are modeled by a multidimensional Wiener process. We prove that any accumulation point of the sequence of iterates generated by sequential convex programming is a candidate locally-optimal solution for the original problem in the sense of the stochastic Pontryagin Maximum Principle. Moreover, we provide sufficient conditions for the existence of at least one such accumulation point. We then leverage these properties to design a practical numerical method for solving non-linear stochastic optimal control problems based on a deterministic transcription of stochastic sequential convex programming.

Temporal deep unfolding for constrained nonlinear stochastic optimal control

This paper proposes a computational technique to solve discrete-time optimal control problems for nonlinear systems subject to stochastic disturbances, hard input constraints, and soft constraints. The approach employs the idea of deep unfolding, which is a recently developed model-based deep learning method that is applicable to iterative algorithms. By regarding each state transition of the dynamical system as an iteration step, these state transitions are unfolded into the layers of a deep neural network. This produces a computational graph that contains trainable parameters that determine control inputs. These parameters are optimized by training the deep neural network using the standard deep learning technique of backpropagation and incremental training. Then, the optimal control inputs are obtained from the optimized parameters. This provides a way to optimize the control inputs that otherwise would be difficult to obtain. The effectiveness of the proposed technique is demonstrated by numerical experiments for the maximum hands-off control problem and constrained model predictive control problems using a highly nonlinear model of a continuous stirred tank reactor.

Connection among Stochastic Hamilton–Jacobi–Bellman Equation, Path-Integral, and Koopman Operator on Nonlinear Stochastic Optimal Control

The path-integral control, which stems from the stochastic Hamilton-Jacobi-Bellman equation, is one of the methods to control stochastic nonlinear systems. This paper gives a new insight into nonlinear stochastic optimal control problems from the perspective of Koopman operators. When a finite-dimensional dynamical system is nonlinear, the corresponding Koopman operator is linear. Although the Koopman operator is infinite-dimensional, adequate approximation makes it tractable and useful in some discussions and applications. Employing the Koopman operator perspective, it is clarified that only a specific type of observable is enough to be focused on in the control problem. This fact becomes easier to understand via path-integral control. Furthermore, the focus on the specific observable leads to a natural power-series expansion; coupled ordinary differential equations for discrete-state space systems are derived. A demonstration for nonlinear stochastic optimal control shows that the derived equations work well.

Constrained model-free reinforcement learning for process optimization

Reinforcement learning (RL) is a control approach that can handle nonlinear stochastic optimal control problems. However, despite the promise exhibited, RL has yet to see marked translation to industrial practice primarily due to its inability to satisfy state constraints. In this work we aim to address this challenge. We propose an “oracle”-assisted constrained Q-learning algorithm that guarantees the satisfaction of joint chance constraints with a high probability, which is crucial for safety critical tasks. To achieve this, constraint tightening (backoffs) are introduced and adjusted using Broyden’s method, hence making the backoffs self-tuned. This results in a methodology that can be imbued into RL algorithms to ensure constraint satisfaction. We analyze the performance of the proposed approach and compare against nonlinear model predictive control (NMPC). The favorable performance of this algorithm signifies a step towards the incorporation of RL into real world optimization and control of engineering systems, where constraints are essential.

Optimal Position and Attitude Control of Quadcopter Using Stochastic Differential Dynamic Programming with Input Saturation Constraints

This paper considers the position and attitude control of a quadcopter in the presence of stochastic disturbances. Basic quadcopter dynamics is modeled as a nonlinear stochastic system described by a stochastic differential equation. Subsequently, the position and attitude control is formulated as a nonlinear stochastic optimal control problem with input saturation constraints. To solve this problem, a continuous-time stochastic differential dynamic programming (DDP) method with input saturation constraints is newly proposed. Finally, numerical simulations demonstrate the effectiveness of the proposed method by comparing it with the linear quadratic Gaussian and the deterministic DDP with input saturation constraints.

Data-Driven Method for Response Control of Nonlinear Random Dynamical Systems

AbstractThe response control of nonlinear random dynamical systems is an important but also difficult subject in scientific and industrial fields. This work merges the decomposition technique of feedback control and the data-driven identification method of stationary response probability density, converts the constrained functional extreme value problem associated with optimal control to an unconstrained optimization problem of multivariable function, and determines the optimal coefficients of preselected control terms by an optimization algorithm. This data-driven method avoids the difficulty of solving the stochastic dynamic programming equation or forward–backward stochastic differential equations encountered in classical control theories, the miss of the conservative mechanism in the nonlinear stochastic optimal control strategy, and the difficulty of judging the integrability and resonance of the controlled Hamiltonian systems encountered in the direct-control method. The application and efficacy of the data-driven method are illustrated by the random response control problems of the Duffing oscillator, van der Pol system, and a two degrees-of-freedom nonlinear system.

Constrained Q-Learning for Batch Process Optimization

Chemical process optimization and control often require satisfaction of constraints for safe operation. Reinforcement learning (RL) has been shown to be a powerful control technique that can handle nonlinear stochastic optimal control problems. Despite this promise, RL has yet to see significant translation to industrial practice due to its inability to satisfy state constraints. This work aims to address this challenge. We propose an “oracle”-assisted constrained Q-learning algorithm that guarantees the satisfaction of joint chance constraints with high probability, which is required for safety critical tasks. To that end, constraint tightening (backoffs) are introduced, which are adjusted using Broyden’s method, hence making the backoffs self-tuned. This results in a general methodology that can be integrated into approximate dynamic programming-based algorithms to guarantee constraint satisfaction with high probability. Finally, a case study is presented to compare the performance of the proposed approach with that of model predictive control (MPC). The superior performance of the proposed algorithm, in terms of constraint handling, signifies a step toward the use of RL in real world optimization and control of systems, where constraints are critical in ensuring safety.

A multilevel approach for stochastic nonlinear optimal control

We consider a class of finite-time horizon nonlinear stochastic optimal control problem. Although the optimal control admits a path integral representation for this class of control problems, efficient computation of the associated path integrals remains a challenging task. We propose a new Monte Carlo approach that significantly improves upon existing methodology. We tackle the issue of exponential growth in variance with the time horizon by casting optimal control estimation as a smoothing problem for a state-space model, and applying smoothing algorithms based on particle Markov chain Monte Carlo. To further reduce the cost, we then develop a multilevel Monte Carlo method which allows us to obtain an estimator of the optimal control with mean squared error with a cost of . In contrast, a cost of is required for the existing methodology to achieve the same mean squared error. Our approach is illustrated on two numerical examples.

On Entropy Regularized Path Integral Control for Trajectory Optimization.

In this article, we present a generalized view on Path Integral Control (PIC) methods. PIC refers to a particular class of policy search methods that are closely tied to the setting of Linearly Solvable Optimal Control (LSOC), a restricted subclass of nonlinear Stochastic Optimal Control (SOC) problems. This class is unique in the sense that it can be solved explicitly yielding a formal optimal state trajectory distribution. In this contribution, we first review the PIC theory and discuss related algorithms tailored to policy search in general. We are able to identify a generic design strategy that relies on the existence of an optimal state trajectory distribution and finds a parametric policy by minimizing the cross-entropy between the optimal and a state trajectory distribution parametrized by a parametric stochastic policy. Inspired by this observation, we then aim to formulate a SOC problem that shares traits with the LSOC setting yet that covers a less restrictive class of problem formulations. We refer to this SOC problem as Entropy Regularized Trajectory Optimization. The problem is closely related to the Entropy Regularized Stochastic Optimal Control setting which is often addressed lately by the Reinforcement Learning (RL) community. We analyze the theoretical convergence behavior of the theoretical state trajectory distribution sequence and draw connections with stochastic search methods tailored to classic optimization problems. Finally we derive explicit updates and compare the implied Entropy Regularized PIC with earlier work in the context of both PIC and RL for derivative-free trajectory optimization.

Nonlinear Stochastic Optimal Control with Input Saturation Constraints Based on Path Integrals

AbstractThis paper is concerned with a finite‐time nonlinear stochastic optimal control problem with input saturation as a hard constraint on the control input. The stochastic Hamilton‐Jacobi‐Bellman (SHJB) equation associated with this problem is derived, and a concrete solution method is also presented by extending an iteration framework of path integral stochastic optimal control. This is the first result that provides an optimal feedback control rigorously satisfying the saturation constraint as a solution to the SHJB equation. © 2020 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

Nonlinear Stochastic Optimal Control of MDOF Partially Observable Linear Systems Excited by Combined Harmonic and Wide-Band Noises

In this paper, nonlinear stochastic optimal control of multi-degree-of-freedom (MDOF) partially observable linear systems subjected to combined harmonic and wide-band random excitations is investigated. Based on the separation principle, the control problem of a partially observable system is converted into a completely observable one. The dynamic programming equation for the completely observable control problem is then set up based on the stochastic averaging method and stochastic dynamic programming principle, from which the nonlinear optimal control law is derived. To illustrate the feasibility and efficiency of the proposed control strategy, the responses of the uncontrolled and optimal controlled systems are respectively obtained by solving the associated Fokker–Planck–Kolmogorov (FPK) equation. Numerical results show the proposed control strategy can dramatically reduce the response of stochastic systems subjected to both harmonic and wide-band random excitations.

Filtering method for linear and non-linear stochastic optimal control of partially observable systems II

In this paper we studied stochastic optimal control problem based on partially observable systems (SOCPP) with a control factor on the diffusion term. A SOCPP has state and observation processes. This kind of problem has also a minimum payoff function. The payoff function should be minimized according to the partially observable systems consist of the state and observation processes. In this regard, the filtering method is used to evaluat this kind of problem and express full consideration of it. Finally, presented estimation methods are used to simulate the solution of a partially observable system corresponding to the control factor of this problem. These methods are numerically used to solve linear and nonlinear cases.

Buffer State is Enough: Simplifying the Design of QoE-Aware HTTP Adaptive Video Streaming

Recently, the prevalence of mobile devices together with the outburst of user-generated contents has fueled the tremendous growth of the Internet traffic taken by video streaming. To improve user-perceived quality-of-experience (QoE), dynamic adaptive streaming via HTTP (DASH) has been widely adopted by practical systems to make streaming smooth under limited bandwidth. However, previous DASH approaches mostly performed complicated rate adaptation based on bandwidth estimation, which has been proven to be unreliable over HTTP. In this paper, we simplify the design by only exploiting client-side buffer state information and propose a pure buffer-based DASH scheme to optimize user QoE. Our approach can not only get rid of the drawback caused by inaccurate bandwidth estimation, but also incur very limited overhead. We explicitly define an integrated user QoE model, which takes playback freezing, bitrate switch, and video quality into account, and then formulate the problem into a non-linear stochastic optimal control problem. Next, we utilize control theory to design a dynamic buffer-based controller for DASH, which determines video bitrate of each chunk to be requested and stabilize the buffer level in the meanwhile. Extensive experiments have been conducted to validate the advantages of our approach, and the results show that our approach can achieve the best performance compared with other alternative approaches.

A quadratic-cost dual control-based approach for optimal trajectory planning under uncertainty

This paper presents a dual control-based approach for optimal trajectory planning under uncertainty. The method approximately converts a nonlinear stochastic optimal control problem whose objective function is a combination of quadratic stage and/or terminal costs, with additive Gaussian process and measurement noises, into a deterministic optimal control problem by augmenting the uncertainty state defined by the square-root of the estimation error covariance matrix. The open-loop solution to the resulting deterministic optimal control reformulation is obtained using an existing pseudo-spectral method. The effectiveness of the proposed dual control-based approach is verified with two numerical examples of trajectory planning for two-dimensional robot motion with lack of observability for localization, which highlights the impact of the dual effect on the shape of designed paths.

Particle Model Predictive Control: Tractable Stochastic Nonlinear Output-Feedback MPC

We combine conditional state density construction with an extension of the Scenario Approach for stochastic Model Predictive Control to nonlinear systems to yield a novel particle-based formulation of stochastic nonlinear output-feedback Model Predictive Control. Conditional densities given noisy measurement data are propagated via the Particle Filter as an approximate implementation of the Bayesian Filter. This enables a particle-based representation of the conditional state density, or information state, which naturally merges with scenario generation from the current system state. This approach attempts to address the computational tractability questions of general nonlinear stochastic optimal control. The Particle Filter and the Scenario Approach are shown to be fully compatible and–based on the time- and measurement-update stages of the Particle Filter–incorporated into the optimization over future control sequences. A numerical example is presented and examined for the dependence of solution and computational burden on the sampling configurations of the densities, scenario generation and the optimization horizon.

LaSalle-Type Theorem and Its Applications to Infinite Horizon Optimal Control of Discrete-Time Nonlinear Stochastic Systems

Based on discrete martingale theory, the LaSalle-type theorem for general discrete-time stochastic systems is obtained and the almost sure stability is in turn discussed. As applications, infinite horizon nonlinear optimal regulator is investigated, and a dynamical programming equation called the Hamilton-Jacobi-Bellman equation is also derived for discrete-time nonlinear stochastic optimal control.

Least Squares Solution for Discrete Time Nonlinear Stochastic Optimal Control Problem with Model-Reality Differences

In this paper, an efficient computational approach is proposed to solve the discrete time nonlinear stochastic optimal control problem. For this purpose, a linear quadratic regulator model, which is a linear dynamical system with the quadratic criterion cost function, is employed. In our approach, the model-based optimal control problem is reformulated into the input-output equations. In this way, the Hankel matrix and the observability matrix are constructed. Further, the sum squares of output error is defined. In these point of views, the least squares optimization problem is introduced, so as the differences between the real output and the model output could be calculated. Applying the first-order derivative to the sum squares of output error, the necessary condition is then derived. After some algebraic manipulations, the optimal control law is produced. By substituting this control policy into the input-output equations, the model output is updated iteratively. For illustration, an example of the direct current and alternating current converter problem is studied. As a result, the model output trajectory of the least squares solution is close to the real output with the smallest sum squares of output error. In conclusion, the efficiency and the accuracy of the approach proposed are highly presented.

Filtering method for linear and non-linear stochastic optimal control of partially observable systems

This paper studies two linear methods for linear and non-linear stochastic optimal control of partially observable problem (SOCPP). At first, it introduces the general form of a SOCPP and states it as a functional matrix. A SOCPP has a payoff function which should be minimized. It also has two dynamic processes: state and observation. In this study, it is presented a deterministic method to find the control factor which has named feedback control and stated a modified complete proof of control optimality in a general SOCPP. After finding the optimal control factor, it should be substituted in the state process to make the partially observable system. Next, it introduces a linear filtering method to solve the related partially observable system with complete details. Finally, it is presented a heuristic method in discrete form for estimating non-linear SOCPPs and it is stated some examples to evaluate the performance of introducing methods.