Deterministic Control Problem Research Articles

Trajectory Optimization of Chance-Constrained Nonlinear Stochastic Systems for Motion Planning Under Uncertainty

In this article, we present generalized polynomial chaos-based sequential convex programming (gPC-SCP) to compute a suboptimal solution for a continuous-time chance-constrained stochastic nonlinear optimal control (SNOC) problem. The approach enables motion planning for robotic systems under uncertainty. The gPC-SCP method involves two steps. The first step is to derive a surrogate problem of deterministic nonlinear optimal control (DNOC) with convex constraints by using gPC expansion and the distributionally robust convex subset of the chance constraints. The second step is to solve the DNOC problem using sequential convex programming for trajectory generation and control. We prove that in the unconstrained case, the optimal value of the DNOC converges to that of SNOC asymptotically and that any feasible solution of the constrained DNOC is a feasible solution of the chance-constrained SNOC. We also present the predictor–corrector extension (gPC-SCP <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^\text{PC}$</tex-math></inline-formula> ) for real-time motion trajectory generation in the presence of stochastic uncertainty. In the gPC-SCP <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^\text{PC}$</tex-math></inline-formula> method, we first predict the uncertainty using the gPC method and then optimize the motion plan to accommodate the uncertainty. We empirically demonstrate the efficacy of the gPC-SCP and the gPC-SCP <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^\text{PC}$</tex-math></inline-formula> methods for the following two test cases: first, collision checking under uncertainty in actuation and physical parameters and second, collision checking with stochastic obstacle model for 3DOF and 6DOF robotic systems. We validate the effectiveness of the gPC-SCP method on the 3DOF robotic spacecraft testbed.

Optimal control for stochastic differential equations and related Kolmogorov equations

&lt;p style='text-indent:20px;'&gt;This paper concerns a stochastic optimal control problem with feedback Markov inputs. The problem is reduced to a deterministic optimal control problem for a Kolmogorov equation where the control for the deterministic problem is of open-loop type. The existence of an optimal control is proved for the deterministic control problem in a particular case. A maximum principle and some first order necessary optimality conditions are derived. Some examples and comments are discussed.&lt;/p&gt;

A survey of average cost problems in deterministic discrete-time control systems

This paper concerns optimal control problems for infinite-horizon discrete-time deterministic systems with the long-run average cost (AC) criterion. This optimality criterion can be traced back to a paper by Bellman [6] for a class of Markov decision processes (MDPs). We present a survey of some of the main approaches to study the AC problem, namely, the AC optimality (or dynamic programming) equation, the steady state approach, and the vanishing discount approach, emphasizing the difference between the deterministic control problem and the corresponding (stochastic) MDP. Several examples illustrate these approaches and related results. We also state some open problems.

Solution to Forward–Backward Stochastic Differential Equations With Random Coefficients and Application to Deterministic Optimal Control

We address forward–backward stochastic differential equations (FBSDEs) with random coefficients. Differently from previous approaches, we consider FBSDEs with coefficients that are random and correlated with forward and backward variables. In addition, expectation terms appear in the forward equation, and this makes challenging to solve coupled FBSDEs, e.g., those coming from corporations’ fiscal policy planning. Using a discretization approach, an analytic solution to the FBSDEs is given in terms of coupled stochastic Riccati-type differential equations. We also apply our results to optimal deterministic control problem for Itô stochastic systems with both random and deterministic coefficients. Our results show that the discretization approach is a powerful tool to decouple complex FBSDEs.

Application of the Tomtit Flock Metaheuristic Optimization Algorithm to the Optimal Discrete Time Deterministic Dynamical Control Problem

A new bio-inspired method for optimizing the objective function on a parallelepiped set of admissible solutions is proposed. It uses a model of the behavior of tomtits during the search for food. This algorithm combines some techniques for finding the extremum of the objective function, such as the memory matrix and the Levy flight from the cuckoo algorithm. The trajectories of tomtits are described by the jump-diffusion processes. The algorithm is applied to the classic and nonseparable optimal control problems for deterministic discrete dynamical systems. This type of control problem can often be solved using the discrete maximum principle or more general necessary optimality conditions, and the Bellman’s equation, but sometimes it is extremely difficult or even impossible. For this reason, there is a need to create new methods to solve these problems. The new metaheuristic algorithm makes it possible to obtain solutions of acceptable quality in an acceptable time. The efficiency and analysis of this method are demonstrated by solving a number of optimal deterministic discrete open-loop control problems: nonlinear nonseparable problems (Luus–Tassone and Li–Haimes) and separable problems for linear control dynamical systems.

A Yosida's parametrix approach to Varadhan's estimates for a degenerate diffusion under the weak Hörmander condition

We adapt and extend Yosida's parametrix method, originally introduced for the construction of the fundamental solution to a parabolic operator on a Riemannian manifold, to derive Varadhan-type asymptotic estimates for the transition density of a degenerate diffusion under the weak Hörmander condition. This diffusion process, widely studied by Yor in a series of papers, finds direct application in the study of a class of path-dependent financial derivatives known as Asian options. We obtain the Varadhan formula−2log⁡p(t,x;T,y)Ψ(t,x;T,y)→1,as T−t→0+, where p denotes the transition density and Ψ denotes the optimal cost function of a deterministic control problem associated to the diffusion. We provide a partial proof of this formula, and present numerical evidence to support the validity of an intermediate inequality that is required to complete the proof. We also derive an asymptotic expansion of the cost function Ψ, expressed in terms of elementary functions, which is useful in order to design efficient approximation formulas for the transition density.

Optimal Open-Loop Control of Discrete Deterministic Systems by Application of the Perch School Metaheuristic Optimization Algorithm

A new hybrid metaheuristic method for optimizing the objective function on a parallelepiped set of admissible solutions is proposed. It mimics the behavior of a school of river perch when looking for food. The algorithm uses the ideas of several methods: a frog-leaping method, migration algorithms, a cuckoo algorithm and a path-relinking procedure. As an application, a wide class of problems of finding the optimal control of deterministic discrete dynamical systems with a nonseparable performance criterion is chosen. For this class of optimization problems, it is difficult to apply the discrete maximum principle and its generalizations as a necessary optimality condition and the Bellman equation as a sufficient optimality condition. The desire to extend the class of problems to be solved to control problems of trajectory bundles and stochastic problems leads to the need to use not only classical adaptive random search procedures, but also new approaches combining the ideas of migration algorithms and swarm intelligence methods. The efficiency of this method is demonstrated and an analysis is performed by solving several optimal deterministic discrete control problems: two nonseparable problems (Luus–Tassone and LiHaimes) and five classic linear systems control problems with known exact solutions.

Stationary distribution extinction and optimal control for the stochastic hepatitis B epidemic model with partial immunity

In this paper, a stochastic model (with random noise transmission) is designed. The model possesses substantial potential to describe the dynamical behavior of the Hepatitis B (HBV) virus and it’s control by applying the strategy of vaccinating an offspring. The number of basic reproductive is calculated and proved that the system holds some sharp threshold properties. It is investigated that the model has a bounded, unique and positive solution subject to initial positive data. Furthermore, the stability of the investigated system has been presented by using stochastic Lyapunov functional theory. Stationary distribution and extinction of the infection are examined by providing sufficient conditions. To control the spread of the disease through some external measures, we used optimal control theory and analyzed stochastic as well as deterministic control problems. For further verification of the obtained analytical results, additional graphical solutions have been presented for the ease of understanding. This study may provide a strong theoretical basis for understanding worldwide chronic infectious diseases.

"A spectral criterion for the existence of the stabilizing solution of a class of Riccati type differential equations with periodic coefficients"

"In this paper, we investigate the existence and uniqueness of a stabilizing solution to a periodic backward nonlinear differential equation. This class of nonlinear equations includes as special cases many of the continuous-time Riccati equations arising both in deterministic and stochastic linear quadratic (LQ) type control problems."

Stochastic homogenization of deterministic control problems

In this paper we study homogenization of a class of control problems in a stationary and ergodic random environment. This problem has been mostly studied in the calculus of variations setting in connection to the homogenization of the Hamilton–Jacobi equation. We extend the result to control problems with more general state dynamics and macroscopically inhomogeneous Lagrangians. Moreover, our approach proves homogenization under weaker growth assumptions on the Lagrangian, even in the well-studied calculus of variations setting.

Optimal Consumption Under a Habit-Formation Constraint

We propose a new optimal consumption model in which the degree of addictiveness of habit formation is directly controlled through a constraint on admissible consumption. In particular, we assume that the individual is unwilling to consume at a rate below a certain proportion 0<α≤1 of her consumption habit, which is the exponentially-weighted average of past consumption rates. α=1 prohibits the habit process to decrease and corresponds to the completely addictive model. α=0 makes the habit-formation constraint moot and corresponds to the non-addictive model. 0<α<1 leads to partially addictive models, with the level of addictiveness increasing with α. In contrast to the existing habit-formation literature, our constraint cannot be incorporated in the objective function through infinite marginal utility, and alters the analysis of the corresponding control problem. Assuming that the individual invests in a risk-free market, we formulate and solve an infinite-horizon, deterministic control problem to maximize the discounted CRRA utility of the individual's consumption-to-habit process subject to the said habit-formation constraint. We derive the optimal consumption policies explicitly in terms of the solution of a nonlinear free-boundary problem, which we analyze in detail. Impatient individuals (or, equivalently, those with more addictive habits) always consume above the minimum rate; thus, they eventually attain the minimum wealth-to-habit ratio. Patient individuals (or, equivalently, those with less addictive habits) consume at the minimum rate if their wealth-to-habit ratio is below a threshold, and above it otherwise. By consuming patiently, these individuals maintain a wealth-to-habit ratio that is greater than the minimum acceptable level. Additionally, we prove that the optimal consumption path is hump-shaped if the initial wealth-to-habit ratio is either: (1) larger than a high threshold; or (2) below a low threshold and the agent is more risk seeking (that is, less risk averse). Thus, we provide a simple explanation for the consumption hump observed by various empirical studies.

Representation of limit values for nonexpansive stochastic differential games

A classical problem in ergodic control theory consists in the study of the limit behaviour of λVλ(⋅) as λ↘0, when Vλ is the value function of a deterministic or stochastic control problem with discounted cost functional with infinite time horizon and discount factor λ. We study this problem for the lower value function Vλ of a stochastic differential game with recursive cost, i.e., the cost functional is defined through a backward stochastic differential equation with infinite time horizon. But unlike the ergodic control approach, we are interested in the case where the limit can be a function depending on the initial condition. For this we extend the so-called non-expansivity assumption from the case of control problems to that of stochastic differential games and we derive that λVλ(⋅) is bounded and Lipschitz uniformly with respect to λ>0. Using PDE methods and assuming radial monotonicity of the Hamiltonian of the associated Hamilton-Jacobi-Bellman-Isaacs equation we obtain the monotone convergence of λVλ(.) and we characterize its limit W0 as maximal viscosity subsolution of a limit PDE. Using BSDE methods we prove that W0 satisfies a uniform dynamic programming principle involving the supremum and the infimum with respect to the time, and this is the key for an explicit representation formula for W0.

Application of bio-inspired methods for optimal open loop discrete time deterministic control problems

The article proposed the novel Sparrow colony optimization (SCO) algorithm to solve the optimal open loop control problem for nonlinear discrete deterministic dynamic systems described by difference equations. Numerical algorithm and software in Python programming language are implemented. Model examples demonstrating the effectiveness of the developed software are solved. Comparison of the considered algorithm results with a Dragonfly optimization algorithm at the solution of model problems is made. The recommended parameters values of the proposed algorithm are suggested.

Conversion of Certain Stochastic Control Problems into Deterministic Control Problems

A class of nonlinear, stochastic staticization control problems (including minimization problems with smooth, convex, coercive payoffs) driven by diffusion dynamics with constant diffusion coefficient is considered. A fundamental solution form is obtained where the same solution can be used for a limited variety of terminal costs without re-solution of the problem. One may convert this fundamental solution form from a stochastic control problem form to a deterministic control problem form. This yields an equivalence between certain second-order (in space) Hamilton-Jacobi partial differential equations (HJ PDEs) and associated first-order HJ PDEs. This reformulation has substantial numerical implications.

A multi-mode expansion method for boundary optimal control problems constrained by random Poisson equations

&lt;p style='text-indent:20px;'&gt;This paper develops efficient numerical algorithms for the optimal control problem constrained by Poisson equations with uncertain diffusion coefficients. We consider both unconstrained condition and box-constrained condition for the control. The algorithms are based upon a multi-mode expansion (MME) for the random state, the finite element approximation for the physical space and the alternating direction method of multipliers (ADMM) or two-block ADMM for the discrete optimization systems. The compelling aspect of our method is that, target random constrained control problem can be approximated to one deterministic constrained control problem under a state variable substitution equality. Thus, the computing resource, especially the memory consumption, can be reduced sharply. The convergence rates of the proposed algorithms are discussed in the paper. We also present some numerical examples to show the performance of our algorithms.&lt;/p&gt;

Large deviations for interacting particle systems: joint mean-field and small-noise limit

We consider a system of stochastic interacting particles in $\mathbb {R}^{d}$ and we describe large deviation asymptotics in a joint mean-field and small-noise limit. Precisely, a large deviation principle (LDP) is established for the empirical measure and the stochastic current, as the number of particles tends to infinity and the noise vanishes, simultaneously. We give a direct proof of the LDP using tilting and subsequently exploiting the link between entropy and large deviations. To this aim we employ consistency of suitable deterministic control problems associated to the stochastic dynamics.

Generalized Dual Dynamic Programming for Infinite Horizon Problems in Continuous State and Action Spaces

We describe a nonlinear generalization of dual dynamic programming (DP) theory and its application to value function estimation for deterministic control problems over continuous state and action spaces, in a discrete-time infinite horizon setting. We prove, using a Benders-type argument leveraging the monotonicity of the Bellman operator, that the result of a one-stage policy evaluation can be used to produce nonlinear lower bounds on the optimal value function that are valid over the entire state space. These bounds contain terms reflecting the functional form of the system's costs, dynamics, and constraints. We provide an iterative algorithm that produces successively better approximations of the optimal value function, and prove under certain assumptions that it achieves an arbitrarily low desired Bellman optimality tolerance at preselected points in the state space, in a finite number of iterations. We also describe means of certifying the quality of the approximate value function generated. We demonstrate the efficacy of the approach on systems whose dimensions are too large for conventional DP approaches to be practical.

Compositional Supervisory Control via Reactive Synthesis and Automated Planning

We show how reactive synthesis and automated planning can be leveraged effectively to find nonmaximal solutions to deterministic supervisory control problems of discrete event systems. To do so, we propose efficient translations of the supervisory control problem into the reactive synthesis and planning frameworks. Notably, our translation methods capture the compositional and reactive nature of control specifications, avoiding a potential exponential explosion found in alternative translation approaches. Additionally, we report on experimental results comparing the efficacy of different tools from the three disciplines, for a particular supervisory control benchmark.

Energy-Efficient Distributed Transmission Scheme for MTC in Dense Wireless Networks: A Mean-Field Approach

In this article, we address the problem of long-term resource allocation for a 5G and beyond dense wireless network. Of particular interest within the considered framework are machine-type communications (MTCs). To establish an optimal operation for an energy-efficient MTC system, the power allocation policy should be designed by taking into consideration both channel and queue states of the devices. This problem formulation belongs to the category of complex stochastic optimization that can be recast as a Markov decision process (MDP) over high-dimensional state space. We circumvent the hurdle inherent to state space of a huge dimension by resorting to mean-field approximation on the MDP. More specifically, we demonstrate that the formulated MDP converges to a deterministic control problem provided that the number of devices approaches infinity. We develop a low-complexity power allocation scheme to efficiently solve the original high-dimensional state space stochastic optimization problem. Our proposed power allocation policy is drawn from the solution of the Hamilton–Jacobi–Bellman equation. It can be understood as a threshold-based policy and can be implemented in a decentralized manner that makes it very appealing for high scale networks. Finally, the simulations analyses are provided to establish the effectiveness of our proposed framework.

Optimal Control of DERs in ADN Under Spatial and Temporal Correlated Uncertainties

The uncertainties of distribution energy resources (DERs) have significant impacts on active distribution networks (ADNs), and it is necessary to control the DER outputs to hedge the negative impacts of their uncertainties on ADNs. However, DER uncertainties are complicated, containing spatial and temporal correlation, which makes it challenging to design proper control schemes, especially when there exist temporal-correlated units such as energy units (EUs). This paper provides an Ito process model to describe the characteristics of stochastic resources and EUs in a unified way, which makes it easy to evaluate the impacts of stochastic resources on temporal-correlated units. Based the moment form of the Ito process model, a moment optimization (MO) approach is provided to transform the stochastic control (SC) problem into an optimization problem with respect to the first-order and second-order moments of the system variables. The scale of MO is comparable to that of the corresponding deterministic control problem, which means that the computational efficiency of MO is much higher than that of traditional approaches. Case studies also show that the proposed approach outperforms existing approaches in both the performance and computational efficiency, which means that the proposed approach has attractive potential for use in large-scale applications.