Discrete-time Stochastic Control Research Articles

On convergence of occupational measures sets of a discrete-time stochastic control system, with applications to averaging of hybrid systems

We establish that, under certain conditions, the set of random occupational measures generated by the state-control trajectories of a discrete-time stochastic system as well as the set of their mathematical expectations converge to a non-random, convex and compact set. We apply these results to the averaging a hybrid system with a slow continuous-time component and a fast discrete-time component. It is shown that the solutions of the hybrid system are approximated by the solutions of a differential inclusion. The novelty of our results is that we allow the state-control space of the fast component to be non-denumerable.

Almost sure exponential stabilization of impulsive Markov switching systems via discrete-time stochastic feedback control

It is well known that noise can stabilize an unstable Markov switching system (MSS). However, given an unstable impulsive Markov switching system (IMSS), can it be stabilized by noise? To the best of the author’s knowledge, this question has been rarely explored. If the stochastic feedback controller is observed in discrete time, the research is more difficult. In this paper, a comparative method is used to establish a noise stabilization strategy based on discrete-time state and mode observations for IMSSs. In addition, we use the stationary distribution of the ergodic Markov chain instead of the M-matrix technique to obtain the pth moment estimation on the gap between the discrete-time observed controlled system’s states and the auxiliary system’s states, so that the control strategy can be applied to a wider class of IMSSs. Finally, we applied the proposed control scheme to impulsive neural networks with Markovian switching (INNs-MS) and impulsive boost converter systems (IBCSs). Through an example and several simulations, we validated the effectiveness and feasibility of this control scheme.

First- and Second-Order Maximum Principles for Discrete-Time Stochastic Optimal Control With Recursive Utilities

This paper deals with the discrete-time stochastic optimal control problems with recursive utilities under weakened convexity assumption. A new stochastic maximum principle is established. Moreover, by constructing two new adjoint equations and two new variational equations for the backward stochastic difference equation (BS <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\Delta$</tex-math></inline-formula> E), we obtain the second-order necessary optimality condition of quasi-singular control. Finally, as an illustration, a discrete-time mean-variance portfolio selection mixed with a recursive utility functional optimization problem is solved.

A maximum principle for discrete-time stochastic optimal control problem with delay

A maximum principle for discrete-time stochastic optimal control problem with delay

On Strategic Measures and Optimality Properties in Discrete-Time Stochastic Control with Universally Measurable Policies

This paper concerns discrete-time infinite-horizon stochastic control systems with Borel state and action spaces and universally measurable policies. We study optimization problems on strategic measures induced by the policies in these systems. The results are then applied to risk-neutral and risk-sensitive Markov decision processes to establish the measurability of the optimal value functions and the existence of universally measurable, randomized or nonrandomized, ϵ-optimal policies, for a variety of average cost criteria and risk criteria. We also extend our analysis to a class of minimax control problems and establish similar optimality results under the axiom of analytic determinacy. Funding: This work was supported by grants from DeepMind, the Alberta Machine Intelligence Institute (AMII), and Alberta Innovates-Technology Futures (AITF).

The general maximum principle for discrete-time stochastic control problems

In this paper, we discuss the problem of discrete-time stochastic control with multiplicative noise, where the control domains may not be convex. We propose a novel approach inspired by the classical spike variation in continuous-time cases, but adapted to the discrete-time setting by perturbing in a random scale instead of time scale. Our approach allows us to obtain the maximum principle recursively without the need for variation equations or adjoint equations, which are typically required in previous methods. Moreover, we provide two maximum principles, one for the general case and the other when feedback forms exist. Our approach covers the previous results when the control domains are convex. Finally, we demonstrate the effectiveness of our new results through an example.

Modified dynamic programming algorithms for GLOSA systems with stochastic signal switching times

A discrete-time stochastic optimal control problem was recently proposed to address the GLOSA (Green Light Optimal Speed Advisory) problem in cases where the next signal switching time is decided in real time and is therefore uncertain in advance. The corresponding numerical solution via SDP (Stochastic Dynamic Programming) calls for substantial computation time, which excludes problem solution in the vehicle’s on-board computer in real time. In this context, the present paper concentrates on the challenge of developing numerical algorithms to solve efficiently the stochastic GLOSA problem. As a first attempt to overcome the computation time bottleneck, a modified version of Dynamic Programming, known as Discrete Differential Dynamic Programming (DDDP) was recently employed for the numerical solution of the stochastic optimal control problem and was demonstrated to achieve results equivalent to those obtained with the ordinary SDP algorithm, albeit with significantly reduced computation times. After an outline of the DDDP approach, the present work considers a second modified version of Dynamic Programming, known as Differential Dynamic Programming (DDP). For the stochastic GLOSA problem, it is demonstrated that DDP achieves quasi-instantaneous (extremely fast) solutions in terms of CPU times, which allows for the proposed approach to be readily executable online, in an MPC (Model Predictive Control) framework, in the vehicle’s on-board computer. The novel numerical approach is tested and demonstrated by use of realistic examples and is compared to the SDP and DDDP solutions. It should be noted that DDP does not require discretization of variables, hence the obtained solutions may be slightly superior to the standard SDP solutions.

Compositional Construction of Safety Controllers for Networks of Continuous-Space POMDPs

In this article, we propose a compositional framework for the synthesis of safety controllers for networks of partially observable discrete-time stochastic control systems (also known as continuous-space partially observable Markov decision processes (POMDPs)). Given an estimator, we utilize a discretization-free approach to synthesize controllers ensuring safety specifications over finite-time horizons. The proposed framework is based on a notion of so-called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">local control barrier functions</i> computed for subsystems in two different ways. In the first scheme, no prior knowledge of estimation accuracy is needed. The second framework utilizes a probability bound on the estimation accuracy using a notion of so-called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">stochastic simulation functions</i> . In both proposed schemes, we derive sufficient small-gain-type conditions in order to compositionally construct control barrier functions for interconnected POMDPs using local barrier functions computed for subsystems. The constructed control barrier functions for the overall networks enable us to compute lower bounds on the probabilities that the interconnected POMDPs avoid certain unsafe regions in finite-time horizons. We demonstrate the effectiveness of our proposed approaches by applying them to an adaptive cruise control problem.

A distributed stochastic approximation algorithm for stochastic LQ control with unknown uncertainty

This paper studies a discrete-time stochastic control problem with linear quadratic criteria over an infinite-time horizon. We focus on control systems whose system matrices are associated with random parameters involving unknown statistical properties. We design a distributed stochastic approximation algorithm to tackle the Riccati equation and derive the optimal controller stabilizing the system. The convergence analysis is provided.

Dual Effect, Certainty Equivalence, and Separation Revisited: A Counterexample and a Relaxed Characterization for Optimality

In this article, we study the optimality of control policies admitting certainty equivalence or separation (of estimation and control) in discrete-time stochastic control, with the following two main contributions. We first revisit the influential theorem given in the seminal 1974 paper by Bar-Shalom and Tse, which studies the equivalence between certainty equivalence (CE) and no-dual-effect (NDE) properties in discrete-time stochastic control problems involving a linear dynamic system with a possibly nonlinear measurement function. We show that there is a subtle error in Bar-Shalom and Tse’s proof of the claim that CE implies NDE. Moreover, we prove that the claim does not hold by providing a counterexample. As our second and primary contribution, we introduce an alternative and a more relaxed notion of dual freeness and establish that this new notion is sufficient to guarantee the separation of estimation and control and CE in the same control problem considered by Bar-Shalom and Tse.

State estimation and optimal control of an inverted pendulum on a cart system with stochastic approximation approach

&lt;abstract&gt; &lt;p&gt;In this paper, optimal control of an inverted pendulum on a cart system is studied. Since the nonlinear structure of the system is complex, and in the presence of random disturbances, optimization and control of the motion of the system become more challenging. For handling this system, a discrete-time stochastic optimal control problem for the system is described, where the external force is considered as the control input. By defining a loss function, namely, the mean squared errors to be minimized, the stochastic approximation (SA) approach is applied to estimate the state dynamics. In addition, the Hamiltonian function is defined, and the first-order necessary conditions are derived. The gradient of the cost function is determined so that the SA approach is employed to update the control sequences. For illustration, considering the values of the related parameters in the system, the discrete-time stochastic optimal control problem is solved iteratively by using the SA algorithm. The simulation results show that the state estimation and the optimal control law design are well performed with the SA algorithm, and the motion of the inverted pendulum cart is addressed satisfactorily. In conclusion, the efficiency of the SA approach for solving the inverted pendulum on a cart system is verified.&lt;/p&gt; &lt;/abstract&gt;

Constructing MDP Abstractions Using Data With Formal Guarantees

This letter is concerned with a data-driven technique for constructing finite Markov decision processes (MDPs) as finite abstractions of discrete-time stochastic control systems with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unknown</i> dynamics while providing formal closeness guarantees. The proposed scheme is based on notions of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">stochastic bisimulation functions</i> (SBF) to capture the probabilistic distance between state trajectories of an unknown stochastic system and those of finite MDP. In our proposed setting, we first reformulate corresponding conditions of SBF as a robust convex program (RCP). We then propose a scenario convex program (SCP) associated to the original RCP by collecting a finite number of data from trajectories of the system. We ultimately construct an SBF between the data-driven finite MDP and the unknown stochastic system with a given confidence level by establishing a probabilistic relation between optimal values of the SCP and the RCP. We also propose two different approaches for the construction of finite MDPs from data. We illustrate the efficacy of our results over a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">nonlinear</i> jet engine compressor with unknown dynamics. We construct a data-driven finite MDP as a suitable substitute of the original system to synthesize controllers maintaining the system in a safe set with some probability of satisfaction and a desirable confidence level.

Sequential Source Coding for Stochastic Systems Subject to Finite Rate Constraints

In this article, we revisit the sequential source-coding framework to analyze fundamental performance limitations of discrete-time stochastic control systems subject to feedback data-rate constraints in finite-time horizon. The basis of our results is a new characterization of the lower bound on the minimum total-rate achieved by sequential codes subject to a total (across time) distortion constraint and a computational algorithm that allocates optimally the rate-distortion, for a given distortion level, at each instant of time and any fixed finite-time horizon. The idea behind this characterization facilitates the derivation of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">analytical</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">nonasymptotic</i> , and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">finite-dimensional</i> lower and upper bounds in two control-related scenarios: a) A parallel time-varying Gauss–Markov process with identically distributed spatial components that are quantized and transmitted through a noiseless channel to a minimum mean-squared error decoder; and b) a time-varying quantized linear quadratic Gaussian (LQG) closed-loop control system, with identically distributed spatial components and with a random data-rate allocation. Our nonasymptotic lower bound on the quantized LQG control problem reveals the absolute minimum data-rates for (mean square) stability of our time-varying plant for any fixed finite-time horizon. We supplement our framework with illustrative simulation experiments.

Maximum principle for discrete-time stochastic control problem of mean-field type

In this work, a discrete-time mean-field type stochastic optimal control problem is studied. The goal is to derive the stochastic maximum principle with convex control domains. L-derivative is applied to handle the mean-field term and a technique of adjoint operator is used to overcome the difficulties of obtaining adjoint equations and duality relation. Then, the stochastic maximum principle for discrete-time mean-field type stochastic optimal control problem is established. Finally, as an illustration of our stochastic maximum principle, a discrete-time mean–variance portfolio selection problem is solved with the decoupling technique which is different from the continuous-time case.

Optimal Trading Algorithms under Regime Switching

In this article, the author is concerned with the problem of efficiently trading a large position in the marketplace when the stock price dynamic follows a regime-switching process. If the execution of a large order is not done properly, this will certainly lead to large losses. Given that the execution of a large position may take several trading days, it is therefore reasonable to assume that the market microstructure may change during the execution of the order. To address this possibility, the author assumes that the stock price follows a regime-switching model. This article is particularly interested in trading algorithms that track market benchmarks such as the volume-weighted average price (VWAP) and the minimum execution shortfall. The author proposes trading algorithms that break the execution order into small pieces and execute them over a predetermined period of time so as to minimize the overall execution shortfall or exceed the overall market VWAP. The underlying problem is formulated as a discrete-time stochastic optimal control problem with resource constraints. The value function and optimal trading strategies are derived in closed form. Numerical simulations with market data are reported to illustrate the pertinence of the approach.

Security tracking control for discrete-time stochastic systems subject to cyber attacks

This paper is concerned with the security tracking problem under the quadratic cost criterion for a class of discrete-time stochastic linear networked control systems (NCSs) exposed to cyber attacks, covering false data injection attacks as well as a class of DoS attacks. Taking into account factors such as network congestion and the defensive role of intrusion detection systems, successful attack events are modeled as a Bernoulli random sequence. To describe the transient trajectory of an NCS under the impact of a random attack, a probabilistic definition of secure trackability is taken. Therefore, an observer-based dynamic output feedback controller is designed in order to achieve the specified probabilistic secure trackability. Specifically, the probabilistic safety output tracking problem is transformed into an input-to-state stability problem in the probabilistic sense for closed-loop systems with some new sufficient conditions, provided that an augmented incremental model is utilized Then, the controller parameters and the upper bound of the quadratic cost function are determined by solving matrix inequalities, while easy-solution forms of the matrix inequalities to be solved are presented by the Schur complementary lemma. Both simulation studies and practical experiments demonstrate the effectiveness of the proposed control scheme.

Maximum principle for discrete-time stochastic optimal control problem and stochastic game

&lt;p style='text-indent:20px;'&gt;This paper is first concerned with one kind of discrete-time stochastic optimal control problem with convex control domains, for which necessary condition in the form of Pontryagin's maximum principle and sufficient condition of optimality are derived. The results are then extended to two kinds of discrete-time stochastic games. Two illustrative examples are studied, for which the explicit optimal strategies are given. This paper establishes a rigorous version of discrete-time stochastic maximum principle in a clear and concise way and paves a road for further related topics.&lt;/p&gt;

An efficient algorithm for stochastic optimal control problems by means of a least-squares Monte-Carlo method

In this work, we provide discrete optimality conditions of the optimal control problems of stochastic differential equations. Euler and Runge–Kutta methods are used for discretization. A Lagrange multiplier method for a discrete-time stochastic optimal control problem is formulated. The discrete adjoint process is obtained in terms of conditional expectations and for both methods. To estimate these nested conditional expectations at each time step via simulation, we use a very powerful new approach, least-squares Monte-Carlo method, developed by Longstaff–Schwartz. This is the first time to solve a stochastic optimal control problem by calculating the nested conditional expectations numerically with the help of a least-squares Monte-Carlo method. Some examples are studied to test and demonstrate the efficiency of the Lagrange multiplier combined with the least-squares Monte-Carlo method.

Decoupled Disturbance Compensator Based Discrete-time Stochastic Sliding Mode Control with Experimental Results

Decoupled Disturbance Compensator Based Discrete-time Stochastic Sliding Mode Control with Experimental Results

Reinforcement Learning Approaches to Optimal Market Making

Market making is the process whereby a market participant, called a market maker, simultaneously and repeatedly posts limit orders on both sides of the limit order book of a security in order to both provide liquidity and generate profit. Optimal market making entails dynamic adjustment of bid and ask prices in response to the market maker’s current inventory level and market conditions with the goal of maximizing a risk-adjusted return measure. This problem is naturally framed as a Markov decision process, a discrete-time stochastic (inventory) control process. Reinforcement learning, a class of techniques based on learning from observations and used for solving Markov decision processes, lends itself particularly well to it. Recent years have seen a very strong uptick in the popularity of such techniques in the field, fueled in part by a series of successes of deep reinforcement learning in other domains. The primary goal of this paper is to provide a comprehensive and up-to-date overview of the current state-of-the-art applications of (deep) reinforcement learning focused on optimal market making. The analysis indicated that reinforcement learning techniques provide superior performance in terms of the risk-adjusted return over more standard market making strategies, typically derived from analytical models.