Discrete-time Stochastic Control Systems Research Articles

On convergence of occupational measures sets of a discrete-time stochastic control system, with applications to averaging of hybrid systems

We establish that, under certain conditions, the set of random occupational measures generated by the state-control trajectories of a discrete-time stochastic system as well as the set of their mathematical expectations converge to a non-random, convex and compact set. We apply these results to the averaging a hybrid system with a slow continuous-time component and a fast discrete-time component. It is shown that the solutions of the hybrid system are approximated by the solutions of a differential inclusion. The novelty of our results is that we allow the state-control space of the fast component to be non-denumerable.

Linear Quadratic Optimal Control of Discrete-Time Stochastic Systems Driven by Homogeneous Markov Processes

Random terms in many natural and social science systems have distinct Markovian characteristics, such as Markov jump-taking values in a finite or countable set, and Wiener process-taking values in a continuous set. In general, these systems can be seen as Markov-process-driven systems, which can be used to describe more complex phenomena. In this paper, a discrete-time stochastic linear system driven by a homogeneous Markov process is studied, and the corresponding linear quadratic (LQ) optimal control problem for this system is solved. Firstly, the relations between the well-posedness of LQ problems and some linear matrix inequality (LMI) conditions are established. Then, based on the equivalence between the solvability of the generalized difference Riccati equation (GDRE) and the LMI condition, it is proven that the solvability of the GDRE is sufficient and necessary for the well-posedness of the LQ problem. Moreover, the solvability of the GDRE and the feasibility of the LMI condition are established, and it is proven that the LQ problem is attainable through a certain feedback control when any of the four conditions is satisfied, and the optimal feedback control of the LQ problem is given using the properties of homogeneous Markov processes and the smoothness of the conditional expectation. Finally, a practical example is used to illustrate the validity of the theory.

Compositional Construction of Safety Controllers for Networks of Continuous-Space POMDPs

In this article, we propose a compositional framework for the synthesis of safety controllers for networks of partially observable discrete-time stochastic control systems (also known as continuous-space partially observable Markov decision processes (POMDPs)). Given an estimator, we utilize a discretization-free approach to synthesize controllers ensuring safety specifications over finite-time horizons. The proposed framework is based on a notion of so-called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">local control barrier functions</i> computed for subsystems in two different ways. In the first scheme, no prior knowledge of estimation accuracy is needed. The second framework utilizes a probability bound on the estimation accuracy using a notion of so-called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">stochastic simulation functions</i> . In both proposed schemes, we derive sufficient small-gain-type conditions in order to compositionally construct control barrier functions for interconnected POMDPs using local barrier functions computed for subsystems. The constructed control barrier functions for the overall networks enable us to compute lower bounds on the probabilities that the interconnected POMDPs avoid certain unsafe regions in finite-time horizons. We demonstrate the effectiveness of our proposed approaches by applying them to an adaptive cruise control problem.

Constructing MDP Abstractions Using Data With Formal Guarantees

This letter is concerned with a data-driven technique for constructing finite Markov decision processes (MDPs) as finite abstractions of discrete-time stochastic control systems with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unknown</i> dynamics while providing formal closeness guarantees. The proposed scheme is based on notions of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">stochastic bisimulation functions</i> (SBF) to capture the probabilistic distance between state trajectories of an unknown stochastic system and those of finite MDP. In our proposed setting, we first reformulate corresponding conditions of SBF as a robust convex program (RCP). We then propose a scenario convex program (SCP) associated to the original RCP by collecting a finite number of data from trajectories of the system. We ultimately construct an SBF between the data-driven finite MDP and the unknown stochastic system with a given confidence level by establishing a probabilistic relation between optimal values of the SCP and the RCP. We also propose two different approaches for the construction of finite MDPs from data. We illustrate the efficacy of our results over a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">nonlinear</i> jet engine compressor with unknown dynamics. We construct a data-driven finite MDP as a suitable substitute of the original system to synthesize controllers maintaining the system in a safe set with some probability of satisfaction and a desirable confidence level.

Mixed optimal control for discrete-time stochastic systems with random coefficients

This paper is concerned with the problem of mixed optimal control for discrete-time systems with random coefficients where “mixed” means that there are two controllers: the deterministic controller and the random controller. Different from previous works, random coefficients and controllers with different information structures arise simultaneously in the optimal control problem, which lead to large difficulty. Using decoupling technique to solve the forward and backward stochastic difference equations, the necessary and sufficient conditions for the solvability of optimal control problem are given. An explicit expression of the optimal control strategies is derived based on a stochastic Riccati-type equation, which is new to the best of knowledge. Finally, a pension management problem is solved with the obtained theoretical results.

Sequential Source Coding for Stochastic Systems Subject to Finite Rate Constraints

In this article, we revisit the sequential source-coding framework to analyze fundamental performance limitations of discrete-time stochastic control systems subject to feedback data-rate constraints in finite-time horizon. The basis of our results is a new characterization of the lower bound on the minimum total-rate achieved by sequential codes subject to a total (across time) distortion constraint and a computational algorithm that allocates optimally the rate-distortion, for a given distortion level, at each instant of time and any fixed finite-time horizon. The idea behind this characterization facilitates the derivation of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">analytical</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">nonasymptotic</i> , and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">finite-dimensional</i> lower and upper bounds in two control-related scenarios: a) A parallel time-varying Gauss–Markov process with identically distributed spatial components that are quantized and transmitted through a noiseless channel to a minimum mean-squared error decoder; and b) a time-varying quantized linear quadratic Gaussian (LQG) closed-loop control system, with identically distributed spatial components and with a random data-rate allocation. Our nonasymptotic lower bound on the quantized LQG control problem reveals the absolute minimum data-rates for (mean square) stability of our time-varying plant for any fixed finite-time horizon. We supplement our framework with illustrative simulation experiments.

Security tracking control for discrete-time stochastic systems subject to cyber attacks

This paper is concerned with the security tracking problem under the quadratic cost criterion for a class of discrete-time stochastic linear networked control systems (NCSs) exposed to cyber attacks, covering false data injection attacks as well as a class of DoS attacks. Taking into account factors such as network congestion and the defensive role of intrusion detection systems, successful attack events are modeled as a Bernoulli random sequence. To describe the transient trajectory of an NCS under the impact of a random attack, a probabilistic definition of secure trackability is taken. Therefore, an observer-based dynamic output feedback controller is designed in order to achieve the specified probabilistic secure trackability. Specifically, the probabilistic safety output tracking problem is transformed into an input-to-state stability problem in the probabilistic sense for closed-loop systems with some new sufficient conditions, provided that an augmented incremental model is utilized Then, the controller parameters and the upper bound of the quadratic cost function are determined by solving matrix inequalities, while easy-solution forms of the matrix inequalities to be solved are presented by the Schur complementary lemma. Both simulation studies and practical experiments demonstrate the effectiveness of the proposed control scheme.

Fault reconstruction and resilient control for discrete-time stochastic systems

In this paper, a novel resilient control technique is proposed for discrete-time stochastic Brownian systems with simultaneous unknown inputs and unexpected faults. Prior to previous work, the stochastic Brownian system under consideration is quite general, where stochastic perturbations exist in states, control inputs, uncertainties, and faults. Moreover, the unknown input uncertainties concerned cannot be fully decoupled. Innovative observer by employing augmented system approach, decomposition observer, and optimization algorithms is proposed to achieve simultaneous estimates of both states and faults. Furthermore, fault reconstruction-based signal compensation is formulated to alleviate the effects from actuator faults and sensor faults. An observer-based controller is eventually constructed to enhance the stability and robustness of the closed-loop dynamic system. The integrated resilient control technique can ensure the system has reliable output even under faults. Both linear systems and Lipschitz nonlinear systems are investigated and the design procedures are addressed, respectively. Finally, the proposed resilient control techniques are validated via an electromechanical servo-system, and an aircraft system.

Control of Discrete-Time Stochastic Systems With Packet Loss by Event-Triggered Approach

Event-triggered schemes are characterized in less communication traffic while maintaining the resulting controlled plant's desired stability and performance criteria. In the presence of packet dropouts, this paper is concerned with the modeling and control problems for a class of discrete-time stochastic systems with event-triggered schemes. Two different mathematical analysis methods are proposed to model the packet loss when an event-triggered scheme is subject to packet loss. First, a stochastic distributed sequence satisfying the Bernoulli process is utilized to model the triggered packets transmitted in the communication networks. Considering the difference between time-triggered and event-triggered schemes, an equivalent model with a random sequence not satisfying a Bernoulli distribution process is also analyzed, which is not the same as some existing results in the literature. Then, the mean-square exponential stability of resulting augmented system is guaranteed and the prescribed H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> performance level is achieved by solving resulting discrete P-problem. Two examples with simulations are provided to validate the analytical results and demonstrate the effectiveness of the proposed co-design techniques.

Formal Synthesis of Stochastic Systems via Control Barrier Certificates

This paper focuses on synthesizing control policies for discrete-time stochastic control systems together with a lower bound on the probability that the systems satisfy the complex temporal properties. The desired properties of the system are expressed as linear temporal logic (LTL) specifications over finite traces. In particular, our approach decomposes the given specification into simpler reachability tasks based on its automata representation. We then propose the use of so-called \emph{control barrier certificate} to solve those simpler reachability tasks along with computing the corresponding controllers and probability bounds. Finally, we combine those controllers to obtain a hybrid control policy solving the considered problem. Under some assumptions, we also provide two systematic approaches for uncountable and finite input sets to search for control barrier certificates. We demonstrate the effectiveness of the proposed approach on a room temperature control and lane-keeping of a vehicle modeled as a four-dimensional single-track kinematic model. We compare our results with the discretization-based methods in the literature.

Compositional abstraction of large-scale stochastic systems: A relaxed dissipativity approach

In this paper, we propose a compositional approach for the construction of finite abstractions (a.k.a. finite Markov decision processes (MDPs)) for networks of discrete-time stochastic control subsystems that are not necessarily stabilizable. The proposed approach leverages the interconnection topology and a notion of finite-step stochastic storage functions, that describes joint dissipativity-type properties of subsystems and their abstractions, and establishes a finite-step stochastic simulation function as a relation between the network and its abstraction. To this end, we first develop a new type of compositionality conditions which is less conservative than the existing ones. In particular, using a relaxation via a finite-step stochastic simulation function, it is possible to construct finite abstractions such that stabilizability of each subsystem is not necessarily required. We then propose an approach to construct finite MDPs together with their corresponding finite-step storage functions for general discrete-time stochastic control systems satisfying an incremental passivability property. We also construct finite MDPs for a particular class of nonlinear stochastic control systems. To demonstrate the effectiveness of the proposed results, we first apply our approach to an interconnected system composed of 4 subsystems such that 2 of them are not stabilizable. We then consider a road traffic network in a circular cascade ring composed of 50 cells, and construct compositionally a finite MDP of the network. We employ the constructed finite abstractions as substitutes to compositionally synthesize policies keeping the density of the traffic lower than 20 vehicles per cell. Finally, we apply our proposed technique to a fully interconnected network of 500 nonlinear subsystems and construct their finite MDPs with guaranteed error bounds on the probabilistic distance between their output trajectories.

Compositional construction of infinite abstractions for networks of stochastic control systems

This paper is concerned with a compositional approach for constructing infinite abstractions of interconnected discrete-time stochastic control systems. The proposed approach uses the interconnection matrix and joint dissipativity-type properties of subsystems and their abstractions described by new notions of so-called stochastic storage functions. The interconnected abstraction framework is based on new notions of so-called stochastic simulation functions, constructed compositionally using stochastic storage functions of components. Using stochastic simulation functions, one can quantify the distance between original interconnected stochastic systems and interconnected abstractions in the probabilistic setting. Accordingly, one can leverage the proposed results to perform analysis and synthesis over abstract interconnected systems, and then carry the results back over concrete ones. In the first part of the paper, we derive dissipativity-type compositional reasoning for the quantification of the distance in probability between the interconnection of stochastic control subsystems and that of their abstractions. Moreover, we focus on a class of discrete-time nonlinear stochastic control systems with independent noises in the abstract and concrete subsystems, and propose a computational scheme to construct abstractions together with their corresponding stochastic storage functions. In the second part of the paper, we consider specifications expressed as syntactically co-safe linear temporal logic formulae and show how a synthesized policy for the abstract system can be refined to a policy for the original system while providing a guarantee on the probability of satisfaction. We demonstrate the effectiveness of the proposed results by constructing an abstraction (totally 3 dimensions) of the interconnection of three discrete-time nonlinear stochastic control subsystems (together 222 dimensions) in a compositional fashion such that the compositionality condition does not require any constraint on the number or gains of the subsystems. We also employ the constructed abstraction as a substitute to synthesize a controller enforcing a syntactically co-safe linear temporal logic specification.

The effect of yeast supplementation on equine fecal microbial population dynamics following abrupt dietary changes

In this paper, we propose a compositional approach for the construction of finite abstractions (a.k.a. finite Markov decision processes (MDPs)) for networks of discrete-time stochastic control subsystems that are not necessarily stabilizable. The proposed approach leverages the interconnection topology and a notion of finite-step stochastic storage functions, that describes joint dissipativity-type properties of subsystems and their abstractions, and establishes a finite-step stochastic simulation function as a relation between the network and its abstraction. To this end, we first develop a new type of compositionality conditions which is less conservative than the existing ones. In particular, using a relaxation via a finite-step stochastic simulation function, it is possible to construct finite abstractions such that stabilizability of each subsystem is not necessarily required. We then propose an approach to construct finite MDPs together with their corresponding finite-step storage functions for general discrete-time stochastic control systems satisfying an incremental passivability property. We also construct finite MDPs for a particular class of nonlinear stochastic control systems. To demonstrate the effectiveness of the proposed results, we first apply our approach to an interconnected system composed of 4 subsystems such that 2 of them are not stabilizable. We then consider a road traffic network in a circular cascade ring composed of 50 cells, and construct compositionally a finite MDP of the network. We employ the constructed finite abstractions as substitutes to compositionally synthesize policies keeping the density of the traffic lower than 20 vehicles per cell. Finally, we apply our proposed technique to a fully interconnected network of 500 nonlinear subsystems and construct their finite MDPs with guaranteed error bounds on the probabilistic distance between their output trajectories.

Bisimulation Equivalence of Discrete-Time Stochastic Linear Control Systems

In this paper, we propose a definition of equivalence via stochastic bisimulation for the class of discrete-time stochastic linear control systems with possibly degenerate normally distributed disturbances. The notion is inspired by the notion of probabilistic bisimulation for probabilistic chains. Geometric necessary and sufficient conditions for checking this notion are derived. Model reduction via Kalman-like decomposition is also proposed. Connections with stochastic reachability are discussed and with finite horizon stochastic safety problems established. A discussion on the use of stochastic bisimulation equivalence for control design is given, and an application to optimal control problems with stochastic reachability specifications is finally presented.

Compositional Synthesis of Finite Abstractions for Continuous-Space Stochastic Control Systems: A Small-Gain Approach

This paper is concerned with a compositional approach for constructing finite abstractions (a.k.a. finite Markov decision processes) of interconnected discrete-time stochastic control systems. The proposed framework is based on a notion of so-called stochastic simulation function enabling us to use an abstract system as a substitution of the original one in the controller design process with guaranteed error bounds. In the first part of the paper, we derive sufficient small-gain type conditions for the compositional quantification of the distance in probability between the interconnection of stochastic control subsystems and that of their (finite or infinite) abstractions. In the second part of the paper, we construct finite abstractions together with their corresponding stochastic simulation functions for the class of linear stochastic control systems. We apply our results to the temperature regulation in a circular building by constructing compositionally a finite abstraction of a network containing 1000 rooms. We use the constructed finite abstractions as substitutes to synthesize policies compositionally regulating the temperature in each room for a bounded time horizon.

Elegant anti-disturbance control for discrete-time stochastic systems with nonlinearity and multiple disturbances

ABSTRACTAn elegant anti-disturbance control (EADC) strategy for a class of discrete-time stochastic systems with both nonlinearity and multiple disturbances, which include the disturbance with partially known information and a sequence of random vectors, is proposed in this paper. A stochastic disturbance observer is constructed to estimate the disturbance with partially known information, based on which, an EADC scheme is proposed by combining pole placement and linear matrix inequality methods. It is proved that the two different disturbances can be rejected and attenuated, and the corresponding desired performances can be guaranteed for discrete-time stochastic systems with known and unknown nonlinear dynamics, respectively. Simulation examples are given to demonstrate the effectiveness of the proposed schemes compared with some existing results.

Iterative learning control for discrete-time stochastic systems with quantized information

An iterative learning control (ILC) algorithm using quantized error information is given in this paper for both linear and nonlinear discrete-time systems with stochastic noises. A logarithmic quantizer is used to guarantee an adaptive improvement in tracking performance. A decreasing learning gain is introduced into the algorithm to suppress the effects of stochastic noises and quantization errors. The input sequence is proved to converge strictly to the optimal input under the given index. Illustrative simulations are given to verify the theoretical analysis.

Control design for discrete-time state-multiplicative noise stochastic systems

Design conditions for existence of the H∞ linear state feedback control for discretetime stochastic systems with state-multiplicative noise and polytopic uncertainties are presented in the paper. Using an enhanced form of the bounded real lemma for discrete-time stochastic systems with state-multiplicative noise, the LMI-based procedure is provided for computation of the gains of linear, as well as nonlinear, state control law. The approach is illustrated on an example demonstrating the validity of the proposed method.

A Maximum Principle for Optimal Control of Discrete-Time Stochastic Systems With Multiplicative Noise

The maximum principle (MP) for the discrete-time stochastic optimal control problems is proved. It is shown that the adjoint equations of the MP are a pair of backward stochastic difference equations.

Asymptotic stability in probability and stabilization for a class of discrete-time stochastic systems

Summary This paper investigates asymptotic stability in probability and stabilization designs of discrete-time stochastic systems with state-dependent noise perturbations. Our work begins with a lemma on a special discrete-time stochastic system for which almost all of its sample paths starting from a nonzero initial value will never reach the origin subsequently. This motivates us to deal with the asymptotic stability in probability of discrete-time stochastic systems. A stochastic Lyapunov theorem on asymptotic stability in probability is proved by means of the convergence theorem of supermartingale. An example is given to show the difference between asymptotic stability in probability and almost surely asymptotic stability. Based on the stochastic Lyapunov theorem, the problem of asymptotic stabilization for discrete-time stochastic control systems is considered. Some sufficient conditions are proposed and applied for constructing asymptotically stable feedback controllers. Copyright © 2014 John Wiley & Sons, Ltd.