Control Of Linear Stochastic Systems Research Articles

Quadratic Tracking Control of Linear Stochastic Systems with Unknown Dynamics Using Average Off-Policy Q-Learning Method

This article investigates the optimal tracking control problem for data-based stochastic discrete-time linear systems. An average off-policy Q-learning algorithm is proposed to solve the optimal control problem with random disturbances. Compared with the existing off-policy reinforcement learning (RL) algorithm, the proposed average off-policy Q-learning algorithm avoids the assumption of an initial stability control. First, a pole placement strategy is used to design an initial stable control for systems with unknown dynamics. Second, the initial stable control is used to design a data-based average off-policy Q-learning algorithm. Then, this algorithm is used to solve the stochastic linear quadratic tracking (LQT) problem, and a convergence proof of the algorithm is provided. Finally, numerical examples show that this algorithm outperforms other algorithms in a simulation.

The Partial Controllability of Linear Stochastic Control Systems with Terminal Constraints and Its Applications to Game-Based Control Systems with Jumps

The Partial Controllability of Linear Stochastic Control Systems with Terminal Constraints and Its Applications to Game-Based Control Systems with Jumps

Region stability analysis and precise tracking control of linear stochastic systems

This paper provides a novel region stability criterion on the basis of the pole (eigenvalue) distribution of the linear stochastic system (LSS). Corresponding stabilization control and tracking control are also discussed, which are more suitable for practical engineering applications. First, in the light of the connection between the eigenvalues of the system and the dynamic properties of LSSs, the concept of region stability is given to describe the finer dynamic behavior of LSSs. The criterion for the region stability is also presented. Second, the region stabilization method is investigated according to region stability, which can ensure that the system satisfies a certain dynamic performance by setting the eigenvalues in a certain convex region. Third, a precise tracking control method of LSSs is addressed as an application of region stabilization. It can regulate the steady state properties and transient response of the tracking signal more precisely. Two examples are given to demonstrate the effectiveness of the new methods.

Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems

This article studies the adaptive optimal stationary control of continuous-time linear stochastic systems with both additive and multiplicative noises, using reinforcement learning techniques. Based on policy iteration, a novel off-policy reinforcement learning algorithm, named optimistic least-squares-based policy iteration, is proposed, which is able to find iteratively near-optimal policies of the adaptive optimal stationary control problem directly from input/state data without explicitly identifying any system matrices, starting from an initial admissible control policy. The solutions given by the proposed optimistic least-squares-based policy iteration are proved to converge to a small neighborhood of the optimal solution with probability one, under mild conditions. The application of the proposed algorithm to a triple inverted pendulum example validates its feasibility and effectiveness.

Periodic and event‐based impulse control for linear stochastic systems with multiplicative noise

SummaryThis paper studies the performance comparison of periodic and event‐based sampling for a class of linear stochastic systems with multiplicative noise, where the impulse control is adopted. By solving boundary value problems, we obtain the analytic expressions of the mean sampling time and the average state variance under the event‐based sampling. It is shown that the event‐based impulse control has substantially smaller average state variance than the periodic control under the same sampling frequency. Particularly, for the integrator case, the performance ratio of the two sampling methods is given explicitly. By simulation, it is demonstrated that the advantage of event‐based sampling over periodic sampling is most obvious for unstable systems, followed by critical stable systems, and least obvious for stable systems.

Controllability of Semilinear Stochastic Generalized Systems in Hilbert Spaces by GE-Evolution Operator Method

Controllability is a basic problem in the study of stochastic generalized systems. Compared with ordinary stochastic systems, the structure of stochastic singular systems is more complex, and it is necessary to study the controllability of stochastic generalized systems in the context of different solutions. In this paper, the controllability of semilinear stochastic generalized systems was investigated by using a GE-evolution operator for integral and impulsive solutions in Hilbert spaces. Some sufficient and necessary conditions were obtained. Firstly, the existence and uniqueness of the integral solution of semilinear stochastic generalized systems were discussed using the GE-evolution operator theory and Banach fixed point theorem. The existence and uniqueness theorem of the integral solution was obtained. Secondly, the approximate controllability of semilinear stochastic generalized systems was studied in the case of the integral solution. Thirdly, the existence and uniqueness of the impulsive solution of semilinear stochastic generalized systems were considered, and some sufficient conditions were provided. Fourthly, the approximate controllability of semilinear stochastic generalized systems was studied for the impulsive solution. At last, the exact controllability of linear stochastic systems was studied in the case of the impulsive solution, with some necessary and sufficient conditions given. The obtained results have important theoretical and practical value for the study of controllability of semilinear stochastic generalized systems.

Learning-Based Adaptive Control for Stochastic Linear Systems With Input Constraints

We propose a certainty-equivalence scheme for adaptive control of scalar linear systems subject to additive, i.i.d. Gaussian disturbances and bounded control input constraints, without requiring prior knowledge of the bounds of the system parameters, nor the control direction. Assuming that the system is at-worst marginally stable, mean square boundedness of the closed-loop system states is proven. Lastly, numerical examples are presented to illustrate our results.

Security tracking control for discrete-time stochastic systems subject to cyber attacks

This paper is concerned with the security tracking problem under the quadratic cost criterion for a class of discrete-time stochastic linear networked control systems (NCSs) exposed to cyber attacks, covering false data injection attacks as well as a class of DoS attacks. Taking into account factors such as network congestion and the defensive role of intrusion detection systems, successful attack events are modeled as a Bernoulli random sequence. To describe the transient trajectory of an NCS under the impact of a random attack, a probabilistic definition of secure trackability is taken. Therefore, an observer-based dynamic output feedback controller is designed in order to achieve the specified probabilistic secure trackability. Specifically, the probabilistic safety output tracking problem is transformed into an input-to-state stability problem in the probabilistic sense for closed-loop systems with some new sufficient conditions, provided that an augmented incremental model is utilized Then, the controller parameters and the upper bound of the quadratic cost function are determined by solving matrix inequalities, while easy-solution forms of the matrix inequalities to be solved are presented by the Schur complementary lemma. Both simulation studies and practical experiments demonstrate the effectiveness of the proposed control scheme.

Convex Optimization for Finite-Horizon Robust Covariance Control of Linear Stochastic Systems

This work addresses the finite-horizon robust covariance control problem for discrete time, partially observable, linear systems affected by random zero mean noise and deterministic uncertain-but-bounded disturbances restricted to lie in what is called ellitopic uncertainty set (e.g., finite intersection of centered at the origin ellipsoids/elliptic cylinders). Performance specifications are imposed on the random state-control trajectory via averaged convex quadratic inequalities, linear inequalities on the mean, chance constrained linear inequalities as well as convex-monotone constraints on the covariance matrix. For this problem we develop a computationally tractable procedure for designing affine control policies, in the sense that the parameters of the policy that guarantees the aforementioned performance specifications are obtained as solutions to an explicit convex program. Our theoretical findings are illustrated by a numerical example.

Where Freshness Matters in the Control Loop: Mixed Age-of-Information and Event-Based Co-Design for Multi-Loop Networked Control Systems

In the design of multi-loop Networked Control Systems (NCSs), wherein each control system is characterized by heterogeneous dynamics and associated with a certain set of timing specifications, appropriate metrics need to be employed for the synthesis of control and networking policies to efficiently respond to the requirements of each control loop. The majority of the design approaches for sampling, scheduling, and control policies include either time-based or event-based metrics to perform pertinent actions in response to the changes of the parameters of interest. We specifically focus in this article on Age-of-Information (AoI) as a recently-developed time-based metric and threshold-based triggering function as a generic Event-Triggered (ET) metric. We consider multiple heterogeneous stochastic linear control systems that close their feedback loops over a shared communication network. We investigate the co-design across the NCS and discuss the pros and cons with the AoI and ET approaches in terms of asymptotic control performance measured by Linear-Quadratic Gaussian (LQG) cost functions. In particular, sampling and scheduling policies combining AoI and stochastic ET metrics are proposed. It is argued that pure AoI functions that generate decision variables solely upon minimizing the average age irrespective of control systems dynamics may not be able to improve the overall NCS performance even compared with purely randomized policies. Our theoretical analysis is validated through several simulation scenarios.

Predictor-Based Output-Feedback Control of Linear Stochastic Systems With Large I/O Delays

In this article, we extend the stability results of cascaded closed-loop predictors for deterministic systems to stochastic linear systems with additive noise and arbitrarily large time-varying input delay, under suitable hypothesis on the delay function. We propose an output-feedback controller design based on a set of differential delay equations with size depending on the delay bound and we show that it ensures exponential stability in mean and mean square stability of the controlled system. The result is extended to multiple delays on separate input signals and to systems with arbitrarily large output delays.

An Impossibility Result Concerning Bounded Average-Moment Control of Linear Stochastic Systems

It is known that strictly unstable linear systems that are subject to nonvanishing additive stochastic noise with unbounded supports cannot be stabilized by using deterministi-cally bounded control inputs. In this paper, we explore similar impossibility results for scenarios where the expected value of the squared control input norm is subject to constraints and the support of the noise distribution is not necessarily unbounded. Specifically, we consider the stabilization problem with control policies that have bounded time-averaged second moments. We obtain values of such average second moment bounds, below which stabilization is not possible and the second moment of the state diverges regardless of the choice of the control policy and the initial state distribution. The results are illustrated with a numerical example.

Spectral decompositions for the solutions of Lyapunov equations for bilinear dynamical systems

In this paper, novel spectral decompositions are obtained for the solutions of generalized Lyapunov equations, which are observed in the study of controllability and observability of the state vector in deterministic bilinear systems. The same equations are used in the stability analysis and stabilization of stochastic linear control systems. To calculate these spectral decompositions, an iterative algorithm is proposed that uses the residues of the resolvent of the dynamics matrix. This algorithm converges for any initial guess, for a non-singular and stable dynamical system. The practical significance of the obtained results is that they allow one to characterize the contribution of individual eigen-components or their pairwise combinations to the asymptotic dynamics of the perturbation energy in deterministic bilinear and stochastic linear systems. In particular, the norm of the obtained eigen-components increases when frequencies of the corresponding oscillating modes approximate each other. Thus, the proposed decompositions provide a new fundamental approach for quantifying resonant modal interactions in a large and important class of weakly nonlinear systems.

Estimating the Impact of Cyber-Attack Strategies for Stochastic Networked Control Systems

Risk assessment is an inevitable step in implementation of a cyber-defense strategy. An important part of this assessment is to reason about the impact of possible attacks. In this paper, we propose a framework for estimating the impact of cyber-attacks in stochastic linear control systems. The framework can be used to estimate the impact of denial of service, rerouting, sign alternation, replay, false data injection, and bias injection attacks. For the stealthiness constraint, we adopt the Kullback-Leibler divergence between residual sequences during the attack. Two impact metrics are considered: (1) The probability that some of the critical states leave a safety region; and (2) The expected value of the infinity norm of the critical states. For the first metric, we prove that the impact estimation problem can be reduced to a set of convex optimization problems. Thus, the exact solution can be found efficiently. For the second metric, we derive an efficient to calculate lower bound. Finally, we demonstrate how the framework can be used for risk assessment on an example.

Dynamic event-triggered control for linear stochastic systems with sporadic measurements and communication delays

This paper deals with the stability of linear stochastic systems in the presence of sporadic measurements and communication delays. Motivated by considerations about the efficient use of the available resources, two so-called dynamic discrete-time event-triggered control (DTETC) strategies tailored to the system are designed, including some existing periodic event-triggered control strategies as particular cases. An impulsive switched system approach and a switched time-delay system approach are developed for modeling and stability analysis of the resulting closed-loop systems, respectively. In addition, it is shown that under the same decay rate, the designed dynamic DTETC may further reduce the conservatism than their static counterparts. Interestingly, the proposed impulsive switched system approach can capture the stabilizing effect of the communication delays. A numerical example is provided to illustrate the theoretical results.

Quantized stabilization of stochastic systems with multiplicative noise under Markovian switching

A problem of quantized state feedback quadratic mean-square stabilization of discrete-time stochastic processes under Markovian switching and multiplicative noise is considered. A static quantizer is used in the feedback channel and the jump Markovian switching is modeled by a discrete-time Markov chain. The control input is simultaneously applied to both the rate vector and the diffusion term. It is shown that the coarsest quantization density that permits quadratic mean-square stabilization of this system is achieved with the use of a logarithmic quantizer, and the coarsest quantization density is determined by an algebraic Riccati equation, which is also the solution to a special linear stochastic Markovian switching control system. Also, sufficient conditions for exponential mean-square stabilization of such systems are also explored. An example is given to demonstrate the obtained results.

Mixed deterministic and random optimal control of linear stochastic systems with quadratic costs

In this paper, we consider the mixed optimal control of a linear stochastic system with a quadratic cost functional, with two controllers—one can choose only deterministic time functions, called the deterministic controller, while the other can choose adapted random processes, called the random controller. The optimal control is shown to exist under suitable assumptions. The optimal control is characterized via a system of fully coupled forward-backward stochastic differential equations (FBSDEs) of mean-field type. We solve the FBSDEs via solutions of two (but decoupled) Riccati equations, and give the respective optimal feedback law for both deterministic and random controllers, using solutions of both Riccati equations. The optimal state satisfies a linear stochastic differential equation (SDE) of mean-field type. Both the singular and infinite time-horizonal cases are also addressed.

Partial Approximate Controllability for Linear Stochastic Control Systems

This paper is devoted to studying controllability problems of linear stochastic control systems with controls only in the drift terms. It is well known that such a system is not exactly controllabl...

Model Predictive Control of Stochastic Linear Systems with Probability Constraints

This paper presents a strategy for computing model predictive control of linear Gaussian noise systems with probability constraints. As usual, constraints are taken on the system state and control input. The novelty relies on setting bounds on the underlying cumulative probability distribution, and showing that the model predictive control can be computed in an efficient manner through these novel bounds— an application confirms this assertion. Indeed real-time experiments were carried out to control a direct current (DC) motor. The corresponding data show the effectiveness and usefulness of the approach.

Bisimulation Equivalence of Discrete-Time Stochastic Linear Control Systems

In this paper, we propose a definition of equivalence via stochastic bisimulation for the class of discrete-time stochastic linear control systems with possibly degenerate normally distributed disturbances. The notion is inspired by the notion of probabilistic bisimulation for probabilistic chains. Geometric necessary and sufficient conditions for checking this notion are derived. Model reduction via Kalman-like decomposition is also proposed. Connections with stochastic reachability are discussed and with finite horizon stochastic safety problems established. A discussion on the use of stochastic bisimulation equivalence for control design is given, and an application to optimal control problems with stochastic reachability specifications is finally presented.