Discrete-time Stochastic Systems Research Articles

Moment propagation of polynomial systems through Carleman linearization for probabilistic safety analysis

We develop a method to approximate the moments of a discrete-time stochastic polynomial system. Our method is built upon Carleman linearization with truncation. Specifically, we take a stochastic polynomial system with finitely many states and transform it into an infinite-dimensional system with linear deterministic dynamics, which describe the exact evolution of the moments of the original polynomial system. We then truncate this deterministic system to obtain a finite-dimensional linear system, and use it for moment approximation by iteratively propagating the moments along the finite-dimensional linear dynamics across time. We provide efficient online computation methods for this propagation scheme with several error bounds for the approximation. Our results also show that precise values of certain moments at a given time step can be obtained when the truncated system is sufficiently large. Furthermore, we investigate techniques to reduce the offline computation load using reduced Kronecker power. Based on the obtained approximate moments and their errors, we also provide hyperellipsoidal regions that are safe for some given probability bound. Those bounds allow us to conduct probabilistic safety analysis online through convex optimization. We demonstrate our results on a logistic map with stochastic dynamics and a vehicle dynamics subject to stochastic disturbance.

Observability and Detectability of Stochastic Labeled Graphs

In this paper, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">observable stochastic graphs and detetectable stochastic graphs</i> are, respectively, defined with the detailed implementation for the observability and detectability of stochastic discrete-time and discrete-state dynamic systems. More specifically, they are generally two classes of vertex-colored and edge-labeled graphs rendering a walking agent therein to determine his initial and current positions, respectively, in probability one by measuring the color sequence of his traversed vertices. In the part of analysis, we follow the aforementioned two definitions and establish the corresponding polynomial-time verifying algorithms. Notably, the implicit formulas are also established to calculate the observability and detectability probability for any pairwise vertex pair. In the synthesis part, the minimal number of colors dyeing the vertices to make any considered graph stochastically observable and detectable is investigated, respectively. Our results indicate that the minimal coloring problem is NP-hard for observability in the stochastic situations but is solvable in polynomial time for detectability in the deterministic cases. The observability and detectability of stochastic finite-valued systems, assembling with the finite-cardinal state spaces, are validated as a compelling application of these two types of directed graphs, whereas the minimal sensors placement problems subject to observability and detectability problems are accordingly interpreted by the minimum set cover algorithm.

Parameter identification of the linear discrete-time stochastic systems with unknown exogenous inputs

The paper addresses a parameter identification problem for linear discrete-time stochastic systems with unknown exogenous inputs. Such systems are considered when solving practical problems related to the measurements processing in the case when it is impossible to do any assumptions about the evolution of unknown input signal or its statistical characteristics that can change over time. We consider a class of discrete time linear stochastic systems with unknown exogenous inputs, where an additional source of a priori uncertainty of the system model is introduced, namely, the unknown parameter, on the elements of which the system model matrices can depend. This formulation of the parameter identification problem under the conditions of unknown inputs and the presence of random noises describes a high degree of uncertainty of a discrete time linear stochastic system. We propose a novel solution to this problem based on the construction of a new instrumental identification criterion. Minimization of this criterion allows for evaluating the unknown system model parameters simultaneously with the estimating of the state vector and unknown exogenous inputs of the system. Numerical experiments confirm the validity and efficiency of the proposed parameter identification method.

Continuous-Time Subspace Identification with Prior Information Using Generalized Orthonormal Basis Functions

This paper presents a continuous-time subspace identification method utilizing prior information and generalized orthonormal basis functions. A generalized orthonormal basis is constructed by a rational inner function, and the transformed noises have ergodic properties. The lifting approach and the Hambo system transform are used to establish the equivalent nature of continuous and transformed discrete-time stochastic systems. The constrained least squares method is adopted to investigate the incorporation of prior knowledge in order to further increase the subspace identification algorithm’s accuracy. The input–output algebraic equation derives an optimal multistep forward predictor, and prior knowledge is expressed as equality constraints. In order to solve an optimization problem with equality constraints characterizing the prior knowledge, the proposed method reduces the computational burden. The effectiveness of the proposed method is provided by numerical simulations.

Data-driven verification and synthesis of stochastic systems via barrier certificates

In this work, we study verification and synthesis problems for safety specifications over unknown discrete-time stochastic systems. When a model of the system is available, barrier certificates have been successfully applied for ensuring the satisfaction of safety specifications. In this work, we formulate the computation of barrier certificates as a robust convex program (RCP). Solving the acquired RCP is hard in general because the model of the system that appears in one of the constraints of the RCP is unknown. We propose a data-driven approach that replaces the uncountable number of constraints in the RCP with a finite number of constraints by taking finitely many random samples from the trajectories of the system. We thus replace the original RCP with a scenario convex program (SCP) and show how to relate their optimizers. We guarantee that the solution of the SCP is a solution of the RCP with a priori guaranteed confidence when the number of samples is larger than a specific value. This provides a lower bound on the safety probability of the original unknown system together with a controller in the case of synthesis. We also discuss an extension of our verification approach to a case where the associated robust program is non-convex and show how a similar methodology can be applied. Finally, the applicability of our proposed approach is illustrated through three case studies.

Application of Migrating Optimization Algorithms in Problems of Optimal Control of Discrete-Time Stochastic Dynamical Systems

The problem of finding the optimal open-loop control for discrete-time stochastic dynamical systems is considered. It is assumed that the initial conditions and external influences are random. The average value of the Bolza functional defined on individual trajectories is minimized. It is proposed to solve the problem by means of classical and modified migrating optimization algorithms. The modification of the migrating algorithm consists of cloning the members of the initial population and choosing different strategies of migratory behavior for the main population and for populations formed by clones. At the final stage of the search for an extremum, an intensively clarifying migration cycle is implemented with the participation of three leaders of the populations participating in the search process. Problems of optimal control of bundles of trajectories of deterministic discrete dynamical systems, as well as individual trajectories, are considered as special cases. Seven model examples illustrating the performance of the proposed approach are solved.

Алгоритмы дискретной фильтрации на основе модифицированной взвешенной ортогонализации Грама –Шмидта для дискретных стохастических систем с мультипликативными и аддитивными шумами

Discrete-time stochastic systems with multiplicative and additive noises describe a wide class of mathematical models of complex systems, for example, industrial-technological, energy, economic, telecommunication, aerospace systems, etc. An important class of algorithms for processing measure ment information in complex systems are Kalman-type discrete-time filtering algorithms. Purpose. Construction of new discrete-time filtering algorithms for discrete-time linear systems with multiplicative and additive noises based on numerically stable modified weighted Gram – Schmidt orthogonalization (MWGS). Methodology. The methods of computational linear algebra were used, namely, the direct procedu re of MWGS-orthogonalization, the theory of Kalman filtering, methods of scientific programming in MATLAB. Findings. New LD-algorithms for discrete-time filtering in covariance and informational form for discrete-time stochastic systems with multiplicative and additive noises are constructed. The algorithms have an extended array form allowing updates for all necessary filter values. The method employs numerically stable modified weighted Gram – Schmidt orthogonalization. Algebraic equiva lence of LD-filters to Kalman-type covariance and informational algorithms for linear discrete time stochastic systems with multiplicative and additive noises is proved. The conducted numerical experiments have shown the effectiveness of the proposed algorithms using the example of solving the problem of parametric estimation of a model of almost rectilinear motion, as well as their savings in computation time compared to previously constructed UD filters. Value. New discrete-time filtering LD-algorithms can be used as a reliable computational alterna tive to the Kalman-type “standard algorithms” since they have the property being numerically stable to machine round-off errors due to the use of the MWGS orthogonalization computational procedure at each iteration of the algorithm. The results can be used to solve problems of measurement information processing in discrete-time systems with multiplicative and additive noise.

SVD-Based Identification of Parameters of the Discrete-Time Stochastic Systems Models with Multiplicative and Additive Noises Using Metaheuristic Optimization

The paper addresses a parameter identification problem for discrete-time stochastic systems models with multiplicative and additive noises. Stochastic systems with additive and multiplicative noises are considered when solving many practical problems related to the processing of measurements information. The purpose of this work is to develop a numerically stable gradient-free instrumental method for solving the parameter identification problems for a class of mathematical models described by discrete-time linear stochastic systems with multiplicative and additive noises on the basis of metaheuristic optimization and singular value decomposition. We construct an identification criterion in the form of the negative log-likelihood function based on the values calculated by the newly proposed SVD-based Kalman-type filtering algorithm, taking into account the multiplicative noises in the equations of the state and measurements. Metaheuristic optimization algorithms such as the GA (genetic algorithm) and SA (simulated annealing) are used to minimize the identification criterion. Numerical experiments confirm the validity of the proposed method and its numerical stability compared with the usage of the conventional Kalman-type filtering algorithm.

Linear Quadratic Optimal Control of Discrete-Time Stochastic Systems Driven by Homogeneous Markov Processes

Random terms in many natural and social science systems have distinct Markovian characteristics, such as Markov jump-taking values in a finite or countable set, and Wiener process-taking values in a continuous set. In general, these systems can be seen as Markov-process-driven systems, which can be used to describe more complex phenomena. In this paper, a discrete-time stochastic linear system driven by a homogeneous Markov process is studied, and the corresponding linear quadratic (LQ) optimal control problem for this system is solved. Firstly, the relations between the well-posedness of LQ problems and some linear matrix inequality (LMI) conditions are established. Then, based on the equivalence between the solvability of the generalized difference Riccati equation (GDRE) and the LMI condition, it is proven that the solvability of the GDRE is sufficient and necessary for the well-posedness of the LQ problem. Moreover, the solvability of the GDRE and the feasibility of the LMI condition are established, and it is proven that the LQ problem is attainable through a certain feedback control when any of the four conditions is satisfied, and the optimal feedback control of the LQ problem is given using the properties of homogeneous Markov processes and the smoothness of the conditional expectation. Finally, a practical example is used to illustrate the validity of the theory.

Necessary and Sufficient Conditions for Stabilizing an Uncertain Stochastic Multidelay System

We focus on the problem of asymptotically mean-square stabilization in discrete-time stochastic systems that exhibit plant uncertainty, multiple input delays, and multiplicative noises. Our innovative contributions are described as follows. First, we employ a reduction method to transform the original model into a delay-free auxiliary system, and establish an equivalent proposition for stabilization based on this reformulation. On the basis of the reformulated model, we propose two stabilization criteria for the uncertainty-free case, including both Lyapunov-type and Riccati-type criteria. More generally, we extend the stabilization result to the uncertain model, and propose a necessary and sufficient stabilization criterion utilizing matrix homogeneous polynomials. Finally, we explore the existence and uniqueness of a delay margin under certain structural restrictions, and provide a closed-form representation of this margin.

Jointly Distributed Filtering Based on Generalized Maximum Correntropy Criterion: Memory-Based Event-Triggered Cases

This article addresses jointly distributed entropy filtering issues based on the generalized maximum correntropy criterion (GMCC) for discrete-time stochastic parameter systems with fault and non-Gaussian noise effects. By taking current and historical triggered information, a memory-based event-triggered scheme with a time-varying threshold is put forward to govern the network communication. According to the constructed jointly distributed entropy filter with a two-step form, the upper bounds of the filtering error covariance matrices are derived and an ideal filter gain is obtained to maximize GMCC. Furthermore, an accessible gain is received via fixed-point iterative rules and the corresponding convergence is disclosed in theory. Finally, an application of the proposed distributed filter in ballistic object tracking is provided to show its effectiveness under non-Gaussian environments.

Active Fault Diagnosis for Stochastic Systems Within Bayesian Minimum Risk Decision Framework

In this paper, the active fault diagnosis (AFD) problem is investigated for a class of discrete-time stochastic systems. We consider multiple fault modes of the system, and the different types of misdiagnoses of these faults lead to different losses. A novel AFD framework is developed based on the Bayesian minimum risk decision framework. The misdiagnosis risk is derived to characterize the total losses caused by the misdiagnoses of all considered faults. The input is specifically designed to improve diagnosis performance by minimizing the risk of misdiagnosis. Since it is intractable to express the misdiagnosis risk in the closed form, we deduce an upper bound on it to establish a novel input design criterion based on the Bhattacharyya distance. The optimal input signal can be obtained for AFD via concave quadratic programming. For diagnosis performance enhancement and computation simplification purposes, we propose two new semi-online implementation manners of the input design for stochastic systems with and without multi-fault modes switching respectively. Given the optimal input signal and the proposed implementation manner, faults are diagnosed based on the minimum misdiagnosis risk decision principle. The effectiveness and superiority of the proposed AFD method are demonstrated by simulation results on a four-tank system.

Stabilization for Discrete-Time Stochastic Systems with Multiple Input Delays

Stabilization for Discrete-Time Stochastic Systems with Multiple Input Delays

Consensus control of second-order stochastic discrete-time multi-agent systems without velocity transmission

Consensus control of second-order stochastic discrete-time multi-agent systems without velocity transmission

Compositional synthesis of control barrier certificates for networks of stochastic systems against [formula omitted]-regular specifications

This paper is concerned with a compositional scheme for the construction of control barrier certificates for interconnected discrete-time stochastic systems. The main objective is to synthesize switching controllers against ω-regular properties that can be described by accepting languages of deterministic Streett automata (DSA) along with providing probabilistic guarantees for the satisfaction of such specifications. The proposed framework leverages the interconnection topology and a notion of so-called control sub-barrier certificates of subsystems, which are used to compositionally construct control barrier certificates of interconnected systems by imposing some dissipativity-type compositionality conditions. We propose a systematic approach to decompose high-level ω-regular specifications into simpler tasks by utilizing the automata corresponding to the specifications. In addition, we formulate an alternating direction method of multipliers (ADMM) optimization problem in order to obtain suitable control sub-barrier certificates of subsystems while satisfying compositionality conditions. For systems with polynomial dynamics, we provide a sum-of-squares (SOS) optimization problem for the computation of control sub-barrier certificates and local controllers of subsystems. Finally, we demonstrate the effectiveness of our proposed approaches by applying them to a physical case study.

On derivative-free extended Kalman filtering and its Matlab-oriented square-root implementations for state estimation in continuous-discrete nonlinear stochastic systems

Recent research in nonlinear filtering and signal processing has suggested an efficient derivative-free Extended Kalman filter (EKF) designed for discrete-time stochastic systems. Such approach, however, has failed to address the estimation problem for continuous-discrete models. In this paper, we develop a novel continuous-discrete derivative-free EKF methodology by deriving the related moment differential equations (MDEs) and sample point differential equations (SPDEs). Additionally, we derive their Cholesky-based square-root MDEs and SPDEs and obtain several numerically stable derivative-free EKF methods. Finally, we propose the MATLAB-oriented implementations for all continuous-discrete derivative-free EKF algorithms derived. They are easy to implement because of the built-in fashion of the MATLAB numerical integrators utilized for solving either the MDEs or SPDEs in use, which are the ordinary differential equations (ODEs). More importantly, these are accurate derivative-free EKF implementations because any built-in MATLAB ODE solver includes the discretization error control that bounds the discretization error arisen and makes the implementation methods accurate. Besides, this is done in automatic way and no extra coding is required from users. The new filters are particularly effective for working with stochastic systems with highly nonlinear and/or nondifferentiable drift and observation functions, i.e. when the calculation of Jacobian matrices are either problematical or questionable. The performance of the novel filtering methods is demonstrated on several numerical tests.

Design of reduced-order and pinning controllers for probabilistic Boolean networks using reinforcement learning

In this paper, we study a stabilization method for probabilistic Boolean networks (PBNs) using Q-learning, which is one of the typical methods in reinforcement learning. A PBN is a class of discrete-time stochastic logical systems in which update functions are randomly chosen from the set of the candidates of Boolean functions. In the existing methods using reinforcement learning, a design method of structured controllers has not been studied. In this paper, we propose reward design methods to derive reduced-order controllers and pinning controllers. The key idea is to adjust the structure of controllers by the reward for the structure of the Q-table. The advantage of the proposed method is that implementation is easy, because the proposed method can be embedded in the existing Q-learning-based stabilization algorithm. In design of pinning controllers, we can calculate not only controllers but also pinning nodes in which the control input is assigned. We demonstrate through numerical examples that the controller obtained from our proposed method is simpler than that from the existing method.

Distributional Robustness in Minimax Linear Quadratic Control with Wasserstein Distance

To address the issue of inaccurate distributions in discrete-time stochastic systems, a minimax linear quadratic control method using the Wasserstein metric is proposed. Our method aims to construct a control policy that is robust against errors in an empirical distribution of underlying uncertainty by adopting an adversary that selects the worst-case distribution at each time. The opponent receives a Wasserstein penalty proportional to the amount of deviation from the empirical distribution. As a tractable solution, a closed-form expression of the optimal policy pair is derived using a Riccati equation. We identify nontrivial stabilizability and observability conditions under which the Riccati recursion converges to the unique positive semidefinite solution of an algebraic Riccati equation. Our method is shown to possess several salient features, including closed-loop stability, a guaranteed-cost property, and a probabilistic out-of-sample performance guarantee.

Optimal transmission scheduling for remote state estimation in CPSs with energy harvesting two-hop relay networks

This paper considers the optimal transmission power scheduling for accurate remote state estimation in cyber–physical systems (CPSs). The plant is modeled as a discrete-time stochastic linear system with sensor measurements transmitted to the remote estimator over an intelligent relay. A dynamic cooperative team is formulated in which the sensor and the relay sharing the same cost function are equipped with corresponding energy harvesters, and need to decide on the transmission power levels to minimize the average error covariance at the remote estimator. Since the nodes make simultaneous decisions to achieve a common goal, the sensor and the relay do not have access to the transmission power that will be selected by another node in advance, and hence they cannot build a value estimation of the cost function directly at each time point, which poses a challenge to determine the optimal transmission policies of both sides. To overcome this challenge, we provide a local-action value function and prove that both nodes select the transmission power with respect to the local-action value function without any loss of performance. In addition, a Markov decision process (MDP) and a multi-agent reinforcement learning algorithm are presented to obtain the optimal transmission power schedule over an infinite time horizon. Finally, simulation examples are provided to corroborate and illustrate the theoretical results.

Compositional Construction of Safety Controllers for Networks of Continuous-Space POMDPs

In this article, we propose a compositional framework for the synthesis of safety controllers for networks of partially observable discrete-time stochastic control systems (also known as continuous-space partially observable Markov decision processes (POMDPs)). Given an estimator, we utilize a discretization-free approach to synthesize controllers ensuring safety specifications over finite-time horizons. The proposed framework is based on a notion of so-called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">local control barrier functions</i> computed for subsystems in two different ways. In the first scheme, no prior knowledge of estimation accuracy is needed. The second framework utilizes a probability bound on the estimation accuracy using a notion of so-called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">stochastic simulation functions</i> . In both proposed schemes, we derive sufficient small-gain-type conditions in order to compositionally construct control barrier functions for interconnected POMDPs using local barrier functions computed for subsystems. The constructed control barrier functions for the overall networks enable us to compute lower bounds on the probabilities that the interconnected POMDPs avoid certain unsafe regions in finite-time horizons. We demonstrate the effectiveness of our proposed approaches by applying them to an adaptive cruise control problem.