Linear Discrete-time Stochastic Systems Research Articles

Adaptive Linear Quadratic Control for Stochastic Discrete-Time Linear Systems With Unmeasurable Multiplicative and Additive Noises

Adaptive Linear Quadratic Control for Stochastic Discrete-Time Linear Systems With Unmeasurable Multiplicative and Additive Noises

Quadratic Tracking Control of Linear Stochastic Systems with Unknown Dynamics Using Average Off-Policy Q-Learning Method

This article investigates the optimal tracking control problem for data-based stochastic discrete-time linear systems. An average off-policy Q-learning algorithm is proposed to solve the optimal control problem with random disturbances. Compared with the existing off-policy reinforcement learning (RL) algorithm, the proposed average off-policy Q-learning algorithm avoids the assumption of an initial stability control. First, a pole placement strategy is used to design an initial stable control for systems with unknown dynamics. Second, the initial stable control is used to design a data-based average off-policy Q-learning algorithm. Then, this algorithm is used to solve the stochastic linear quadratic tracking (LQT) problem, and a convergence proof of the algorithm is provided. Finally, numerical examples show that this algorithm outperforms other algorithms in a simulation.

SVD-Based Parameter Identification of Discrete-Time Stochastic Systems with Unknown Exogenous Inputs

This paper addresses the problem of parameter identification for discrete-time stochastic systems with unknown exogenous inputs. These systems form an important class of dynamic stochastic system models used to describe objects and processes under a high level of a priori uncertainty, when it is not possible to make any assumptions about the evolution of the unknown input signal or its statistical properties. The main purpose of this paper is to construct a new SVD-based modification of the existing Gillijns and De Moor filtering algorithm for linear discrete-time stochastic systems with unknown exogenous inputs. Using the theoretical results obtained, we demonstrate how this modified algorithm can be applied to solve the problem of parameter identification. The results of our numerical experiments conducted in MATLAB confirm the effectiveness of the SVD-based parameter identification method that was developed, under conditions of unknown exogenous inputs, compared to maximum likelihood parameter identification when exogenous inputs are known.

Verification of Parametric Properties of Linear Discrete-time Stochastic Systems

We study the problem of verifying parametric properties of linear discrete-time stochastic systems (linear DTSSs) over bounded and unbounded time horizons. A linear DTSS captures random processes, where the one-step transition relation between the current random vector X and the next-step random vector X’ is linear and is given by X’ = AX + W, where A is an n x n matrix and W is a random noise vector. We assume that the initial and noise random vectors are multivariate normal, each of which is expressed as a pair of two parameters, namely, mean and covariance. We are interested in verifying whether the system always satisfies a given parametric property specified as a subset of the parameters’ space over bounded or unbounded time horizons. For bounded time steps, we reduce the problem to the satisfiability of a semidefinite programming problem. For the unbounded time steps, we propose a novel abstraction procedure to reduce the verification problem to that of a finite graph, wherein, the nodes of the graph correspond to the regions of a partition of the random vector space, in contrast to existing works that partition the state-space. More precisely, we partition the parameter space of normal random vectors, namely, the space of means and covariance matrices, and apply semi-definite programming to compute the edges. We show that our abstraction procedure is sound. Our experiments with varying number of vehicle dynamics and different dimensions of random models show that our methods are reasonably scalable.

Robust Estimators for Missing Observations in Linear Discrete-Time Stochastic Systems with Uncertainties

As a first approach to estimating the signal and the state, Theorem 1 proposes recursive least-squares (RLS) Wiener fixed-point smoothing and filtering algorithms that are robust to missing measurements in linear discrete-time stochastic systems with uncertainties. The degraded quantity is given by multiplying the Bernoulli random variable by the degraded signal caused by the uncertainties in the system and observation matrices. The degraded quantity is observed with additional white observation noise. The probability that the degraded signal is present in the observation equation is assumed to be known. The design feature of the proposed robust estimators is the fitting of the degraded signal to a finite-order autoregressive (AR) model. Theorem 1 is transformed into Corollary 1, which expresses the covariance information in a semi-degenerate kernel form. The autocovariance function of the degraded state and the cross-covariance function between the nominal state and the degraded state is expressed in semi-degenerate kernel forms. Theorem 2 shows the robust RLS Wiener fixed-point and filtering algorithms for estimating the signal and state from degraded observations in the second method. The robust estimation algorithm of Theorem 2 has the advantage that, unlike Theorem 1 and the usual studies, it does not use information on the existence probability of the degraded signal. This is a unique feature of Theorem 2.

Parameter Identification of the Discrete-Time Stochastic Systems with Multiplicative and Additive Noises Using the UD-Based State Sensitivity Evaluation

The paper proposes a new method for solving the parameter identification problem for a class of discrete-time linear stochastic systems with multiplicative and additive noises using a numerical gradient-based optimization. The constructed method is based on the application of a covariance UD filter for the above systems and an original method for evaluating state sensitivities within the numerically stable, matrix-orthogonal MWGS transformation. In addition to the numerical stability of the proposed algorithm to machine roundoff errors due to the application of the MWGS-UD orthogonalization procedure at each step, the main advantage of the obtained results is the possibility of analytical calculation of derivatives at a given value of the identified parameter without the need to use finite-difference methods. Numerical experiments demonstrate how the obtained results can be applied to solve the parameter identification problem for the considered stochastic system model.

Parameter identification of the linear discrete-time stochastic systems with unknown exogenous inputs

The paper addresses a parameter identification problem for linear discrete-time stochastic systems with unknown exogenous inputs. Such systems are considered when solving practical problems related to the measurements processing in the case when it is impossible to do any assumptions about the evolution of unknown input signal or its statistical characteristics that can change over time. We consider a class of discrete time linear stochastic systems with unknown exogenous inputs, where an additional source of a priori uncertainty of the system model is introduced, namely, the unknown parameter, on the elements of which the system model matrices can depend. This formulation of the parameter identification problem under the conditions of unknown inputs and the presence of random noises describes a high degree of uncertainty of a discrete time linear stochastic system. We propose a novel solution to this problem based on the construction of a new instrumental identification criterion. Minimization of this criterion allows for evaluating the unknown system model parameters simultaneously with the estimating of the state vector and unknown exogenous inputs of the system. Numerical experiments confirm the validity and efficiency of the proposed parameter identification method.

Алгоритмы дискретной фильтрации на основе модифицированной взвешенной ортогонализации Грама –Шмидта для дискретных стохастических систем с мультипликативными и аддитивными шумами

Discrete-time stochastic systems with multiplicative and additive noises describe a wide class of mathematical models of complex systems, for example, industrial-technological, energy, economic, telecommunication, aerospace systems, etc. An important class of algorithms for processing measure ment information in complex systems are Kalman-type discrete-time filtering algorithms. Purpose. Construction of new discrete-time filtering algorithms for discrete-time linear systems with multiplicative and additive noises based on numerically stable modified weighted Gram – Schmidt orthogonalization (MWGS). Methodology. The methods of computational linear algebra were used, namely, the direct procedu re of MWGS-orthogonalization, the theory of Kalman filtering, methods of scientific programming in MATLAB. Findings. New LD-algorithms for discrete-time filtering in covariance and informational form for discrete-time stochastic systems with multiplicative and additive noises are constructed. The algorithms have an extended array form allowing updates for all necessary filter values. The method employs numerically stable modified weighted Gram – Schmidt orthogonalization. Algebraic equiva lence of LD-filters to Kalman-type covariance and informational algorithms for linear discrete time stochastic systems with multiplicative and additive noises is proved. The conducted numerical experiments have shown the effectiveness of the proposed algorithms using the example of solving the problem of parametric estimation of a model of almost rectilinear motion, as well as their savings in computation time compared to previously constructed UD filters. Value. New discrete-time filtering LD-algorithms can be used as a reliable computational alterna tive to the Kalman-type “standard algorithms” since they have the property being numerically stable to machine round-off errors due to the use of the MWGS orthogonalization computational procedure at each iteration of the algorithm. The results can be used to solve problems of measurement information processing in discrete-time systems with multiplicative and additive noise.

SVD-Based Identification of Parameters of the Discrete-Time Stochastic Systems Models with Multiplicative and Additive Noises Using Metaheuristic Optimization

The paper addresses a parameter identification problem for discrete-time stochastic systems models with multiplicative and additive noises. Stochastic systems with additive and multiplicative noises are considered when solving many practical problems related to the processing of measurements information. The purpose of this work is to develop a numerically stable gradient-free instrumental method for solving the parameter identification problems for a class of mathematical models described by discrete-time linear stochastic systems with multiplicative and additive noises on the basis of metaheuristic optimization and singular value decomposition. We construct an identification criterion in the form of the negative log-likelihood function based on the values calculated by the newly proposed SVD-based Kalman-type filtering algorithm, taking into account the multiplicative noises in the equations of the state and measurements. Metaheuristic optimization algorithms such as the GA (genetic algorithm) and SA (simulated annealing) are used to minimize the identification criterion. Numerical experiments confirm the validity of the proposed method and its numerical stability compared with the usage of the conventional Kalman-type filtering algorithm.

Linear Quadratic Optimal Control of Discrete-Time Stochastic Systems Driven by Homogeneous Markov Processes

Random terms in many natural and social science systems have distinct Markovian characteristics, such as Markov jump-taking values in a finite or countable set, and Wiener process-taking values in a continuous set. In general, these systems can be seen as Markov-process-driven systems, which can be used to describe more complex phenomena. In this paper, a discrete-time stochastic linear system driven by a homogeneous Markov process is studied, and the corresponding linear quadratic (LQ) optimal control problem for this system is solved. Firstly, the relations between the well-posedness of LQ problems and some linear matrix inequality (LMI) conditions are established. Then, based on the equivalence between the solvability of the generalized difference Riccati equation (GDRE) and the LMI condition, it is proven that the solvability of the GDRE is sufficient and necessary for the well-posedness of the LQ problem. Moreover, the solvability of the GDRE and the feasibility of the LMI condition are established, and it is proven that the LQ problem is attainable through a certain feedback control when any of the four conditions is satisfied, and the optimal feedback control of the LQ problem is given using the properties of homogeneous Markov processes and the smoothness of the conditional expectation. Finally, a practical example is used to illustrate the validity of the theory.

Optimal transmission scheduling for remote state estimation in CPSs with energy harvesting two-hop relay networks

This paper considers the optimal transmission power scheduling for accurate remote state estimation in cyber–physical systems (CPSs). The plant is modeled as a discrete-time stochastic linear system with sensor measurements transmitted to the remote estimator over an intelligent relay. A dynamic cooperative team is formulated in which the sensor and the relay sharing the same cost function are equipped with corresponding energy harvesters, and need to decide on the transmission power levels to minimize the average error covariance at the remote estimator. Since the nodes make simultaneous decisions to achieve a common goal, the sensor and the relay do not have access to the transmission power that will be selected by another node in advance, and hence they cannot build a value estimation of the cost function directly at each time point, which poses a challenge to determine the optimal transmission policies of both sides. To overcome this challenge, we provide a local-action value function and prove that both nodes select the transmission power with respect to the local-action value function without any loss of performance. In addition, a Markov decision process (MDP) and a multi-agent reinforcement learning algorithm are presented to obtain the optimal transmission power schedule over an infinite time horizon. Finally, simulation examples are provided to corroborate and illustrate the theoretical results.

A New Event-Triggered Control Scheme for Stochastic Systems

This article studies event-triggered control of stochastic linear discrete-time systems with discounted quadratic cost functions. A new dynamic event-triggering condition is proposed, which has simultaneously stochastic and deterministic features. The designed event-triggered control system ensures the control performance to be within a desirable level relative to that using periodic time-triggered control, while discarding the unnecessary transmissions. By adjusting the parameters, the proposed event-triggering condition can be reduced to some existing ones in the literature. It is shown that the three features (dynamic, stochastic, and deterministic) are all helpful to further increase the average interevent times. Then, the criteria in terms of the parameters are presented to ensure mean-square stability of the closed-loop systems. Moreover, an improved version of the proposed event-triggering condition is given to enlarge the minimum interevent times. Finally, numerical simulations are given to illustrate the efficiency and feasibility of the proposed results.

Recursive Least-Squares Wiener Consensus Filter and Fixed-Point Smoother in Distributed Sensor Networks

Distributed Kalman filter (DKF) is classified into the information fusion Kalman filter (IFKF), i. e. the centralized Kalman filter (CKF), and the Kalman consensus filter (KCF) in distributed sensor networks. The KCF has the advantage to improve the estimate of the state at the sensor node uniformly by incorporating the information of the observations and the filtering estimates at the neighbor nodes. In the first devised KCF, a user adjusts the consensus gain. This paper designs the recursive least-squares (RLS) Wiener consensus filter and fixed-point smoother that do not need to be adjusted in linear discrete-time stochastic systems. In addition to the observation equation at the sensor node, an observation equation is introduced excessively. Here, the new observation is the sum of the filtering estimates of the signals at the neighbor nodes of the sensor node. Thus, it is interpreted that the RLS Wiener consensus estimators incorporate the information of the observations at the neighbor nodes indirectly because the observations are used in the calculations of the filtering estimates. A numerical simulation example shows that the proposed RLS Wiener consensus filter and fixed-point smoother are superior in estimation accuracy to the RLS Wiener estimators.

Robust H2 Analysis of Discrete-Time Linear Systems Characterized by Random Polytopes and Time-Varying Parameters

This paper deals with discrete-time linear stochastic systems characterized by random polytopes and time-varying parameters. Random polytopes and time-varying parameters are introduced so that they can represent a sort of temporal variations in the distributions of coefficient random matrices of the systems. An H2 norm is defined for associated stochastic systems, and H2 performance analysis is discussed. In particular, we derive a numerically tractable linear matrix inequality condition for such analysis, as an extension of our earlier results about H2 control for stochastic systems without random polytopes.

Robust Control of Discrete-Time Systems With Coefficient Matrices Given by Polytopic Martingales

This paper is concerned with robust control of discrete-time linear stochastic systems with coefficient matrices given by polytopic martingales. To the best of our knowledge, this class of stochastic systems have not been dealt with as a target of control due to the absence of required theory. For such systems, we discuss the following two types of approaches for robust stabilization: One is the proposed stochastic control approach using the martingale property of the coefficient matrices, and the other is a deterministic control approach without using the information. Through theoretical and numerical comparisons of the two approaches, we demonstrate the effectiveness of the proposed stochastic control approach in the sense of conservativeness.

Minimum-Variance and Low-Complexity Data-Driven Probabilistic Safe Control Design

This paper presents a direct minimum-variance (MV) data-driven safe control design approach for uncertain linear discrete-time stochastic systems. The superiority of the direct MV approach is shown by developing and comparing direct versus indirect learning approaches and MV versus certainty-equivalence (CE) approaches. First, it is shown that probabilistic safety can be guaranteed by ensuring probabilistic λ-contractivity of the safe set. Four data-based convex optimization-based algorithms are introduced to ensure the probabilistic λ-contractivity of the safe set (i.e., direct and indirect CE, direct and indirect MV). It is shown that while the CE approach results in a risk-neutral control design method with no robustness guarantees, the MV approach results in a risk-averse control design with probabilistic safety guarantees. This is because MV approach aims at learning a control gain that minimizes the variance of the state of the closed-loop system with respect to the safe set, and thus minimizes the risk of safety violation. Besides, it is shown that the direct learning approach requires weaker data richness conditions (i.e., lower sample complexity) than the indirect learning approach. Two simulation examples are provided to verify that the direct MV learning approach outperforms the other three approaches since it leads to low-complexity (i.e., low sample complexity and convex optimization) safe learning with a high probability of safety guarantees.

H-Infinity Tracking Controller for Linear Discrete-Time Stochastic Systems with Uncertainties

H-Infinity Tracking Controller for Linear Discrete-Time Stochastic Systems with Uncertainties

Security Investment in Cyber-Physical Systems: Stochastic Games With Asymmetric Information and Resource-Constrained Players

This article considers remote state estimation in cyber-physical systems (CPSs) with multiple sensors. Each plant is modeled by a discrete-time stochastic linear system with measurements of each sensor transmitted to the corresponding remote estimator over a shared communication network when their securities are interdependent due to network-induced risks. A dynamic nonzero-sum game with asymmetric information is formulated in which each sensor subject to a resource budget constraint needs to decide whether to invest in security for sending data packets, taking the behaviors of other sensors into account. To overcome the difficulty in characterizing or computing the Nash equilibria (NE), the game with asymmetric information is transformed into another game with symmetric information such that the equilibrium of the original game can be obtained by solving the equilibrium of the new game. Under certain conditions, we devise a backward induction algorithm to obtain a subclass of NE of the original game, known as common information-based Markov perfect equilibria (CIBMPE). Finally, a numerical example is provided to illustrate the results obtained.

Minimal-order observer-based distributed fault detection and isolation for stochastic multi-agent systems

In this paper, the problem of distributed fault detection and isolation (FDI) is investigated for a class of linear discrete-time stochastic multi-agent systems (MASs) with additive Gaussian white noises. By using the information received from the generalized neighbor agents, a set of residual generators are designed for one agent based on the minimal-order observers. After dividing the MAS into several first-order components, the residuals are designed to be robust to the faults in some designated components and sensitive to the faults in all the other components. Combining with FDI strategies, multiple concurrent faults in the generalized neighbor agents can be detected and isolated simultaneously. In addition, a necessary condition is established for the observer to have the minimum order. By means of the statistical method, a set of hypothesis tests are derived to detect and isolate the faults. Finally, a simulation example is presented to show the feasibility and effectiveness of the proposed methods.

Optimal Sensor Precision for Multirate Sensing for Bounded Estimation Error

We address the problem of determining optimal sensor precisions for estimating the states of linear time-varying discrete-time stochastic dynamical systems, with guaranteed bounds on the estimation errors. This is performed in the Kalman filtering framework, where the sensor precisions are treated as variables. They are determined by solving a constrained convex optimization problem, which guarantees the specified upper bound on the posterior error variance. Optimal sensor precisions are determined by minimizing the l1 norm, which promotes sparseness in the solution and indirectly addresses the sensor selection problem. The theory is applied to realistic flight mechanics and astrodynamics problems to highlight its engineering value. These examples demonstrate the application of the presented theory to a) determine redundant sensing architectures for linear time invariant systems, b) accurately estimate states with low-cost sensors, and c) optimally schedule sensors for linear time-varying systems.