Markovian Control Problems Research Articles

Reinforcement Q-learning algorithm for H∞ tracking control of discrete-time Markov jump systems

In this paper, the H ∞ tracking control problem of linear discrete-time Markov jump systems is studied by using the data-based reinforcement learning method. Specifically, a new performance index function is established by using Markov chain and weighted sum technique, and thus the tracking game algebraic Riccati equation with weight vector and discount factor is obtained. A Q-learning algorithm is proposed to solve the tracking game algebra Riccati equation problem online without knowing the information of the system model. In addition, the convergence analysis of the algorithm is given, and it is proved that the added probing noise will not bias the algorithm. Finally, two simulation examples are given to verify the effectiveness of the proposed algorithm.

Fuzzy H∞ Control of Discrete-Time Nonlinear Markov Jump Systems via a Novel Hybrid Reinforcement Q -Learning Method.

In this article, a novel hybrid reinforcement Q -learning control method is proposed to solve the adaptive fuzzy H∞ control problem of discrete-time nonlinear Markov jump systems based on the Takagi-Sugeno fuzzy model. First, the core problem of adaptive fuzzy H∞ control is converted to solving fuzzy game coupled algebraic Riccati equation, which can hardly be solved by mathematical methods directly. To solve this problem, an offline parallel hybrid learning algorithm is first designed, where system dynamics should be known as a prior. Furthermore, an online parallel Q -learning hybrid learning algorithm is developed. The main characteristics of the proposed online hybrid learning algorithms are threefold: 1) system dynamics are avoided during the learning process; 2) compared with the policy iteration method, the restriction of the initial stable control policy is removed; and 3) compared with the value iteration method, a faster convergence rate can be obtained. Finally, we provide a tunnel diode circuit system model to validate the effectiveness of the present learning algorithm.

Improved aperiodic sampled-data control for singular Markov jump systems under incomplete transition descriptions

This paper studies the aperiodic sampled-data control problem of singular Markov jump systems under uncertain and unknown transition rates. First, taking into account the system information from x(tk) to x(tk+1), a novel two-sided looped mode-dependent Lyapunov–Krasovskii function (LKF) is constructed, which provides more freedom to test the feasibility of the LMI condition in the criterion. Based on this LKF and an improved integral inequality, a stochastic admissibility condition is given by linear matrix inequalities (LMIs), which is less conservative. Then, by using a lemma of equivalent set, a controller algorithm without equality constraint is proposed by LMIs to ensure the stability of the system. Finally, several examples are provided to verify the validity of the main results.

Asynchronous control for Markov jump systems subject to actuator saturation and partial mode information

This article studies the control problem of Markov jump systems subject to actuator saturation and partial mode information. The asynchronous phenomenon between the controller and the plant is addressed by introducing a hidden Markov model. By means of convex hull method, some sufficient conditions are obtained to guarantee the stochastic stability of the saturated Markov jump systems. Finally, a numerical example is given to illustrate the effectiveness of the algorithm.

Asynchronous Control of Markov Jump Systems Under Aperiodic DoS Attacks

The secure asynchronous control problem of Markov jump systems (MJSs) against aperiodic discrete denial-of-service (DoS) attacks is discussed in this brief. DoS attacks cut off the communication networks, resulting in the loss of state information. The impact of DoS attacks may cause mismatched behavior between controlled plant modes and controller modes, which can be illustrated by a hidden Markov model (HMM). An asynchronous controller is utilized to overcome the mismatched phenomenon. By employing multiple Lyapunov functions and iterative methods, and analyzing the conditions on malicious DoS attacks, sufficient conditions are derived to guarantee mean square stable of MJSs under aperiodic DoS attacks. By solving a set of linear matrix inequalities (LMIs), the asynchronous controller is designed. At last, the effectiveness of the proposed theoretical method is certified by a numerical simulation.

Zero‐sum game‐based security control of unknown nonlinear Markov jump systems under false data injection attacks

In order to solve the security control problem of discrete-time unknown Markov jump systems (MJSs) subject to false data injection attacks, a new zero-sum game-based control method is developed in this paper by using model-free adaptive dynamic programming algorithm. The controller and the attacker are regarded as two players. An optimal cost function of MJSs is firstly introduced based on the evolution of the Markov chain. Then, a critic-action-attack framework is established to approximate the cost function, control input and false data, respectively. Additionally, the boundedness of the neural network (NN) weight estimate errors is guaranteed via Lyapunov theory. In the end, the validity of the proposed approach is verified by simulation of a single link robot arm.

Input-output finite-time control of Markov jump systems with round-robin protocol: An dynamic event-triggered approach

This article investigates the control problem of networked Markov jump systems (MJSs). First, to describe the asynchronization between the plant modes and the controller modes in MJSs, the hidden Markov model is introduced and the asynchronous control technique is developed. Next, the dynamic event-triggered mechanism (ETM) is utilized to decrease the frequency of data transmission. Moreover, the round-robin protocol (RRP) is introduced to reduce the amount of communicated data by allowing only one node to access the network. At last, the concept of input-output finite time stability (IO-FTS) is introduced and taken into consideration in the controller design. The highlight of this work is the introduction of both the dynamic ETM and the RRP to alleviate the communication load. Finally, a simulation example is proposed to illustrate the effectiveness of the theoretical results.

Optimal control of nonlinear Markov jump systems by control parametrisation technique

This paper considers an optimal control problem of nonlinear Markov jump systems with continuous state inequality constraints. Due to the presence of continuous-time Markov chain, no existing computation method is available to solve such an optimal control problem. In this paper, a derandomisation technique is introduced to transform the nonlinear Markov jump system into a deterministic system, which simultaneously gives rise to an equivalent deterministic dynamic optimisation problem. The control parametrisation technique is then used to partition the time horizon into a sequence of subintervals such that the control function is approximated by a piecewise constant function consistent with the partition. The heights of the piecewise constant function on the corresponding subintervals are taken as decision variables to be optimised. In this way, the approximate dynamic optimisation problem is an optimal parameter selection problem, which can be viewed as a finite dimensional optimisation problem. To solve it using a gradient-based optimisation method, the gradient formulas of the cost function and the constraint functions are derived. Finally, a real-world practical problem involving a bioreactor tank model is solved using the method proposed.

TIME-INCONSISTENT MARKOVIAN CONTROL PROBLEMS UNDER MODEL UNCERTAINTY WITH APPLICATION TO THE MEAN-VARIANCE PORTFOLIO SELECTION

In this paper, we study a class of time-inconsistent terminal Markovian control problems in discrete time subject to model uncertainty. We combine the concept of the sub-game perfect strategies with the adaptive robust stochastic control method to tackle the theoretical aspects of the considered stochastic control problem. Consequently, as an important application of the theoretical results and by applying a machine learning algorithm we solve numerically the mean-variance portfolio selection problem under the model uncertainty.

Static Output Constrained Control for Discrete-Time Hidden Markov Jump Linear Systems

This paper studies the static output quadratic control problem of discrete-time Markov jump linear systems (MJLS) with hard constraints on the norm of the state and control variables. Both cases the finite horizon as well as the infinite horizon are considered. Regarding the Markov chain parameter $\theta (k)$ , it is assumed that the controller only has access to a detector which emits signals $\widehat {\theta }(k)$ providing information on the parameter $\theta (k)$ . The goal is to design a static output feedback linear control using the information provided by detector $\widehat {\theta }(k)$ in order to minimize an upper bound for the quadratic cost and satisfy the hard constraints. For the infinite horizon case it is also imposed that the controller stochastically stabilizes the closed loop system. LMIs (linear matrix inequalities) are formulated in order to obtain a solution for these optimization problems. The cases in which the initial conditions are fixed and when it is desired to maximize an estimate of the domain of an invariant set are also analyzed. Some numerical examples are presented for the purpose of illustrating the results obtained.

A dual representation result for value functions in stochastic control of infinite dimensional groups

We study here the problem of dual representation of the value functions associated to linear-convex stochastic control problems in infinite dimensional Hilbert spaces. Since the dual state runs backwards in time, it turns out that the dual representation has the meaning of a classical (Markov) control problem only if the primal linear state equation is driven by the generator of a group. In the general case, a dual representation of the value function still holds, but such a representation cannot be reduced to solving a dual Hamilton-Jacobi-Bellman equation.

A Detector-Based Approach for the Constrained Quadratic Control of Discrete-Time Markovian Jump Linear Systems

This paper considers the quadratic control problem of discrete-time Markov jump linear systems with constraints on the norm of the state and control variables. We assume that the Markov chain parameter is not available, and instead, there is a detector, which emits signals providing information on this parameter. It is desired to derive a feedback linear control using the information provided by this detector in order to stochastically stabilize the closed-loop system, satisfy the constraints whenever the initial conditions belong to an invariant set, and minimize an upper bound for the quadratic cost. We show that a linear matrix inequality (LMI) optimization problem can be formulated in order to obtain a solution for this problem. Two other related problems, one for minimizing the guaranteed quadratic cost considering fixed initial conditions and the other for maximizing an estimate of the domain of an invariant set, can also be formulated using our LMI approach. This paper is concluded with some numerical simulations.

Fault-Tolerant Control of Time-Delay Markov Jump Systems With <inline-formula> <tex-math notation="LaTeX"> $It\hat{o}$</tex-math> </inline-formula> Stochastic Process and Output Disturbance Based on Sliding Mode Observer

This paper focuses on the fault-tolerant control problem of Markov jump systems (MJS) with $It\hat{o}$ stochastic process and output disturbances. Such a problem widely exists in practical systems such as mobile manipulator systems. Since MJS can suitably describe mobile manipulator systems, in this paper, a new approach based on the MJS model is proposed. First, a proportional-derivative sliding mode observer (SMO) and an observer-based controller are designed and synthesized. Two new theorems are derived to ensure the close-loop stochastic stability and the reachability of the sliding mode surface. Compared with the existing works, the system model is more general, which could describe a larger variety of plants or processes. The controller design procedure is simplified by solving the sliding mode parameters and the controller gain simultaneously with only one linear matrix inequality problem. In addition, the augmented fault vector can be reconstructed by employing a descriptor SMO. Simulations are provided to demonstrate the validity of the derived theorems and the effectiveness of the proposed algorithm.

A two-dimensional control problem arising from dynamic contracting theory

We study a dynamic corporate finance contracting model in which the firm’s profitability fluctuates and is impacted by the unobservable managerial effort. Thereby, we introduce in an agency framework the issue of strategic liquidation. We show that the principal’s problem takes the form of a two-dimensional fully degenerate Markov control problem. We prove regularity properties of the value function and derive explicitly the optimal contract that implements full effort. Our regularity results appear in some recent studies, but with heuristic proofs that do not clarify the importance of the regularity of the value function at the boundaries.

Sliding mode control for state-delayed Markov jump systems with partly unknown transition probabilities

In this paper, we probe the $$H_{\infty }$$ control problem of state-delayed Markov jump systems with partly unknown transition probabilities by using sliding mode control approach. The $$H_{\infty }$$ performance index is to improve the system performance of the control system. The exact information of transition probabilities is assumed to be partially known, and the bounds of nonlinear function are unknown. Firstly, a sliding mode observer is designed to estimate the unmeasured state. Secondly, an integral sliding mode surface is constructed such that the reduced-order sliding motion is insensitive to all admissible uncertainties, nonlinearities and external disturbances. Thirdly, we design an adaptive sliding mode controller to maintain the state trajectories in the sliding mode surface. The sufficient condition of stochastic stability for the closed-loop system is derived via Lyapunov stability theory. Finally, a numerical example is given to show the effectiveness of the proposed results.

A Stochastic Model with Inflation, Growth and Technology for the Political Business Cycle

This paper analyzes an augmented political business cycle model taking into account the effect of employment creation decisions by the ruling party jointly on inflation and growth. The objective is to maximize voter support in the next election that depends on the rate of unemployment as well as that of growth and inflation. We allow for stochasticity in the New Keynesian Phillips Curve model for the relationship between inflation and unemployment as well as in a benchmark labour productivity function for analyzing the growth rate. We provide explicit solution paths of the affine Markov control problem that results from our formulation. We also provide numerical illustrations with plausible parametric configurations to generate more insight into our model. Our results are broadly in line with the conventional wisdom of Phillips curve, with inflation and unemployment being roughly negatively related. The growth rate is, as expected, negatively related to the unemployment. We observe that, in the sequel, it is the lowering of the rate of inflation that provides support for the ruling party. Thus, electoral pressures drive the government to engage in cost control rather than productive investment (e.g., boosting employment or output growth).

Observer-based sliding mode control of Markov jump systems with random sensor delays and partly unknown transition rates

ABSTRACTIn this paper, the sliding mode control problem of Markov jump systems (MJSs) with unmeasured state, partly unknown transition rates and random sensor delays is probed. In the practical engineering control, the exact information of transition rates is hard to obtain and the measurement channel is supposed to subject to random sensor delay. Design a Luenberger observer to estimate the unmeasured system state, and an integral sliding mode surface is constructed to ensure the exponential stability of MJSs. A sliding mode controller based on estimator is proposed to drive the system state onto the sliding mode surface and render the sliding mode dynamics exponentially mean-square stable with H∞ performance index. Finally, simulation results are provided to illustrate the effectiveness of the proposed results.

H2-control and the separation principle for discrete-time jump systems with the Markov chain in a general state space

ABSTRACTThis paper deals with the H2 optimal control problem of discrete-time Markov jump linear systems (MJLS) considering the case in which the Markov chain takes values in a general Borel space . It is assumed that the controller has access only to an output variable and to the jump parameter. The goal, in this case, is to design a dynamic Markov jump controller such that the H2-norm of the closed-loop system is minimised. It is shown that the H2-norm can be written as the sum of two H2-norms, such that one of them does not depend on the control, and the other one is obtained from the optimal filter for an infinite-horizon filtering problem. This result can be seen as a separation principle for MJLS with Markov chain in a Borel space considering the infinite time horizon case.

Optimal Control of DC-DC Buck Converter via Linear Systems With Inaccessible Markovian Jumping Modes

The note presents an algorithm for the average cost control problem of continuous-time Markov jump linear systems. The controller assumes a linear state-feedback form and the corresponding control gain does not depend on the Markov chain. In this scenario, the control problem is that of minimizing the long-run average cost. As an attempt to solve the problem, we derive a global convergent algorithm that generates a gain satisfying necessary optimality conditions. Our algorithm has practical implications, as illustrated by the experiments that were carried out to control an electronic dc-dc buck converter. The buck converter supplied a load that suffered abrupt changes driven by a homogeneous Markov chain. Besides, the source of the buck converter also suffered abrupt Markov-driven changes. The experimental results support the usefulness of our algorithm.

Quadratic control with partial information for discrete-time jump systems with the Markov chain in a general Borel space

This paper deals with the finite horizon quadratic optimal control problem of discrete-time Markov jump linear systems (MJLS) considering the case in which the Markov chain takes values in a general Borel space S. It is assumed that the controller has access to an output variable as well as the jump parameter. The goal is to design a dynamic Markov jump controller such that the closed loop system minimizes the quadratic functional cost of the system over a finite horizon period of time. It is shown that an optimal controller can be obtained from two S-coupled difference Riccati equations, one associated to a filtering problem and the other one associated to a control problem in which the state variable is fully available. By S-coupled it is meant that the difference Riccati equations are coupled via an integral over S. This result, which can be seen as a separation principle for discrete-time MJLS, generalizes previous ones that were restricted to the Markov chain taking values in a finite set.