Markov Control Research Articles

Policy iteration-based adaptive optimal control for Markov jump systems: a transition-probability-free asynchronous approach

Policy iteration-based adaptive optimal control for Markov jump systems: a transition-probability-free asynchronous approach

Simulated annealing algorithm-assisted asynchronous control for networked Markov jump systems under DoS attacks

Simulated annealing algorithm-assisted asynchronous control for networked Markov jump systems under DoS attacks

Hybrid-triggered H∞ control for Markov jump systems with quantizations and hybrid attacks

The hybrid-triggered quantized H∞ control problem is investigated for discrete-time Markov jump systems (MJSs) under hybrid cyber attacks. A novel hybrid-triggered mechanism obeying Bernoulli distribution between the time-triggered mechanism and the adaptive event-triggered mechanism is introduced. The triggered condition considers the average value between current measured output and latest triggered output to avoid the unnecessary triggered data released. Meanwhile, a quantizer is adopted to optimize the data transmission rate and an observer-based controller is designed to resist the impact of deception attacks and aperiodic DoS attacks on the system. Utilizing Lyapunov stability theory and iterative methods, sufficient conditions are obtained to ensure that the closed-loop MJSs are asymptotically mean-square stable with H∞ performance. Then, an algorithm for gain matrices and triggered matrices is given. Finally, the effectiveness and availability of the proposed method are verified by a numerical example and a DC motor model.

A new insight on the event‐triggered state feedback control for Markov jump systems

A new insight on the event‐triggered state feedback control for Markov jump systems

Reinforcement Q-learning algorithm for H∞ tracking control of discrete-time Markov jump systems

In this paper, the H ∞ tracking control problem of linear discrete-time Markov jump systems is studied by using the data-based reinforcement learning method. Specifically, a new performance index function is established by using Markov chain and weighted sum technique, and thus the tracking game algebraic Riccati equation with weight vector and discount factor is obtained. A Q-learning algorithm is proposed to solve the tracking game algebra Riccati equation problem online without knowing the information of the system model. In addition, the convergence analysis of the algorithm is given, and it is proved that the added probing noise will not bias the algorithm. Finally, two simulation examples are given to verify the effectiveness of the proposed algorithm.

Event-triggered finite-time fault-tolerance control and simultaneous disturbance rejection for Markov jump systems with general transition probabilities

This paper employs an event-triggered control approach to investigate the simultaneous finite-time fault-tolerant control and disturbance rejection problem for stochastic Markovian jump systems with general transition probabilities. Firstly, in conjunction with the event-triggered mechanism, a novel composite observer is designed, which can not only simultaneously estimate states and faults as well as disturbances of the system, but also guarantee that the error system is stochastically finite-time bounded. Subsequently, leveraging the obtained estimations, an active fault-tolerant controller with anti-disturbance performance is constructed to guarantee the finite-time boundedness of closed-loop system. Finally, the effectiveness of the proposed scheme is verified by distinct examples.

Adaptive distributed tracking control for Markov jump multiagent systems with a non-strict leader

This paper investigates the leader-following mean square distributed tracking control problem (DTCP) for a Markov jump multiagent systems (MJMASs) with the general linear dynamical model over an undirected graph. The non-strict control input of the leader is non-zero, unknown, and unavailable to any followers. First, the jump characteristics of the system for the single agent is modeled by a continuous Markov chain. Second, in order to address the leader–follower mean square DTCP, the distributed mode-dependent static controller and the fully distributed mode-dependent adaptive controller are designed for all followers based on the information from the agents’ neighbors and itself, respectively. Then, to further analyze the stability of the MJMASs, two categories of decomposed related Lyapunov functions are constructed using Lyapunov stability theory. The sufficient conditions for the existence of the distributed controller are obtained by derivation. Finally, two simulation cases are presented to verify the validity and feasibility of the proposed strategies.

Maximizing the probability of visiting a set infinitely often for a Markov decision process with Borel state and action spaces

Abstract We consider a Markov control model with Borel state space, metric compact action space, and transitions assumed to have a density function with respect to some probability measure satisfying some continuity conditions. We study the optimization problem of maximizing the probability of visiting some subset of the state space infinitely often, and we show that there exists an optimal stationary Markov policy for this problem. We endow the set of stationary Markov policies and the family of strategic probability measures with adequate topologies (namely, the narrow topology for Young measures and the $ws^\infty$ -topology, respectively) to obtain compactness and continuity properties, which allow us to obtain our main results.

Event-triggered secure control for Markov jump neural networks with time-varying delays and subject to cyber-attacks via state estimation fuzzy approach

The investigation uses a state estimation fuzzy technique to address the problem of dynamic event-triggered secure control in Markov jump neural networks with time-varying delays subject to cyber-attacks. Two different deception attacks are given, each employing stochastic variables with a Bernoulli distribution. Specifically, this article presents a dynamic event-triggered strategy and energy restrictions to reduce network traffic and protect network resources during deception attacks. To prevent the system state from becoming unavailable, a state estimator is concurrently developed using the output signal. The application of Lyapunov stability theory yields sufficient criteria that ensure neural network stability. The study also accurately describes the controller gain's by solving Linear matrix inequality. Numerical examples are provided to demonstrate the theoretical results.

Tracking control of heterogeneous Markov jump multi-agent systems based on PI observer

This paper discusses the problem of tracking control for heterogeneous Markov jump multi-agent systems (MASs) with directed communication topology. For each follower, a distributed Proportional-Integral (PI) observer is constructed. A synchronous tracking controller is developed by considering the state of the node itself and the observation state of the observer, in conjunction with the feedback control strategy. To address the inconsistency between the controller mode and the system mode, the hidden Markov model (HMM) is employed to describe the relationship between different modes, an asynchronous controller with dual-mode dependence is constructed to stabilize the system and the stability criterion is obtained. Simulation results demonstrate that, under the condition of system mode transitions satisfying Markov jumps, both controllers can effectively achieve the tracking control of heterogeneous MASs.

Quantized event-triggered-based finite-time $$\text {H}_\infty $$ control for interval type-2 fuzzy Markov jump systems with random coupling delays

Quantized event-triggered-based finite-time $$\text {H}_\infty $$ control for interval type-2 fuzzy Markov jump systems with random coupling delays

Asynchronous output feedback control for fuzzy Markov jump systems with actuator faults and deception attacks

Asynchronous output feedback control for fuzzy Markov jump systems with actuator faults and deception attacks

Fault-tolerant control for Markov jump nonlinear systems based on an observer and finite-time fast integral terminal sliding mode control

This paper develops a novel fault-tolerant control based on an observer for underactuated robot manipulators with unknown external disturbances, uncertainties, and actuator faults. First, unlike the traditional approach of treating the robot system as a parameter time-varying system, the underactuated robot manipulator with different drive modes can be considered as one of the classical Markov jump nonlinear systems (MJNSs). Second, an adaptive disturbance observer is designed to estimate the state and disturbances of the system. Finally, based on the observation results, a nonsingular fast integral terminal sliding mode controller (NFITSMC) is utilized to implement fault-tolerant control of the system. Compared with traditional observer and terminal sliding mode controller, the adoption of the novel controller and observer can improve response time and reduce chattering. Especially, in order to eliminate chattering, the integral term is introduced into the nonsingular fast terminal sliding mode controller. Structuring Lyapunov–Krasovskii functional (LKF) and based on linear matrix inequalities (LMIs) techniques, the convergence of control strategy and tracking errors is proved. The simulation results show that the actuator faults can be observed successfully and the error system is finite-time stable.

Integral sliding mode control and stability for Markov jump systems with structured perturbations and time-varying delay driven by fractional Brownian motion

The issues of stability and sliding mode control (SMC) for time-varying delay Markov jump systems (MJSs) with structured perturbations constrained by fractional Brownian motion (fBm) are explored. First, constructing a novel Lyapunov–Krasovskii functional (LKF) with exponential terms that contain the double-integral term, the pth moment exponential stability conditions are derived by utilizing the generalized fractional Itoˆ formula and conditional mathematical expectation. Subsequently, by designing the innovative integral sliding mode surface (SMS) associated with time-varying delay and the SMC law, the state trajectories of the dynamic systems can reach the designed SMS within a finite time. Ultimately, the numerical experiment is executed to confirm and ensure the accuracy and reliability of the obtained results.

An intermediate-observer-based robust [formula omitted] control of Markov jump systems with mismatched quantization

This paper focused on H2 anti-disturbance control of Markov jump systems with mismatched quantization. The system state is unknown and transition probabilities are allowed to be known, uncertain and completely unknown. Meanwhile, the mismatched quantization is presented by a polytopic uncertainty form. An intermediate observer is built to simultaneously estimate state and matched disturbances. A composite controller is built in terms of an observer-based state feedback, an estimated matched disturbance and a quantization error compensation. By utilizing the Lyapunov stability method, sufficient conditions for the closed-loop system to be stocastically stable are obtained in the form of linear matrix inequalities. The proposed scheme is validated by a comparative simulation example.

Adaptive neural network‐based security asynchronous control for uncertain Markov jump power systems with dead zone under DoS attack

AbstractThe article deals with the security control stabilization problem of uncertain Markov jump power systems with input dead zone under stochastic denial‐of‐service (DoS) attack. DoS attack is modeled as a discrete‐time Markov process. Dual hidden Markov models are respectively used to detect the modes of the original power systems and the one under DoS attack. Based on the detected modes and neural networks (NNs), adaptive NN‐based security asynchronous control strategies are proposed, where both state feedback and output feedback are studied simultaneously. With the developed control laws, all trajectories of the closed‐loop systems are bounded stable in the stochastic setting. Simulation results demonstrate the correctness and usefulness of the proposed techniques.

Adaptive neural quantized control for full-state constrained Markov jumping nonlinear systems with incomplete transition probabilities and unknown control directions

This article devises an adaptive quantized control scheme for full-state constrained Markov jumping nonlinear systems with incomplete transition probabilities and unknown control directions. First, the barrier Lyapunov functions were utilized to achieve full-state constraints on the investigated system, while the Nussbaum function has been adopted to overcome the issue of unknown control directions. Then, by means of command filtered backstepping control technique, an adaptive quantized controller is developed, where an improved error compensation mechanism was established to eliminate the effect of filter error, and a hysteresis quantizer was employed to diminish the transmission rate. Furthermore, it is demonstrated that the designed controller assures that all signals of the closed-loop system are bounded in the mean square sense. Finally, two illustrative examples were provided to validate the effectiveness of the proposed control method.

Prescribed-time fault-tolerant tracking control for Markov jump nonlinear systems based on saturation dynamics filters

This paper investigates the problem of prescribed-time fault-tolerant tracking control for a class of Markov jump nonlinear systems affected by actuator faults and input saturation. First, a new time-varying constraint function and an error transformation are invoked to transform the original prescribed-time tracking control problem into a tracking control problem with deferred constraint for tracking error. Furthermore, the parameter correction terms are designed, on which a novel Markov jump saturation dynamic filter is constructed to handle the unfavorable effect of input saturation. Meanwhile, the projection operator technique and the norm estimation methodology are integrated to devise adaptive laws to estimate the unknown actuator faults evolving in accordance with the Markov chain. A new prescribed-time fault-tolerant tracking control scheme is put forward that realizes the deferred constraint on the tracking error, which in turn ensures that the tracking error of the system converges to a prescribed accuracy within a prescribed-time. Finally, two simulation examples, one of which implements a prescribed-time fault-tolerant tracking control for rotary steerable drilling tool system, are presented to illustrate the effectiveness and practicality of the developed scheme.

A homotopy-based reinforcement learning scheme to optimal control for Markov switched interconnected systems

ABSTRACT In this paper, a homotopy-based reinforcement learning optimal control method is developed for Markov switched interconnected systems with unknown system dynamics. By utilising the subsystem decomposition method and parallel learning control method, the solution of the game coupled algebraic Riccati equations with jumping parameters is approximated. To dispense with the requirement of initial stability, a homotopy-based policy iteration is introduced, which can place unstable poles into a stable plane. In this regard, a model-free reinforcement learning method is presented to design the optimal controller for Markov switched interconnected systems. Finally, the effectiveness of the proposed method is verified by a numerical example and a practical example.

THE METHOD OF MARKING ITS CHAINS OF OCCURRENCES WITH A MORPHISM NUMDER IN THE N-SCHEME OF THE MARKOV ALGORITHM

To detect and classify λ-voids, we study a way of marking own chains of occurrences of words in each other by the morphism number determined in the N-scheme of the Markov algorithm. In the framework of the category-theoretic approach, the N-scheme of the Markov algorithm is considered as a topos form of the Markov algorithm. The topos explores additional properties of classifiers. The idea of introducing the Markov control flow algorithm into the N-schema is subjected to a comprehensive theoretical analysis. The control flows in the Turing algorithm and the topos form of the Markov algorithm are subjected to a comparative analysis. From the standpoint of the theoretical provisions of the constructivist approach of the Soviet (Leningrad) school, the conditions for equivalence and non-equivalence of the Markov and Turing algorithms are considered. The form of the Markov algorithm equivalent and non-equivalent to the Turing algorithm is synthesized, substantiated and analyzed. To detect the final occurrences in its chain of occurrences, necessary for the detection and classification of λ-voids, the composition of the elements of the N-scheme of the Markov algorithm, which is responsible for this, is specified. For some unknown algorithm for numbering its chains of occurrences of words in each other using the database of the N-scheme of the Markov algorithm, the method of its implementation by the imitating Turing algorithm is considered. The idea of Turing's algorithm imitating some of the algorithms used for the synthesis of artificial intelligence is analyzed. In the context of this study, it is shown that the imitation of the Turing algorithm is due to the need for a second pass through the data, organized by accessing the database in the N-scheme of the Markov algorithm. A description is given for the method of marking own chains of occurrences with the morphism number in the N-scheme of the Markov algorithm. It is concluded that it is necessary to decompose the instruction for checking whether the chain of occurrences belongs to its own chain into fragments, the length of which will be relatively small, for which we consider the method of marking the chains of occurrences with a word, as well as convolution.