Markovian Control Problems Research Articles

Gradient-based optimization techniques for the design of static controllers for Markov jump linear systems with unobservable modes

The paper formulates the static control problem of Markov jump linear systems, assuming that the controller does not have access to the jump variable. We derive the expression of the gradient for the cost motivated by the evaluation of 10 gradient-based optimization techniques. The numerical efficiency of these techniques is verified by using the data obtained from practical experiments. The corresponding solution is used to design a scheme to control the velocity of a real-time DC motor device subject to abrupt power failures. Copyright © 2014 John Wiley & Sons, Ltd.

The Expected Total Cost Criterion for Markov Decision Processes under Constraints: A Convex Analytic Approach

This paper deals with discrete-time Markov decision processes (MDPs) under constraints where all the objectives have the same form of expected total cost over the infinite time horizon. The existence of an optimal control policy is discussed by using the convex analytic approach. We work under the assumptions that the state and action spaces are general Borel spaces, and that the model is nonnegative, semicontinuous, and there exists an admissible solution with finite cost for the associated linear program. It is worth noting that, in contrast to the classical results in the literature, our hypotheses do not require the MDP to be transient or absorbing. Our first result ensures the existence of an optimal solution to the linear program given by an occupation measure of the process generated by a randomized stationary policy. Moreover, it is shown that this randomized stationary policy provides an optimal solution to this Markov control problem. As a consequence, these results imply that the set of randomized stationary policies is a sufficient set for this optimal control problem. Finally, our last main result states that all optimal solutions of the linear program coincide on a special set with an optimal occupation measure generated by a randomized stationary policy. Several examples are presented to illustrate some theoretical issues and the possible applications of the results developed in the paper.

The Expected Total Cost Criterion for Markov Decision Processes under Constraints: A Convex Analytic Approach

This paper deals with discrete-time Markov decision processes (MDPs) under constraints where all the objectives have the same form of expected total cost over the infinite time horizon. The existence of an optimal control policy is discussed by using the convex analytic approach. We work under the assumptions that the state and action spaces are general Borel spaces, and that the model is nonnegative, semicontinuous, and there exists an admissible solution with finite cost for the associated linear program. It is worth noting that, in contrast to the classical results in the literature, our hypotheses do not require the MDP to be transient or absorbing. Our first result ensures the existence of an optimal solution to the linear program given by an occupation measure of the process generated by a randomized stationary policy. Moreover, it is shown that this randomized stationary policy provides an optimal solution to this Markov control problem. As a consequence, these results imply that the set of randomized stationary policies is a sufficient set for this optimal control problem. Finally, our last main result states that all optimal solutions of the linear program coincide on a special set with an optimal occupation measure generated by a randomized stationary policy. Several examples are presented to illustrate some theoretical issues and the possible applications of the results developed in the paper.

Second moment constraints and the control problem of Markov jump linear systems

SUMMARYThis paper addresses the optimal solution for the regulator control problem of Markov jump linear systems subject to second moment constraints. We can characterize and obtain the solution explicitly using linear matrix inequalities techniques. The constraints are imposed on the second moment of both the system state and control vector, and the optimal solution is obtained in a computable form. To illustrate the usefulness of the approach, specially that for systems subject to abrupt variations and physical limitations, we present an application for one joint of the European Robotic Arm. Copyright © 2012 John Wiley & Sons, Ltd.

Observer-based finite-time control of time-delayed jump systems

This paper provides the observer-based finite-time control problem of time-delayed Markov jump systems that possess randomly jumping parameters. The transition of the jumping parameters is governed by a finite-state Markov process. The observer-based finite-time H ∞ controller via state feedback is proposed to guarantee the stochastic finite-time boundedness and stochastic finite-time stabilization of the resulting closed-loop system for all admissible disturbances and unknown time-delays. Based on stochastic finite-time stability analysis, sufficient conditions that ensure stochastic robust control performance of time-delay jump systems are derived. The control criterion is formulated in the form of linear matrix inequalities and the designed finite-time stabilization controller is described as an optimization one. The presented results are extended to time-varying delayed MJSs. Simulation results illustrate the effectiveness of the developed approaches.

Generalized Coupled Algebraic Riccati Equations for Discrete-time Markov Jump with Multiplicative Noise Systems

In this paper we consider the existence of the maximal and mean square stabilizing solutions for a set of generalized coupled algebraic Riccati equations (GCARE for short) associated to the infinite-horizon stochastic optimal control problem of discrete-time Markov jump with multiplicative noise linear systems. The weighting matrices of the state and control for the quadratic part are allowed to be indefinite. We present a sufficient condition, based only on some positive semidefinite and kernel restrictions on some matrices, under which there exists the maximal solution and a necessary and sufficient condition under which there exists the mean square stabilizing solution for the GCARE. We also present a solution for the discounted and long run average cost problems when the performance criterion is assumed to be composed by a linear combination of an indefinite quadratic part and a linear part in the state and control variables. The paper is concluded with a numerical example for pension fund with regime switching.

Generalized Coupled Algebraic Riccati Equations for Discrete-Time Markov Jump with Multiplicative Noise Systems

In this paper we consider the existence of the maximal and mean square stabilizing solutions for a set of generalized coupled algebraic Riccati equations (GCARE for short) associated to the infinite-horizon stochastic quadratic optimal control problem of discrete-time Markov jump with multiplicative noise linear systems. The weighting matrices of the state and control for the quadratic part are allowed to be indefinite. We present a sufficient condition under which there exists the maximal solution and a necessary and sufficient condition under which there exists the mean square stabilizing solution for the GCARE.

Indefinite quadratic with linear costs optimal control of Markov jump with multiplicative noise systems

In this paper we consider the stochastic optimal control problem of discrete-time Markov jump with multiplicative noise linear systems. The performance criterion is assumed to be formed by a linear combination of a quadratic part and a linear part in the state and control variables. The weighting matrices of the state and control for the quadratic part are allowed to be indefinite. We present a necessary and sufficient condition under which the problem is well posed and a state feedback solution can be derived from a set of coupled generalized Riccati difference equations interconnected with a set of coupled linear recursive equations. For the case in which the quadratic-term matrices are non-negative, this necessary and sufficient condition can be written in a more explicit way. The results are applied to a problem of portfolio optimization.

CONTROL OF MARKOV JUMP LINEAR SYSTEMS WITH STATE AND INPUT CONSTRAINTS: A NECESSARY OPTIMALITY CONDITION

This paper presents necessary optimality conditions for the control problem of discrete-time Markov jump linear systems (M JLS) with linear feedback gains, subject to constraints on the system state and input control variables. Instead of hard, deterministic constraints, the constraints considered here are given in terms of the first and second moments of the state and input variables, thus adequate to in the stochastic setting. The jumping state of the Markov chain and the continuous part of the MJLS are assumed to be accessible to the controller. The necessary conditions are derived and they are attained numerically by means of a method that iteratively performs a variational search using a set of LMI's. It searches through monotonically decreasing costs and converges to a sequence of feedback gains that satisfies the optimality conditions. The usefulness of the method is illustrated by a numerical example.

A separation principle for the [formula omitted]-control of continuous-time infinite Markov jump linear systems with partial observations

In this paper we devise a separation principle for the H 2 optimal control problem of continuous-time Markov jump linear systems with partial observations and the Markov process taking values in an infinite countable set S . We consider that only an output and the jump parameters are available to the controller. It is desired to design a dynamic Markov jump controller such that the closed loop system is stochastically stable and minimizes the H 2 -norm of the system. As in the case with no jumps, we show that an optimal controller can be obtained from two sets of infinite coupled algebraic Riccati equations, one associated with the optimal control problem when the state variable is available, and the other one associated with the optimal filtering problem. An important feature of our approach, not previously found in the literature, is to introduce an adjoint operator of the continuous-time Markov jump linear system to derive our results.

Portfolio optimization under transaction costs in the CRR model

In the CRR model we introduce a transaction cost structure which covers piecewise proportional, fixed and constant costs. For a general utility function we formulate the problem of maximizing the expected utility of terminal wealth as a Markov control problem. An existence result is given and optimal strategies can be described by solutions of the dynamic programming equation. For logarithmic utility we provide detailed solutions in the one-period case and provide examples for the multi-dimensional case and for complex cost structures. For a combination of fixed and proportional costs a fast multi-period algorithm is introduced.

The Scalarization Approach to Multiobjective Markov Control Problems: Why Does It Work?

This paper concerns discrete-time multiobjective Markov control processes on Borel spaces and unbounded costs. Under mild assumptions, it is shown that the usual “scalarization approach” to obtain Pareto policies for the multiobjective control problem is in fact equivalent to solving the dual of a certain multiobjective infinite-dimensional linear program. The latter program is obtained from a multiobjective measure problem which is also used to prove the existence of strong Pareto policies, that is, Pareto policies whose cost vector is the closest to the control problem’s virtual minimum.

Weak detectability and the linear-quadratic control problem of discrete-time Markov jump linear systems

The paper deals with the stochastic concepts of weak detectability and weak observability for Markov jump linear systems, which is an special class of composite linear systems. The concepts are explored here to strengthen the similarities with the corresponding concepts of deterministic detectability and observability. We introduce a collection of matrices, referred to as the observability matrices. We show that weak observability is equivalent to full rank of each matrix in the set of observability matrices. In addition, we present a stochastic counterpart of the well known result on the invariance of trajectories within non-observable subspaces. These characterizations allow us to clarify the relationship between weak detectability and mean square detectability and to provide a testable condition for weak detectability. Relying on the assumption of weak detectability, we develop a method for solving the linear quadratic problem that is based on iterations of uncoupled algebraic Riccati equations, which converges to the solution of the coupled algebraic Riccati equation if and only if the system is mean-square stabilizable. Numerical examples are included.

The H2-control for jump linear systems: cluster observations of the Markov state

The H 2-norm control problem of discrete-time Markov jump linear systems is addressed in this paper when part of, or the total of the Markov states is not accessible to the controller. The non-observed part of the Markov states is grouped in a number of clusters of observations; the case with a single cluster retrieves the situation when no Markov state is observed. The control action is provided in linear feedback form, which is invariant on each cluster, and this restricted complexity setting is adopted, aiming at computable solutions. We explore a recent result by de Oliveira, Bernussou, and Geromel (Systems Control Lett. 37 (1999) 261) involving an LMI characterization to establish a H 2 solution that is stabilizing in the mean square sense. The novelty of the method is that it can handle in LMI form the situation ranging from no Markov state observation to complete state observation. In addition, when the state observation is complete, the optimal H 2-norm solution is retrieved.

Markov control problems under communication constraints

In traditional control systems theory one assumes that the controller receives the observations from the sensors instantaneously and exactly, and the control action it initiates is immediately effective. In reality, however, the controller and the sensors may be geographically separated from the plant and the observations may be transmitted from the sensors to the controller across a communication channel. In addition to adding noise, the channel can also put a constraint on the rate of information transmission and introduce delays. In a similar way the path from the controller back to the plant can be modeled as a communication channel with noise and delays Hence the need for a control theory that explicitly takes into account these aspects. The aim of the present work is to view the problem in its totality and come up with a set of paradigms that can form a basis for analysis and synthesis of such control systems

The H2-Control For Markovian Jump Linear Systems with Cluster Observations of the Markov State

The H2-norm control problem of discrete-time Markov jump linear systems is addressed in this paper when part of, or the total of the Markov states are not accessible to the controller. We adopt a restricted complexity setting in which the control action is in linear feedback form, aiming at computable solutions. We explore a recent result involving a LMI characterization to establish a solution for this class of incomplete observation problems. When the state observation is complete, we retrieve the optimal H2-norm solution. The setting we adopt has a natural appeal to the robust analysis, and some interpretations and a related problem is exposed.

Optimal control of a queueing system with heterogeneous servers and setup costs

This paper considers a queueing model with batch Poisson input and two heterogeneous servers, where the service times are exponentially distributed. The faster server is always on, but the slower server is only used when the queue length exceeds a certain level. Activating the slower server involves fixed set-up costs. Also there are linear operating costs and linear holding costs. The class of two-level hysteretic control rules is considered. Rather than proving the overall average cost optimality of a hysteretic rule, the purpose of this paper is to develop a tailor-made policy iteration algorithm for computing the optimal switch-on and switch-off levels for the slower server. An embedding method is used that is generally applicable to structured Markovian control problems with an infinitely large state space.

Uncoupled Riccati iterations for the linear quadratic control problem of discrete-time Markov jump linear systems

This paper deals with recursive methods for solving coupled Riccati equations arising in the linear quadratic control for Markovian jump linear systems. Two algorithms, based on solving uncoupled Riccati equations at each iteration, are presented. The standard method for this problem relies on finite stage approximations with receding horizon, whereas the methods presented here are based on sequences of stopping times to define the terminal time of the approximating control problems. The methods can be ordered in terms of rate of convergence. Comparisons with other methods in the current literature are also presented.

Average Optimality in Markov Control Processes via Discounted-Cost Problems and Linear Programming

This paper shows the existence of solutions to the average-cost problem for Markov control processes on Borel spaces, with possibly unbounded costs and noncompact control constraint sets. This is done via a combination of the well-known vanishing discount approach and recent results on the linear programming formulation for both discounted- and average-cost Markov control problems.

Survey of linear programming for standard and nonstandard Markovian control problems. Part II: Applications

This paper deals with some applications of Markov decision models for which the linear programming method is efficient. These models are replacement models (with the optimal stopping problem as special case), separable models (including the inventory model as special case) and the multi-armed bandit model. In the companion paper “Survey of linear programming for standard and nonstandard Markovian control problems. Part I: Theory”, general linear programming methods are discussed. These linear programming formulations are the starting point for the efficient methods that will be derived for the special models.