Adaptive Control Of Markov Chains Research Articles

Adaptive control of Markov chains with average cost

We present an adaptive control scheme for the control of Markov chains to minimize long-run average cost when the system transition and reward structures are unknown. Q-factors are estimated along a single sample path of the system and control actions are applied based on the latest estimates. We prove that an optimal policy is obtained asymptotically with probability one. More importantly, we prove that optimal system performance is achieved as well, which means that the performance of the system can not be bettered even if the system transition and reward structures are known. An example is given to illustrate our adaptive control scheme.

A reinforcement learning neural network for adaptive control of Markov chains

In this paper we consider the problem of reinforcement learning in a dynamically changing environment. In this context, we study the problem of adaptive control of finite-state Markov chains with a finite number of controls. The transition and payoff structures are unknown. The objective is to find an optimal policy which maximizes the expected total discounted payoff over the infinite horizon. A stochastic neural network model is suggested for the controller. The parameters of the neural net, which determine a random control strategy, are updated at each instant using a simple learning scheme. This learning scheme involves estimation of some relevant parameters using an adaptive critic. It is proved that the controller asymptotically chooses an optimal action in each state of the Markov chain with a high probability.

Minimizing the learning loss in adaptive control of Markov chains under the weak accessibility condition

We consider the adaptive control of Markov chains under the weak accessibility condition with a view to minimizing the learning loss. A certainty equivalence control with a forcing scheme is constructed. We use a stationary randomized control scheme for forcing and compute a maximum likelihood estimate of the unknown parameter from the resulting observations. We obtain an exponential upper bound on the rate of decay of the probability of error. This allows us to choose the rate of forcing appropriately, whereby we achieve a o(f(n) log n) learning loss for any function as .

Minimizing the learning loss in adaptive control of Markov chains under the weak accessibility condition

We consider the adaptive control of Markov chains under the weak accessibility condition with a view to minimizing the learning loss. A certainty equivalence control with a forcing scheme is constructed. We use a stationary randomized control scheme for forcing and compute a maximum likelihood estimate of the unknown parameter from the resulting observations. We obtain an exponential upper bound on the rate of decay of the probability of error. This allows us to choose the rate of forcing appropriately, whereby we achieve a o(f(n) log n) learning loss for any function as .

Markov Decision Problems and State-Action Frequencies

Consider a controlled Markov chain with countable state and action spaces. Basic quantities that determine the values of average cost functionals are identified. Under some regularity conditions, these turn out to be a collection of numbers, one for each pair, describing for each state the relative number of uses of each action. These frequencies, which are defined pathwise, are shown to determine the state-action that, in the finite case, are known to determine the costs. This is extended to the countable case, allowing for unbounded costs. The space of frequencies is shown to be compact and convex, and the extreme points are identified with stationary deterministic policies. Conditions under which the search for optimality in several optimization problems may be restricted to stationary policies are given. These problems include the standard Markov decision process, as well as constrained optimization (both in terms of average cost functionals) and variability-sensitive optimization. An application to a queueing problem is given, where these results imply the existence and explicit computation of optimal policies in constrained optimization problems. The pathwise definition of the conditional frequencies implies that their values can be controlled directly; moreover, they depend only on the limiting behavior of the control. This has immediate application to adaptive control of Markov chains, including adaptive control under constraints.

Adaptive control of Markov chains with local updates

An iterative algorithm is proposed for adaptive expected average cost control of finite state Markov chains. The algorithm estimates the unknown parameter using a strongly consistent parameter estimator, and uses this estimate to update the control policy. Each update may apply to as few as one or as many as all states as long as all states are updated infinitely often as the number of iterations goes to infinity; in this sense the algorithm does local updates. No prior knowledge of the optimal control policy is assumed. The algorithm requires only a small amount of computation at each step, is easily implementable, and converges to the optimal policy in finite time. Its complexity, for queueing systems governed by birth-death processes, is comparable to that of a learning automaton but is based on the optimality equation of dynamic programming.

An optimization-oriented approach to the adaptive control of Markov chains

We consider the control of a dynamic system modeled as a Markov chain. The transition probability matrix of the Markov chain depends on the control u and also on an unknown parameter α0. The unknown parameter belongs to a given finite set A . The long run average cost depends on the control policy and the unknown parameter. Thus, a direct approach to the optimization of the performance is not feasible. A common procedure calls for an on-line estimation of the unknown parameter and the minimization of the cost functional using the estimate in lieu of the true parameter. It is well known that this certainty equivalence (CE) solution may fail to achieve optimal performance, even asymptotically. In this presentation of a new optimization-oriented approach to adaptive control, we consider a composite functional which simultaneously takes care of the estimation and control needs. The global minimum of this composite functional coincides with the minimum of the original cost functional. Thus, its joint minimization with respect to control and parameter estimates would yield the optimal control policy. This joint minimization is not feasible, but it suggests an algorithm that asymptotically achieves the desired goal. The transient behavior of the algorithm, as well as the situation when \alpha^{0}\notin A are also investigated.

A Survey of Some Results in Stochastic Adaptive Control

Some results in discrete-time stochastic adaptive control are surveyed. The survey divides itself into two parts—Bayesian and non-Bayesian adaptive control. In the former area, the problems of converting an incompletely observed system into a completely observed one, multi-armed bandit processes, Bayesian adaptive control of Markov chains and Bayesian adaptive control of linear systems are exposed and surveyed. In the latter area, non-Bayesian adaptive control of Markov chains and the self-tuning regulator are dealt with. Proofs are given, where appropriate, to illustrate the methods involved.

Identification and Adaptive Control of Markov Chains

Consider a countable state controlled Markov chain whose transition probability is specified up to an unknown parameter $\alpha $ taking values in a compact metric space A. To each $\alpha $ is associated a prespecified stationary control law $\zeta (\alpha )$. The adaptive control law selects at each time t the control action $\zeta (\alpha _t ,x_t )$ where $x_t$ is the state and $\alpha_t$ is the maximum likelihood estimate of $\alpha $. The asymptotic behavior of this control scheme is investigated for the cases when the true parameter value $\alpha_0 $ does or does not belong to A, and for the case when $\zeta $ is chosen to minimize an average cost criterion. The analysis uses an appropriate extension of the notions of recurrence to nonstationary Markov chains.