Decentralized control of finite state Markov processes

Abstract

We are concerned with the control of a particular class of dynamic systems -- finite state Markov chains. The information pattern available is the so-called one step delay sharing information pattern. Using this information pattern, the dynamic programming algorithm can be explicitly carried out to obtain the optimal policy. The problems are discussed under three different cost criteria -- finite horizon problem with expected total cost, infinite horizon problem with discounted cost, and infinite horizon problem with average expected cost. The solution of the problem is possible with the one step delay sharing decentralized pattern because, as in the centralized control of Markov chains, a separation principle holds (this is not true for multiple step delay sharing). Hence the decentralized problem can essentially be reduced to a (more complicated) centralized one; this reduction is carried out in detail. With some modifications, the Policy Iteration Algorithm and Sondik's Algorithm are readily applied to find the optimal policies for these problems.