Finite Generalised Markov Programming

Abstract

An example of a finite generalised Markov programming problem is the following pro? duction control situation: how should the rate of production of a machine be controlled by choosing successively from a finite number of possible rates to minimise average expected running cost when demand per unit time is (independent) Poisson and there are given production cost (rate dependent), stock holding and stockout costs and costs associated with changing production rates? Stated thus, the subject is quite approachable to the general O.R. practitioner. However, as with most of the Mathematical Centre Tracts, this book is very much a work of mathematics. It is concerned with the computa? tional intricacies of a class of stochastic control problems. Specifically this class consists of systems where there is an underlying process with the structure of a finite state Markov renewal process which can be controlled by choosing interventions (made only at transition times of the renewal process) from a finite set. An intervention induces a reward or cost and an instantaneous change of state of the process. An intervention policy is chosen to maximise average rewards or minimise average costs. The assump? tions of finiteness of the state space and the intervention set specialise earlier work in this field with the aim of arriving at simplified numerical solution procedures. The book starts with a good introduction to the notational and mathematical preliminaries followed by a full description of the problems to be studied. It goes on to show that there exist finite convergence schemes to solve these problems by now standard policy iteration methods. Next there is detailed work on variations of the basic schemes using ideas of optimal stopping. This is followed by numerical comparisons of perform? ance of these schemes. The book finishes with a chapter on problems with a discounting structure and one on sensitivity questions. The text is aimed at the academic probabilist and as such the book is a good one. However, although the potential for practical applications might be wide, little provision is made for considering examples. Someone trying to solve a practical problem would need a great deal of mathematical maturity and a lot of patience to arrive at a scheme for setting up numerical procedures from the material of this book. In this respect, I think the book would have greatly beriefited from a larger discussion of where the how this theory could be applied.