A model for multi-time scaled sequential decision making processes

Abstract

We propose a simple analytical model, called M time-scale Markov Decision Process (MMDP), for hierarchically structured sequential decision making processes, where decisions in each level in the M-level hierarchy are made in M different time-scales. In this model, the state space and action space of each level in the hierarchy are non-overlapping with those of the other levels, respectively, and the hierarchy is structured in a pyramid sense such that a decision made at level m(slower time-scale) state and/or the state will affect the evolutionary decision making process of the lower level m+1 (faster time-scale) until a new decision is made at the higher level but the lower level decisions themselves do not affect the higher level transition dynamics. The performance produced by the lower level decisions will affect the higher level decisions. A hierarchical objective is defined such that the finite-horizon value of following a (nonstationary) policy at the level m+1 over a decision epoch of the level m plus an immediate reward at the level m is the single step reward for the level m decision making process. From this we define multi-level optimal value function and derive multi-level optimality equation. We then give some example control problems that can be modeled as MMDPs.