Optimal Stopping of Seasonal Observations and Projection of a Markov Chain

Abstract

We consider the recently solved problem of Optimal Stopping of Seasonal Observations and its more general version. Informally, there is a finite number of dice, each for a state of “underlying” finite MC. If this MC is in a state k, then k-th die is tossed. A Decision Maker (DM) observes both MC and the value of a die, and at each moment of discrete time can either continue observations or to stop and obtain a discounted reward. The goal of a DM is to maximize the total expected discounted reward. This problem belongs to an important class of stochastic optimization problems—the problem of optimal stopping of Markov chains (MCs). The solution was obtained via an algorithm which is based on the general, so called, State Elimination algorithm developed by the author earlier. An important role in the solution is played by the relationship between the fundamental matrix of a transient MC in the “large” state space and the fundamental matrix for the modified underlying transient MC. In this paper such relationship is presented in a transparent way using the general concept of a projection of a Markov model. The general relationship between two fundamental matrices is obtained and used to clarify the solution of the optimal stopping problem.