Control and inference of structured Markov models

Abstract

This Ph.D. thesis studies structured Markov models (SMMs) in the context of applied probability, stochastic modelling and applied statistics. SMMs are Markov chains on countable state spaces, whose transition rate matrices have particular structures, such as bidiagonal or tridiagonal block matrices. From a modelling point of view, applicability, robustness, simplicity, and tractability of SMMs make them a very important tool for studying complex systems. This is mainly due to the fact that for evaluation of performance measures of SMMs, there are efficient numerical analysis methods, commonly called matrix analytic methods (MAM). Some popular classes of SMMs are phase-type (PH) distributions, Markovian arrival processes (MAPs) and quasi-birth-and-death (QBD) processes. This thesis describes the outcomes of three research projects dealing with SMMs and their applications in stochastic modelling and applied statistics.In the first project, we introduce the notion of burstiness for a MAP. We call a stationary MAP bursty if both the squared coefficient of variation of inter-arrival times and the asymptotic index of dispersion of counts (IDC) are greater than unity. The simplest bursty MAP is a Hyperexponential renewal process with further classes, as we establish, being the Markov modulated Poisson process (MMPP), the Markov transition counting process (MTCP) and the Markov switched Poisson process (MSPP). Of these, MMPP has been used most often in applications for modelling bursty phenomena. Much of the popularity of MMPP stems from the intuition that it serves as a good model of bursty traffic. However, when MMPPs are viewed through the lens of the inter-arrival process, there is no proof to show that MMPPs are bursty. We provide analytical proofs to show that all of the MAPs mentioned above are bursty.Further, we investigate relations between these bursty MAPs. One of our main results is establishing a duality in terms of first and second moments of counts between MTCPs and a rich class of MMPPs which we refer to as slow-MMPPs (modulation is slower than the arrivals). Such a duality further confirms the applicability of MTCP as an alternative to MMPP. We augment our analytic results with numerical illustrations.In the second project, we consider a simple discrete-time controlled queueing system, where the controller has a choice of which server to use at each time slot, and server performance varies according to a Markov modulated random environment. We explore the role of information on the system stability region. At the extreme cases of information availability, that is when there is either full information or no information, stability regions and maximally stabilizing policies are trivial. But in the more realistic cases where only the environment state of the selected server is observed, only the service successes are observed or only the queue length is observed, finding throughput maximizing control laws is a challenge. To handle these situations, we devise a partially observable Markov decision process (POMDP) formulation of the problem and illustrate properties of its solution. We further model the system under given decision rules, using a QBD structure to find a matrix analytic expression for the stability bound. We use this formulation to illustrate how the stability region grows as the number of controller belief states increases.The third project focuses on the statistical methodology of semi-Markov processes, as motivated by the study of the trajectory of patients in intensive care units (ICUs) in hospitals. The main result of this project is the comparison of two approaches for defining and estimating semi-Markov models that are applied in ICUs. One approach is based on sojourn times, and the other approach is based on transition rates of the Markov jump process. We show that the second model has fewer parameters and its likelihood can be considered as the product of likelihoods of simpler two-state models. The comparison of these approaches helps to build models for predicting risks and chances of expected trajectories of patients through ICUs. Moreover, we extend the model to the case of having a multi-absorption PH distribution as the sojourn time distribution or intensity distribution of the semi-Markov model.The result of the projects mentioned above have been published or submitted to prestigious journals and peer-reviewed conferences, see [8–11, 146]. Further papers are currently in final stages of preparation.