A numerical approach to joint continuous and impulsive control of Markov chains

Abstract

Abstract This work considers the continuous and impulse control of the finite state Markov chains in continuous time. In general continuous control governs the transition rates between the states of Markov chain (MC), so the instants and directions of the state changes are random. Meanwhile sometimes there is an urgent necessity to realise the transition which leads to immediate change of the state. Since such transitions need different efforts and produce different effects on the function of the MC itself, one can consider this situation as an impulse control, and if at the same time there is a possibility of graduate control we are coming to the problem of joint continuous and impulse control. In the article we develop the martingale representation of the MC governed by joint continuous and impulse control and give an approach to the solution of the optimal control problem on the basis of the dynamic programming equation. This equation has a form of quasivariational inequality which in the case of finite state MC can be reduced to the solution of the system of ordinary differential equation with one switching line. We give the proof of the verification theorem and find the numerical solution for problems with deterministic and stochastic impulse controls.