Robust Control of Markov Decision Processes

Abstract

Listen

Translate article

Markov Decision problems are encountered frequently in real-life applications. The problems are often prone to uncertainties in environment dynamics. In many practical problems, ignorance of potential risk limits the performance of the optimal policy. This is a well explored field with lots of rich literature. We specifically focusing on uncertainty pertaining to two settings: uncertainty in reward/cost and uncertainty in transition dynamics. We conducted two numerical experiments to verify the efficacy of uncertainty set modelling. The first experiment is the classic aircraft path routing problem when the storm map in the environment follows an uncertain Markov Process. The other is the machine replacement problem, with uncertain replacement cost and uncertain transition matrix. Our experimental results showed the general improvement of the optimized performance when uncertainty sets are modelled with robust or chance-constraint formulations.