Congestion control using policy rollout

Abstract

We consider the congestion-control problem in a communication network with multiple traffic sources, each modelled as a fully-controllable stream of fluid traffic and associated with a unique round-trip delay. The bandwidth available to the controlled sources is stochastic due to high-priority cross traffic, described by a Markov-modulated fluid. The goal is to maximize a linear combination of the throughput, delay, and traffic loss at the bottleneck node, while achieving fairness among controlled sources. The control problem is posed as a Markov decision process (MDP). We heuristically solve the MDP via a technique called policy rollout. Our empirical study demonstrates that the control scheme performs significantly better than conventional congestion controllers. We further find that employing different estimates of the Q-value in solving the MDP leads to comparable overall cumulative rewards, although the component contributions can be quite different.