Improving End-To-End Latency Fairness Using a Reinforcement-Learning-Based Network Scheduler

Abstract

In services such as metaverse, which should provide a constant quality of service (QoS) regardless of the user’s physical location, the end-to-end (E2E) latency must be fairly distributed over any flow in the network. To this end, we propose a reinforcement learning (RL)-based scheduler for minimizing the maximum network E2E latency. The RL model used the double deep Q-network (DDQN) with the prioritized experience replay (PER). In order to see the performance change according to the type of RL agent, we implemented a single-agent environment where the controller is an agent and a multi-agent environment where each node is an agent. Since the agents were unable to identify E2E latencies in the multi-agent environment, the state and reward were formulated using the estimated E2E latencies. To precisely evaluate the RL-based scheduler, we set out benchmark algorithms to compare with which a network-arrival-time-based heuristic algorithm (NAT-HA) and a maximum-estimated-delay-based heuristic algorithm (MED-HA). The RL-based scheduler, first-in-first-out (FIFO), round-robin (RR), NAT-HA, and MED-HA were compared through large-scale simulations on four network topologies. The simulation results in fixed-packet generation scenarios showed that our proposal, the RL-based scheduler, achieved the minimization of maximum E2E latency in all the topologies. In other scenarios with random flow generation, the RL-based scheduler and MED-HA showed the lowest maximum E2E latency for all topologies. Depending on the topology, the maximum E2E latency of NAT-HA was equal to or larger than that of the RL-based scheduler. In terms of fairness, the RL-based scheduler showed a higher level of fairness than that of FIFO and RR. NAT-HA had similar or lower fairness than the RL-based scheduler depending on the topology, and MED-HA had the same level of fairness as the RL-based scheduler.