QDRL: QoS-Aware Deep Reinforcement Learning Approach for Tor's Circuit Scheduling

Abstract

Tor is a popular anonymity network adopted by more than two million users to preserve their privacy. Tor was mainly developed as a low-latency network to support interactive web browsing and messaging applications. However, bandwidth acquisitive applications such as BitTorrent consume a considerable percentage of Tor traffic. This results in an unfair allocation of the available bandwidth and significant degradation in the Quality-of-service (QoS) delivered to users. This paper presents a QoS-aware deep reinforcement learning approach for Tor's circuit scheduling (QDRL). We propose a design that coalesces the two scheduling levels originally presented in Tor and addresses it as a single resource allocation problem. We use the QoS requirements of different applications to set the weight of active circuits passing through a relay. Furthermore, we propose a set of approaches to achieve the optimal trade-off between system fairness and efficiency. We designed and implemented a reinforcement-learning-based scheduling approach ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TRLS</i> ), a convex-optimization-based scheduling approach ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CVX-OPT</i> ), and an average-rate-based proportionally fair heuristic ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AR-PF</i> ). We also compare the proposed approaches with basic heuristics and with the implemented scheduler in Tor. We show that our reinforcement-learning-based approach (TRLS) achieved the highest QoS-aware fairness level with a resilient performance to the changes in an environment with a dynamic nature, such as the Tor network.