TITE: A transformer-based deep reinforcement learning approach for traffic engineering in hybrid SDN with dynamic traffic

Abstract

Hybrid Software Defined Networks (Hybrid SDNs), with a partial upgrade of legacy routers to SDN switches in traditional distributed networks, currently stand as a prevailing network architecture. Traffic Engineering (TE) in hybrid SDN requires the efficient and timely acquisition of a routing policy to adapt to dynamically changing traffic demands, which has recently become a hot topic. Ignoring the hidden relations of consecutive states and the compact representation of the network environment, previous Deep Reinforcement Learning (DRL)-based studies suffer from the convergence problem by only establishing the direct relationship between individual Traffic Matrix (TM) and routing policy. Therefore, to enhance TE performance in hybrid SDNs under dynamically changing traffic demands, we propose to integrate the Transformer model with DRL to establish the relationship between consecutive states and routing policies. The temporal characteristic among consecutive states can effectively assist DRL in solving the convergence problem. To obtain a compact and accurate description of the network environment, we propose to jointly consider TM, routing action, and reward in designing the state of the network environment. To better capture the temporal relations among consecutive states of the network environment, we design a multi-feature embedding module and achieve positional encodings in the Transformer model. The extensive experiments demonstrate that once the convergence problem is solved, the proposed Transformer-based DRL method can efficiently generate routing policies that adapt well to dynamic network traffic.