Reinforcement Learning-Based Particle Swarm Optimization for End-to-End Traffic Scheduling in TSN-5G Networks

Abstract

With the rapid development of the Industrial Internet of Things (IIoT), massive IIoT devices connect to industrial networks via wired and wireless. Furthermore, industrial networks pose new requirements on communications, such as strict latency boundaries, ultra-reliable transmission, and so on. To this end, time-sensitive networking (TSN) embedded fifth-generation (5G) wireless communication technology (i.e., TSN-5G networks), is considered the most promising solution to address these challenges. TSN can provide deterministic end-to-end latency and reliability for real-time applications in wired networks. 5G supports ultra-reliable and low-latency communications (uRLLC), providing increased flexibility and inherent mobility support in the wireless network. Thus, the integration of TSN and 5G provides numerous benefits, including increased flexibility, lower commissioning costs, and seamless interoperability of various devices, regardless of whether they use a wired or wireless interface. Nonetheless, the potential barriers between the TSN and 5G systems, such as clock synchronization and end-to-end traffic scheduling, are inevitable. Time synchronization has been studied in many works, so this paper focuses on the end-to-end traffic scheduling problem in TSN-5G networks. We propose a novel integrated TSN and 5G industrial network architecture, where the 5G system acts as a logical TSN-capable bridge. Based on this network architecture, we design a Double Q-learning based hierarchical particle swarm optimization algorithm (DQHPSO) to search for the optimal scheduling solution. The DQHPSO algorithm adopts a level-based population structure and introduces Double Q-learning to adjust the number of levels in the population, which evades the local optimum to further improve the search efficiency. Extensive simulations demonstrate that the DQHPSO algorithm can increase the scheduling success ratio of time-triggered flows compared to other algorithms.