A Routing Optimization Method for Software-Defined SGIN Based on Deep Reinforcement Learning

Abstract

As space networks become more and more important, the Space-ground Integration network (SGIN) has received unprecedented attention. However, dynamic changes of topology and link status of satellite networks bring many challenges to routing optimization in the SGIN. Traditional routing optimization methods do not perform well, as they do not consider changes of topology and link status, as well as the association between flows. Since the Machine Learning (ML) technologies have shown significant advantages in dynamic routing optimization, we proposed a Machine Learning-based Space-ground Integration Networking (ML-SSGIN) framework that combines Software-Defined Networking (SDN) technologies to solve this challenge. To evaluate the feasibility of the proposed framework, the Deep Deterministic Policy Gradient (DDPG), a Deep Reinforcement Learning (DRL) algorithm, is deployed to perform routing optimization, which can make routing decisions based on real-time link status. In particular, we utilize a neural network that integrates Long Short-Term Memory Network (LSTM) and Dense layers for its actor and critic part to improve perceptual capabilities of contextual correlations between flows. We compared the proposed DDPG neural network with the one only having the Dense layers. The results show that the proposed architecture is feasible and effective. What's more, compared to Open Shortest Path First (OSPF) algorithm, our proposed routing optimization method can adapt to continuously change flows, and link status, which improves end-to-end throughput and latency.