HA-D3QN: Embedding virtual private cloud in cloud data centers with heuristic assisted deep reinforcement learning

Abstract

The virtual network embedding problem is embedding virtual networks (VNs) in a substrate network so that revenue or accept ratio is maximized. Previous study usually assumes disclosed communication demand among the virtual nodes in a VN, mismatching real-world cloud computing scenarios. In this paper, we propose a new VN abstraction based on the widely used Virtual Private Cloud model, where internal communication demand is unknown to cloud providers. In contrast with the majority of existing research, we allow the co-location of the virtual nodes belonging to the same VN, and introduce the concept of switching capacity for practical resource reservation. We categorize the substrate resources in cloud data centers into additive and non-additive for the first time, and devise our algorithms accordingly. After formulating the problem, we propose a solution framework named HA-D3QN (Heuristic Assisted Dueling Double Deep Q Network). Essentially, HA-D3QN selects the best responses to different system states by combining the D3QN deep reinforcement learning structure and the candidate actions, which are generated by our proposed heuristic algorithms for addressing the exponentially large action space. Finally, we conduct extensive simulation experiments, the results of which verify the effectiveness of our approach.