MARL Sim2real Transfer: Merging Physical Reality With Digital Virtuality in Metaverse

Abstract

Metaverse is an artificial virtual world mapped from and interacting with the real world. In metaverse, digital entities coexist with their physical counterparts. Powered by deep learning, metaverse is inevitably becoming more intelligent in the interactions between reality and virtuality. However, it is confronted with a nontrivial problem known as sim2real transfer when deep learning techniques try to bridge the reality gap between the physical world and simulations. In this article, we use multiagent deep reinforcement learning (MARL) to implement collective intelligence for digital entities as well as their physical counterparts. To model the immersive environments in metaverse, we define a nonstationary variant of Markov games and propose a recurrent MARL solution to it. Based on the solution, MARL sim2real transfer that bridges real and virtual multiple unmanned aerial vehicle (multi-UAV) systems is successfully conducted by employing recurrent multiagent deep deterministic policy gradient (R-MADDPG) with the domain randomization technique. Additionally, we use perception-control modularization to improve the generalization performance of MARL policies and make training more efficient.