What should be the input: Investigating the environment representations in sim-to-real transfer for navigation tasks

Abstract

While training an end-to-end navigation network in the real world is usually costly, simulation serves as a safe and low-cost tool in this training process. However, training neural network models in simulation brings up the problem of effectively transferring the model from simulation to the real world (sim-to-real). In this work, we regard the environment representation as a crucial element in this transfer process, and we propose a visual information pyramid (VIP) model to investigate a practical environment representation systematically. A novel representation composed of spatial and semantic information synthesis is established accordingly, where noise model embedding is particularly considered. To explore the effectiveness of the proposed representation, we compared its performance with other popularly used representations in the literature, such as RGB image, depth image, and semantic segmentation image, in both simulated and real-world scenarios. Results suggest that our environment representation stands out. Furthermore, an analysis on the feature map is implemented to investigate the effectiveness through hidden layer reaction, which could be irradiative for future researches on sim-to-real learning-based navigation.