Analysis of Depth and Semantic Mask for Perceiving a Physical Environment Using Virtual Samples Generated by a GAN

Abstract

Micro aerial vehicles (MAVs) can make explorations in 3D environments using technologies capable of perceiving the environment to map and estimate the location of objects that could cause collisions, such as Simultaneous Localization and Mapping (SLAM). Nevertheless, the agent needs to move during the environment mapping, reducing the flying time to employ additional activities. It has to be noted that adding more devices (sensors) to MAVs implies more power consumption. Since more energy to perform tasks is required, growing the dimensions of MAVs limits the flying time. Contrarily, Generative Adversarial Networks (GAN) have demonstrated the usefulness of creating images from one domain to another, but the GAN domain changes require a large number of samples. Therefore, an interoperability coefficient is employed to determine a minimum number of samples to connect the different domains. In order to prove the coefficient, the performance to estimate the depth and semantic mask between authentic and virtual samples with the number limited of samples is analyzed. Consequently, an RGB-D sensor can be replaced by a few samples of a real scenario based on GANs. Although GAN allows creating images with depth and semantic mask information, there is an additional problem to be tackled: the presence of intrinsic noise, where a simple GAN architecture is not enough. In this proposal, the performance of this solution against a physical RGB-D sensor (Microsoft Kinect V1) and other state-of-the-art approaches is compared. Experimental results allow us to affirm that this proposal is a viable option to replace a physical RGB-D sensor with limited information.