Comparing metrics for evaluating 3D map quality in natural environments

Abstract

In this study, we focus on addressing the challenge of measuring the 3D-map quality in natural environments. Specifically, we consider scenarios where the map is built using a robot’s 3D-Lidar point cloud observations, with potential uncertainty in the robot localization. When considering a natural environment, such as a park or a forest, unstructured by nature, another difficulty arises: the data becomes extremely sparse. As a result, measuring the map quality becomes even more challenging. This study aims to compare the effectiveness of various metrics in measuring the 3D-map quality. Firstly, we evaluate these metrics in a controlled experimental setup, where the reconstructed map is created by progressively degrading the reference map using different degradation models. Secondly, we compare their ability to measure 3D-map quality at a local level, across various simulated environments, ranging from structured to unstructured. Finally, we conduct a qualitative comparison to demonstrate the robustness of certain metrics to noise in the robot localization. This qualitative comparison is done both in simulation and in a real world experiment. Ultimately, we synthesize the properties of these metrics and provide practical recommendations for their selection.