A synthetic dataset for Visual SLAM evaluation

Abstract

Vision-based self-localization methods are key functionalities for various research topics. Recent research results on related fields have catalyzed several accurate, versatile and reliable real-time Visual SLAM systems suitable for self-localization under a wide variety of environmental preconditions. These methods extend their functionalities from being only a good camera tracker to being able to recursively build up camera’s surroundings. The fast development of Visual SLAM research has proposed demands on innovating evaluation methods for Visual SLAM systems. However, retrieving images and ground truth from various kinds of environments, estimating calibration parameters between several sensors and annotating useful labels all require cumbersome human labor and will introduce inevitable errors. In this paper, we propose a method that uses virtually established models to automatically generate photorealistic images with accurate ground truth and several kinds of pixel-level annotations useful for Visual SLAM development and evaluation. We build and render a challenging dataset in low-texture environments with large scale camera movement, multiple moving objects and varying luminance status. We also propose several new evaluation criteria that can fully take advantage of ground truth and annotations from synthetic datasets. Experiments are conducted using the proposed datasets and criteria with several state-of-the-art Visual SLAM methods to demonstrate the functionality of our datasets.