Parallel Point Clouds: Hybrid Point Cloud Generation and 3D Model Enhancement via Virtual–Real Integration

Yonglin Tian,Zhongzheng Guo,Zilei Wang,Fei-Yue Wang,Yu Shen,Xiao Wang

doi:10.3390/rs13152868

Abstract

Three-dimensional information perception from point clouds is of vital importance for improving the ability of machines to understand the world, especially for autonomous driving and unmanned aerial vehicles. Data annotation for point clouds is one of the most challenging and costly tasks. In this paper, we propose a closed-loop and virtual–real interactive point cloud generation and model-upgrading framework called Parallel Point Clouds (PPCs). To our best knowledge, this is the first time that the training model has been changed from an open-loop to a closed-loop mechanism. The feedback from the evaluation results is used to update the training dataset, benefiting from the flexibility of artificial scenes. Under the framework, a point-based LiDAR simulation model is proposed, which greatly simplifies the scanning operation. Besides, a group-based placing method is put forward to integrate hybrid point clouds, via locating candidate positions for virtual objects in real scenes. Taking advantage of the CAD models and mobile LiDAR devices, two hybrid point cloud datasets, i.e., ShapeKITTI and MobilePointClouds, are built for 3D detection tasks. With almost zero labor cost on data annotation for newly added objects, the models (PointPillars) trained with ShapeKITTI and MobilePointClouds achieved 78.6% and 60.0% of the average precision of the model trained with real data on 3D detection, respectively.

Highlights

Published: 22 July 2021With the rapid development of deep learning, computer vision technology has achieved competitive results on many tasks compared to humans, such as image classification [1,2], object detection [3,4], medical imaging [5,6], and object tracking [7]
Taking the Light Detection and Ranging (LiDAR) used in autonomous driving as an example, due to the sparsity and incompleteness of the point clouds generated by LiDAR, annotators need to refer to images or other inputs to precisely label the whole point cloud
With only 40 cars in our MobilePointClouds dataset, we achieved 60.0% and 82.2% of the performance of the PointPillars model trained with real KITTI for 3D and BEV precision, respectively, under moderate mode

Summary

Introduction

Published: 22 July 2021With the rapid development of deep learning, computer vision technology has achieved competitive results on many tasks compared to humans, such as image classification [1,2], object detection [3,4], medical imaging [5,6], and object tracking [7]. Challenges still exist, especially in 3D perception tasks [8], due to the increase of the data dimension. The annotation of point clouds generated by active sensors such as Light Detection and Ranging (LiDAR) is time consuming. Taking the LiDAR used in autonomous driving as an example, due to the sparsity and incompleteness of the point clouds generated by LiDAR, annotators need to refer to images or other inputs to precisely label the whole point cloud. This greatly increases the difficulty and time for point cloud annotation

Methods

Findings

Discussion

Conclusion