D-S Augmentation: Density-Semantics Augmentation for 3-D Object Detection

Abstract

Cameras and light detection and ranging (LiDAR) sensors are commonly used together in autonomous vehicles to provide rich color and texture information and information on the location of objects, respectively. However, fusing image and point-cloud information remains a key challenge. In this article, we propose D-S augmentation as a 3-D object detection method based on point-cloud density and semantic augmentation. Our proposed approach first performs 2-D bounding box detection and instance segmentation on an image. Then, a LiDAR point cloud is projected onto an instance segmentation mask, and a fixed number of random points are generated. Finally, a global <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${N}$ </tex-math></inline-formula> -nearest neighbor clustering is used to associate random and projected points to give depth to virtual points and complete point-cloud density augmentation (P-DA). Then, point-cloud semantic augmentation (P-SA) is performed, in which the instance segmentation mask of an object is used to associate it with the point cloud. The instance-segmented class labels and segmentation scores are assigned to the projected cloud, and the projection points added with 1-D features are inversely mapped to the point-cloud space to obtain a semantically augmented point cloud. We conducted extensive experiments on the nuScenes (Caesar et al., 2020) and KITTI (Geiger et al., 2012) datasets. The results demonstrate the effectiveness and efficiency of our proposed method. Notably, D-S augmentation outperformed a LiDAR-only baseline detector by +7.9% in terms of mean average precision (mAP) and +5.1% in terms of nuScenes detection score (NDS) and outperformed the state-of-the-art multimodal fusion-based methods. We also present the results of ablation studies to show that the fusion module improved the performance of a baseline detector.