Context‐guided ground truth sampling for multi‐modality data augmentation in autonomous driving

Abstract

AbstractData augmentation is an important pre‐processing step for object detection in 2D image and 3D point clouds. However, studies on multimodal data augmentation are extremely limited compared to single‐modal work. Moreover, simultaneously ensuring consistency and rationality when pasting both image and point cloud samples is a major challenge in multimodal methods. In this study, a novel multimodal data augmentation method based on ground truth sampling (GT sampling) is proposed for generating content‐rich synthetic scenes. A GT database and scene ground database based on the raw training set is initially built, following which the context of the image and point cloud is used to guide the paste location and filtering strategy of the GT samples. The proposed method can avoid the cluttered features caused by random pasting of samples; the image context information can help the model to learn the correlation between the object and the environment more comprehensively, and the point cloud context information can reduce occlusion in the case of long‐distance objects. The effectiveness of the proposed strategy is demonstrated on the publicly available KITTI dataset. Utilizing the multimodal 3D detector MVXNet as an implementation tool, our experiments evaluate different superimposition strategies ranging from context‐free sample pasting methods to context‐guided new training scenes. In comparison with existing GT sampling methods, our method exhibits a relative performance improvement of 15% on benchmark datasets. In ablation studies, our sample pasting strategy achieves a +2.81% gain compared with previous work. In conclusion, considering the multimodal context of modelled objects is crucial for placing them in the correct environment.