3D Object Detection and Tracking Using Monocular Camera in CARLA

Abstract

Vehicle 3D extents and trajectories are crucial cues for many autonomous driving tasks such as path planning, motion prediction, etc [1]. This paper proposes a novel online deep learning framework to tackle the 3D object detection and track problem using a monocular camera. The framework can estimate the complete 3D information from a sequence of 2D images and associate the objects over time. By obtaining continuous frames from the front camera, our network robustly tracks the 3D bounding boxes for each observation and provides its location P with orientation θ, dimension D and 2D projection of its 3D center c. The training dataset is generated from the CARLA simulator and trained using Faster R-CNN [2] on a 2080 Super GPU. The 2D vehicles and centers test accuracy reaches 95% and 3D tracking performance can reach 81% MOTP [3] in CARLA [4] environment.