Combining Monocular Camera and 2D Lidar for Target Tracking Using Deep Convolution Neural Network based Detection and Tracking Algorithm

Abstract

Moving target detection and tracking is one of the key technologies for research of intelligent mobile robot and image recognition. The commonly used sensors include cameras, lidar and infrared sensor. Although lidar is able to locate target precisely thanks to its range and angular information in clear environment, it is difficult to detect target in cluttered environment. On the other hand, with the development of detection models based on deep learning, the target detection can be well solved using images from monocular camera, however, only using monocular camera is hard to determine the depth information of the target object, it can not accurately obtain the position of the target object. Therefore, in order to take advantage of pros of both monocular camera and 2d lidar, this paper proposes a target tracking algorithm for a mobile robot by combining a monocular camera and 2D lidar. The CSI camera is used to detect and track the target by deep convolution neural network models YOLOv5 and Deep SORT, respectively, while the 2D lidar is used to determine the angle and distance of the target corresponding to the one detected and tracked using images from camera. Extensive experiments have been conducted on a mobile robot, showing the effectiveness of the proposed target tracking algorithm.