MmWave radar tracking and sensor fusion with camera

Abstract

W-band millimeter wave (mmWave) radar is becoming an essential sensor in advanced driver assistance system (ADAS) and autonomous driving fields [1-5], including adaptive cruise control, pedestrian detection, collision avoidance, lane changing monitoring, and emergency braking. ADAS systems comprise of multiple sensors, such as radar, camera, and LiDAR. Each type of sensor offers its advantage, e.g., radar has a reliably moving target detection and is robust to adverse weather conditions. But radar has a relatively poor angular resolution comparing with its optical counterparts, which can achieve 0.1-1 degree resolution in both azimuth and elevation directions. The shortcoming of radar will result in uncertainty, inaccuracy, and missing or ghost detections at certain distances. One way to reduce the uncertainties and failures is to fuse radar outputs with heterogeneous sensors [4, 6].mmWave radar research has gained immense popularity with the recent increasing demand of automotive radars in ADAS and autonomous driving industry. Among all the researches on mmWave radar applications, tracking [7-9], target recognition [10-12], and sensor fusion [13-15] are the most popular topics. In this chapter, we will start with a reliable radar tracking method using extended Kalman filter (EKF) and discuss the nonlinearities of radar tracker. Then, we will further develop a radar-camera sensor fusion by constructing a fusion-extended Kalman filter (fusion-EKF) in the remaining of the chapter.Sensor fusion has been developing rapidly in recent years. However, there are limited studies discussing fusion of mmWave radars with other sensors since radars provide a limited number of detection points representing targets-of-interest [4], which make it difficult to recognize from a snapshot of radar detection. But if fusion can be achieved before the target classification of radar, a fusion system may fully take the advantage of radar, i.e., increasing the reliability of detecting moving targets, avoiding blockage, and tracking dramatically. In this chapter, we aim to increase mmWave radar's informative capability about targets and further its versatility by fusion with monocameras as an example.Specifically, we introduce a fusion-EKF, which is designed to fuse data from heterogeneous sensors such as mmWave radar and monocamera with real-time fusion algorithm running for tracking. Sensor fusion and association are done within the fusion-EKF using a homography estimation method (HEM) [16], timeline alignment, and region search. Reliable detection and cross-validated target tracking are also realized. As we do not use any machine learning-based approaches to realize the fusion, the introduced fusion method achieves a low computational complexity, which is ideal for implementing in real-time systems. The experimental results also show that the proposed system can provide a reliable tracking and detecting result with low calculation costs. An embedded system like Arduino or Raspberry Pi can be utilized to process the data for real-time applications.For the new fusion-EKF, a new concept is introduced, i.e., error bounds (EBs), which is defined as the sensor's region of approximation. EBs are not from the uncertainty of sensors [17] but the sensors' resolutions from their respective perspectives. An HEM is applied in associating heterogeneous sensors via their EBs. The fusion-EKF is designed to take both radar and camera as inputs and associate the data inside the filter to obtain ideal target tracking outputs. Data association of the fusion-EKF is capable to support tracking of multiple targets.The structure of this chapter is as follows. In section 5.2, related work on mmWave radar tracking and sensor fusion are studied and presented. In section 5.3, radar-EKF tracking methodology is shown. In section 5.4, the sensor fusion fusion-EKF is presented with preprocessing, data association and sensor synchronization. In section 5.5, results of radar-camera fusion-EKF and root mean square errors (RMSEs) of improved EBs are shown.