Radar Sensor Fusion Research Articles

Robust Gait Recognition Based on Deep CNNs With Camera and Radar Sensor Fusion

In recent years, gait recognition has emerged as an important and promising solution for human identification. Generally, gait recognition is based on a single type of sensor, such as a camera or a radar. However, data of a single modality may only capture inadequate gait features of a person, such as camera data lacking the intuitive micro-motion pattern information and radar data lacking the information about gait appearance, making gait-based human identification system vulnerable to complex covariate conditions, e.g. cross-view and cross-walking-condition. To build a robust and reliable gait-based human identification system, in this study, we propose a multi-sensor gait recognition framework with deep convolutional neural networks (CNNs) by fusing camera gait energy images (GEIs) and radar time-Doppler spectrograms. To learn the fine-grained gait appearance features, we propose a body-part spatial attention (BPSA) module to obtain more discriminative body part representations of GEIs. To learn the gait micro-motion pattern, we propose a long-short temporal relation modeling (LSTRM) module to obtain the local and global micro-motion representation of time-Doppler spectrograms. Finally, we fuse the discriminative body part representation and the micro-motion pattern at the multi-scale feature space to obtain richer and more robust gait features for human identification. We provide an extensive empirical evaluation in terms of various complex covariate conditions, namely, cross-view and cross-walking-condition. Experiments on 121 subjects with eight views and three walking conditions of camera and radar data show our proposed method is more robust and accurate.

Camera, LiDAR, and Radar Sensor Fusion Based on Bayesian Neural Network (CLR-BNN)

<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Perception</i> in automated vehicles (AV) is the main factor in achieving safe driving. In this perception task, multi-object detection (MOD) in diverse driving situations is the main challenge. Our recent survey [Ravindran <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> (2021)] shows the limitations of deep neural networks (DNN) in predicting the uncertainties of object detection in MOD. This research proposed a camera, LiDAR and RADAR sensor fusion Bayesian neural network (CLR-BNN) to improve detection accuracy and reduce uncertainties in diverse driving situations using these three primary sensing devices. The experiments were performed using the nuScence dataset with incorporation of various noises. The CLR-BNN performed better than its deterministic sensor fusion model (CLR-DNN) in terms of mAP. The CLR-BNN also showed improvement in categorical and bounding box location uncertainty using sensor fusion in diverse driving conditions. The uncertainty predictions of the CLR-BNN were validated using the calibration curve and other performance metrics.

Radar Voxel Fusion for 3D Object Detection

Automotive traffic scenes are complex due to the variety of possible scenarios, objects, and weather conditions that need to be handled. In contrast to more constrained environments, such as automated underground trains, automotive perception systems cannot be tailored to a narrow field of specific tasks but must handle an ever-changing environment with unforeseen events. As currently no single sensor is able to reliably perceive all relevant activity in the surroundings, sensor data fusion is applied to perceive as much information as possible. Data fusion of different sensors and sensor modalities on a low abstraction level enables the compensation of sensor weaknesses and misdetections among the sensors before the information-rich sensor data are compressed and thereby information is lost after a sensor-individual object detection. This paper develops a low-level sensor fusion network for 3D object detection, which fuses lidar, camera, and radar data. The fusion network is trained and evaluated on the nuScenes data set. On the test set, fusion of radar data increases the resulting AP (Average Precision) detection score by about 5.1% in comparison to the baseline lidar network. The radar sensor fusion proves especially beneficial in inclement conditions such as rain and night scenes. Fusing additional camera data contributes positively only in conjunction with the radar fusion, which shows that interdependencies of the sensors are important for the detection result. Additionally, the paper proposes a novel loss to handle the discontinuity of a simple yaw representation for object detection. Our updated loss increases the detection and orientation estimation performance for all sensor input configurations. The code for this research has been made available on GitHub.

Sequential Human Gait Classification With Distributed Radar Sensor Fusion

This paper presents different information fusion approaches to classify human gait patterns and falls in a radar sensors network. The human gaits classified in this work are both individual and sequential, continuous gait collected by a FMCW radar and three UWB pulse radar placed at different spatial locations. Sequential gaits are those containing multiple gait styles performed one after the other, with natural transitions in between, including fall events developing from walking gait in some cases. The proposed information fusion approaches operate at signal and decision level. For the signal level combination, a simple trilateration algorithm is implemented on the range data from the 3 UWB radar sensors, achieving good classification results with the proposed Bi-LSTM (Bidirectional LSTM neural network) as classifier, without exploiting conventional micro-Doppler information. For the decision level fusion, the classification results of individual radars using the Bi-LSTM network are combined with a robust Naive Bayes Combiner (NBC), and this showed subsequent improvement compared to the single radar case thanks to multi-perspective views of the subjects. Compared to conventional SVM and Random Forest classifiers, the proposed approach yields +20% and +17% improvement in the classification accuracy of individual gaits for the range-only trilateration method and NBC decision fusion method, respectively. When classifying sequential gaits, the overall accuracy for the two proposed methods reaches 93% and 90%, with validation via a ’leaving one participant out’ approach to test the robustness with subjects unknown to the network.

Encoder-Camera-Ground Penetrating Radar Sensor Fusion: Bimodal Calibration and Subsurface Mapping

In this article, we report system and algorithmic developments for a sensing suite comprising a camera and a ground penetrating radar (GPR) with a wheel encoder designed for both surface and subsurface infrastructure inspection, which is a multimodal mapping task. To fuse different sensor modalities properly, we solve a novel GPR-camera calibration problem and a synchronization-challenged sensor fusion problem. First, we design a calibration rig, model the GPR imaging process, introduce a mirror to obtain the joint coverage between the camera and the GPR, and employ the maximum-likelihood estimator to estimate the relative pose between the camera and the GPR with error analysis. Second, we propose a data collection scheme using the customized artificial landmarks to synchronize camera images (temporally evenly spaced) and GPR/encoder data (spatially evenly spaced). We also employ pose graph optimization with location discrepancy as penalty functions to perform data fusion for 3-D reconstruction. We have tested our system in physical experiments. The results show that our system successfully fuses encoder-camera-GPR sensory data and accomplishes a metric 3-D reconstruction. Moreover, our sensor fusion approach reduces the end-to-end distance error from 6.4 to 0.7 cm in a real bridge inspection experiment if comparing to the counterpart that only uses encoder measurements.

LiDAR and Radar Sensor Fusion for Localizing Autonomous Vehicles

LiDAR and Radar Sensor Fusion for Localizing Autonomous Vehicles

Camera and Radar Sensor Fusion for Robust Vehicle Localization via Vehicle Part Localization

Many production vehicles are now equipped with both cameras and radar in order to provide various driver-assistance systems (DAS) with position information of surrounding objects. These sensors, however, cannot provide position information accurate enough to realize highly automated driving functions and other advanced driver-assistance systems (ADAS). Sensor fusion methods were proposed to overcome these limitations, but they tend to show limited detection performance gain in terms of accuracy and robustness. In this study, we propose a camera-radar sensor fusion framework for robust vehicle localization based on vehicle part (rear corner) detection and localization. The main idea of the proposed method is to reinforce the azimuth angle accuracy of the radar information by detecting and localizing the rear corner part of the target vehicle from an image. This part-based fusion approach enables accurate vehicle localization as well as robust performance with respect to occlusions. For efficient part detection, several candidate points are generated around the initial radar point. Then, a widely adopted deep learning approach is used to detect and localize the left and right corners of target vehicles. The corner detection network outputs their reliability score based on the localization uncertainty of the center point in corner parts. Using these position reliability scores along with a particle filter, the most probable rear corner positions are estimated. Estimated positions (pixel coordinate) are translated into angular data, and the surrounding vehicle is localized with respect to the ego-vehicle by combining the angular data of the rear corner and the radar's range data in the lateral and longitudinal direction. The experimental test results show that the proposed method provides significantly better localization performance in the lateral direction, with greatly reduced maximum errors (radar: 3.02m, proposed method: 0.66m) and root mean squared errors (radar: 0.57m, proposed method: 0.18m).

Detection scheme for a partially occluded pedestrian based on occluded depth in lidar–radar sensor fusion

Object detections are critical technologies for the safety of pedestrians and drivers in autonomous vehicles. Above all, occluded pedestrian detection is still a challenging topic. We propose a new detection scheme for occluded pedestrian detection by means of lidar–radar sensor fusion. In the proposed method, the lidar and radar regions of interest (RoIs) have been selected based on the respective sensor measurement. Occluded depth is a new means to determine whether an occluded target exists or not. The occluded depth is a region projected out by expanding the longitudinal distance with maintaining the angle formed by the outermost two end points of the lidar RoI. The occlusion RoI is the overlapped region made by superimposing the radar RoI and the occluded depth. The object within the occlusion RoI is detected by the radar measurement information and the occluded object is estimated as a pedestrian based on human Doppler distribution. Additionally, various experiments are performed in detecting a partially occluded pedestrian in outdoor as well as indoor environments. According to experimental results, the proposed sensor fusion scheme has much better detection performance compared to the case without our proposed method.

Firefighting Robot Stereo Infrared Vision and Radar Sensor Fusion for Imaging through Smoke

Firefighting robots are actively being researched to reduce firefighter injuries and deaths as well as increase their effectiveness on performing tasks. There has been difficulty in making firefighting robots autonomous because the commonly used sensors for autonomous robot navigation do not perform well in fire smoke-filled environments where low visibility and high temperature are present. In order to overcome these limitations, a multi-spectral vision system was developed that uses sensor fusion between stereo thermal infrared (IR) vision and frequency modulated-continuous wave (FMCW) radar to locate objects through zero visibility smoke in real-time. In this system, the stereo IR vision was used to obtain 3-D information about the scene while the radar provided more accurate distances of objects in the field of view. Through globally matching radar objects with those in the 3-D image, the accuracy of the stereo IR vision map was updated removing the far-field inaccuracy of the stereo IR as well as ghost objects created due to stereo mismatch. The system was sufficiently fast to provide real-time matching of objects in the scene allowing for dynamic reaction object tracking and locating. Through large-scale fire experiments with and without smoke in the field of view, the distance error for the stereo IR vision was reduced from 1% to 19.0% to 1% to 2% due to sensor fusion of the stereo IR with FMCW radar.

Collision Sensing by Stereo Vision and Radar Sensor Fusion

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> To take advantage of both stereo cameras and radar, this paper proposes a fusion approach to accurately estimate the location, size, pose, and motion information of a threat vehicle with respect to a host one from observations that are obtained by both sensors. To do that, we first fit the contour of a threat vehicle from stereo depth information and find the closest point on the contour from the vision sensor. Then, the fused closest point is obtained by fusing radar observations and the vision closest point. Next, by translating the fitted contour to the fused closest point, the fused contour is obtained. Finally, the fused contour is tracked by using rigid body constraints to estimate the location, size, pose, and motion of the threat vehicle. Experimental results from both synthetic data and real-world road test data demonstrate the success of the proposed algorithm. </para>

A Robust State Space Model for the Characterization of Extended Returns in Radar Target Signatures

Analysis of radar scattering from targets with curved boundaries, such as objects comprising cylindrical and conical shapes, is important to many aerospace applications. The radar return is composed of a well-characterized physical optics response in the illuminated region where the transmitter and receiver are not shadowed by the object, and a combination of modal responses (e.g., creeping waves and edge-diffracted fields) in the shadow region. The modal responses have longer down-range than scattering centers located on the object, and therefore, produce extended (or off-body) returns in ISAR images, which are not well-understood. However, these returns are strongly dependent on local features of the object, and thus contain valuable information with regard to the target's geometrical and physical composition. Multiple reflections from illuminated facets, as well as multiply diffracted waves, can also add coherently in the direction of the receiver and produce such returns. This paper applies a robust, coherent-processing system identification technique, originally developed for radar sensor fusion, to estimate amplitude and phase of the scatterers that characterize extended returns in the target signature. Examples are presented that highlight the extraction of creeping waves using measured data on a cone-sphere.