Person Recognition Based on Micro-Doppler and Thermal Infrared Camera Fusion for Firefighting

Abstract

This paper examines the recognition of real persons, mirrored persons and other objects using thermal infrared (TIR) images and radar micro-Doppler (µ-D). Mirrored persons lead to confusion of firefighters, when only a TIR camera is used. However, mirrored persons exhibit the µ- D of the mirroring objects, hence radar can resolve this ambiguity. In this paper, multiple sensor fusion architectures are investigated for this classification task. The first approach uses an attention stage, where bounding boxes of candidates for real/mirrored persons are determined in TIR images. These bounding boxes are associated to the radar targets and subsequently classified. A joint classification of the radar µ- D and TIR image at measurement level is compared to a separate classification with subsequent combination (object level). Furthermore, a classification of the complete scene is proposed, omitting the TIR attention stage and data association. Experiments with real measurements are used for an evaluation of the presented approaches.