Through-Wall Human Pose Reconstruction Based on Cross-Modal Learning and Self-Supervised Learning

Abstract

Recent through-wall radar (TWR) systems can reconstruct the pose of human targets blocked by occlusion. They rely on the fusion of optical and radar data to avoid the painful annotation burden. However, the fusion process is not always reliable, especially for human joint coordinates that carry 3-D spatial information. Inspired by cross-modal learning and self-supervised learning, this letter proposes a two-stage 3-D human pose reconstruction method for TWR systems. In the cross-modal supervision stage, the pretrained optical model provides initial noisy labels extracted from optical images. In the self-supervision stage, supervised labels and the model weight are corrected circularly with radar images. The self-supervision enhances the robustness of the model and the reliability of labels. It can be directly extended to existing radar-based pose reconstruction methods, and hardly requires extra training time. Experiments show the model beats state of the art (SOTA) for reconstructing 3-D poses from TWR images and contains robust generalization in unseen wall-occlusive scenes.