STAFFormer: Spatio-temporal adaptive fusion transformer for efficient 3D human pose estimation

Abstract

Existing two-stage methods for 3D Human Pose Estimation often use 2D poses as input, which are then lifted to obtain 3D representations. This typically involves frame-by-frame estimation, whether in 2D or 3D, resulting in high computational demands unsuitable for edge devices. Due to the continuity of human movement, the differences between adjacent frames can be minimal, a question arises: is frame-by-frame estimation necessary? Previous works demonstrated the feasibility of using Transformer-based models to estimate poses with sparse frames, focusing on either temporal or spatial dependencies but neglecting holistic spatio-temporal correlations. To address this, we introduce the Spatio-Temporal Adaptive Fusion Transformer (STAFFormer). First, STAFFormer recovers dense temporal frames from sparsely sampled ones obtained from a 2D pose estimator through the Temporal Dense Frame Recovery (TDFR) module. This significantly reduces the computational complexity. Second, STAFFormer employs an adaptive fusion attention mechanism, enhancing accuracy by attentively navigating both spatial and temporal dimensions through the Spatio-Temporal Adaptive Fusion (STAF) module. Furthermore, We introduce a kinematic coherence loss to adapt to subtle joint movements, improving pose estimation fidelity. Finally, we explore the possibility of integrating different pre-training strategies using extensive marker-based datasets. Experimental results on challenging datasets show our network achieves state-of-the-art performance with low computational complexity.