Deep Reconstruction of 3-D Human Poses From Video

Abstract

Deep Learning advances have made it possible to recover full 3D meshes of human models from individual images. However, extension of this notion to videos for recovering temporally coherent poses is still under-explored. A major challenge in this direction is the lack of appropriately annotated video data for learning the desired computational models. Existing human pose datasets only provide 2D or 3D skeleton joint annotations, whereas the datasets are also insufficiently recorded in constrained environments. We first contribute a technique to synthesize monocular action videos with rich 3D annotations that are suitable for learning computational models for full mesh 3D human pose recovery. Compared to the existing methods which simply ‘`texture-map’' clothes onto the 3D human pose models, our approach incorporates Physics-based realistic cloth deformations with human body movements. The generated videos cover a large variety of human actions, poses, and visual appearances, while the annotations record accurate human pose dynamics and human body surface information. Our second major contribution is an end-to-end trainable Recurrent Neural Network for full pose mesh recovery from monocular video. Using the proposed video data and a Long-Short-Term-Memory recurrent structure, our network explicitly learns to model the temporal coherence in videos and imposes geometric consistency over the recovered meshes. We establish the effectiveness of the proposed model with quantitative and qualitative analysis using the proposed and benchmark datasets.