PCHPS: The Estimation of 3D Hand Pose and Shape using Point Cloud from a Single Depth Image

Abstract

The estimation of 3D hand pose and shape is important for vision-based applications such as augmented reality and virtual reality. Most current methods in 3D hand analysis from a single depth image only focus on joints positions, which cannot fully express the 3D shape of hand. In this paper, Point Cloud for Hand Pose and Shape estimation (PCHPS) which combines the structure of the encoder of So-Net and the structure of Graph Convolutional Neural Network (GCNN) is proposed to recover a complete hand shape and locate hand joint positions simultaneously. A million-scale synthetic dataset with ground truth of 3D meshes and poses called SynHand5M is used to train PCHPS. The normalized point clouds and SOM nodes are inputted to PCHPS to capture the complex structure of 3D hand. A joint training strategy with real and synthetic datasets is introduced to fine-tune PCHPS on real-world datasets without 3D ground truth. Experiments are conducted on the NYU, ICVL and MSRA dataset to compete with the state-of-the-art methods, which shows that PCHPS can achieve superior results in the estimation of 3D hand pose and is real time.