MTMVC: Semi-supervised 3D hand pose estimation using multi-task and multi-view consistency

Abstract

The high performance of state-of-the-art deep learning methods for 3D hand pose estimation heavily depends on a large annotated training set. However, it is difficult and time-consuming to obtain the annotations for 3D hand poses. To leverage unannotated images to reduce the annotation cost, we propose a semi-supervised method based on Multi-Task and Multi-View Consistency (MTMVC) for hand pose estimation. First, we obtain the joints based on heatmap prediction and coordinate regression parallelly and encourage their consistency. Second, we introduce multi-view consistency to encourage the predicted poses to be rotation-invariant. Thirdly, to make the network pay more attention to the hand region, we propose a spatially weighted consistency. Experiments on four public datasets showed that our proposed MTMVC outperformed existing semi-supervised hand pose estimation methods, and by only using half of the annotations, the accuracy of our method was comparable to those of several state-of-the-art fully supervised methods.