3D hand reconstruction via aggregating intra and inter graphs guided by prior knowledge for hand-object interaction scenario

Abstract

Recently, 3D hand reconstruction has gained more attention in human–computer cooperation, especially for hand-object interaction scenario. However, it still remains challenge to improve estimation accuracy and ensure physical plausibility from a single RGB image. To overcome the challenge, we propose a 3D hand reconstruction network combining the benefits of model-based and model-free approaches to balance accuracy and physical plausibility for hand-object interaction scenario. Firstly, we present a novel topology-aware MANO pose parameters regression module from 2D joints directly. Moreover, we further carefully design a vertex-joint mutual graph-attention module guided by MANO to jointly refine hand meshes and joints, which model the dependencies of vertex-vertex and joint-joint and capture the correlation of vertex-joint for aggregating intra-graph and inter-graph node features respectively. The experimental results demonstrate that our method achieves state-of-the-art performance on recently benchmark datasets HO3DV2 and Dex-YCB, and outperforms all only model-based approaches or model-free approaches.