3D hand reconstruction with both shape and appearance from an RGB image

Abstract

Modeling 3D hands with geometry details and appearance can increase perceptual immersion and realism in many applications. However, traditional 3D representations such as meshes and voxel grids can not represent high-quality hand shapes with their property of the discrete surface and fixed topology. To address these problems, we introduce implicit function for more detailed shape reconstruction, and design a Structure-aware Signed Distance Function (S-SDF) to reconstruct hand shape in arbitrary resolution. In this way, our method not only focuses on independent fingers but also keeps the relationship between fingers. Such implicit function of 3D representation associated with part semantics and structure of the hand is central to generating realistic hand shapes of diverse variations. Meanwhile, in order to avoid time-consuming and laborious 3D annotating of texture, we present a self-supervised appearance synthesis approach. Extensive experiments on widely used datasets demonstrate that the proposed method reconstructs more realistic hands compared with previous methods.