Improved shape-from-template method with perspective space constraints for disappearing features

Abstract

Inferring the 3D surface shape of a known template from 2D images captured by a monocular camera is a challenging problem. Due to the severely underconstrained nature of the problem, inferring shape accurately becomes particularly challenging when the template exhibits high curvature, resulting in the disappearance of feature points and significant differences between the inferred and actual deformations. To address this problem, this paper proposes a concise and innovative approach that utilizes a physical simulator incorporating the object’s material properties and deformation law. We utilize a view frustum space constructed from the contours of a monocular camera image to effectively restrict the physically-based free motion of the template. Additionally, we employ mesh denoising techniques to ensure the smoothness of the surface following deformation. To evaluate our shape inference results, we utilize a ground truth 3D point cloud generated from multiple viewpoint images. The results demonstrate the superior performance of our approach compared to other methods in accurately inferring deformations, particularly in scenarios where feature points are unobservable. This method carries significant practical implications across diverse domains, including virtual reality, digital modeling, and medical surgery training.