Shape-From-Template with Curves

Abstract

Shape-from-Template (SfT) is the problem of using a shape template to infer the shape of a deformable object observed in an image. The usual case of SfT is ‘Surface’ SfT, where the shape is a 2D surface embedded in 3D, and the image is a 2D perspective projection. We introduce ‘Curve’ SfT, comprising two new cases of SfT where the shape is a 1D curve. The first new case is when the curve is embedded in 2D and the image a 1D perspective projection. The second new case is when the curve is embedded in 3D and the image a 2D perspective projection. We present a thorough theoretical study of these new cases for isometric deformations, which are a good approximation of ropes, cables and wires. Unlike Surface SfT, we show that Curve SfT is only ever solvable up to discrete ambiguities. We present the necessary and sufficient conditions for solvability with critical point analysis. We further show that unlike Surface SfT, Curve SfT cannot be solved locally using exact non-holonomic Partial Differential Equations. Our main technical contributions are two-fold. First, we give a stable, global reconstruction method that models the problem as a discrete Hidden Markov Model. This can generate all candidate solutions. Second, we give a non-convex refinement method using a novel angle-based deformation parameterization. We present quantitative and qualitative results showing that real curve shaped objects such as a necklace can be successfully reconstructed with Curve SfT.