Three dimensional model-based tracking using texture learning and matching

Abstract

In post-production, a traditional method for creating some special effects is named “rotoscopy”. This technique is used in animation and consists of segmenting and modifying a video sequence by hand and for every frame (i.e., adding an animated synthetic character to a real scene). Our method makes this process automatic by tracking a rigid object whose geometry is known. This new approach is based upon a two-steps process: One or several “keyframes” are used in a preliminary interactive calibration session, so that a 3D model of this object is positioned correctly on these images (its projection fits to the object in the image). We use this match to texture the 3D model. Then, a 3D predictor gives a position of the object model in the next image and the fine tuning of this position is obtained by simply minimizing the error between the textured model in this position and the real image of the object. This minimization is performed with respect to the six degree of freedom (DOF) of the model position (three translation parameters and three rotation ones). This procedure is iterated at each frame. Test sequences show how robust this method is.