Observation-oriented silhouette-aware fast full body tracking with Kinect

Abstract

We present a novel approach to track full human body mesh with a single depth camera, e.g. Microsoft Kinect, using a template body model. The proposed observation-oriented tracking mainly targets at fitting the body mesh silhouette to the 2D user boundary in video stream by deforming the body. It is fast to be integrated into real-time or interactive applications, which is impossible with traditional iterative optimization based approaches. Our method is a composite of two main stages: user-specific body shape estimation and on-line body tracking. We first develop a novel method to fit a 3D morphable human model to the actual body shape of the user in front of the depth camera. A strategy, making use of two constrains, i.e. point clouds from depth images and correspondence between foreground user mask contour and the boundary of projected body model, is designed. On-line tracking is made possible in successive steps. At each frame, the joint angles of template skeleton are optimized towards the captured Kinect skeleton. Then, the aforementioned contour correspondence is adopted to adjust the projected body model vertices towards the contour points of foreground user mask, using a Laplacian deformation technique. Experimental results show that our method achieves fast and high quality tracking. We also show that the proposed method is benefit to three applications: virtual try-on, full human body scanning and applications in manufacturing systems.