Human body reshaping and its application using multiple RGB-D sensors

Abstract

Automatic reshaping of human bodies is a computer vision and graphics technique with many applications. It manipulates various shape attributes of the visual appearance of a person without any manual editing. Keeping coherent reshaping results across many video frames is more challenging. In this paper, we present a novel pipeline to reshape the human body using noisy depth data from multiple RGB-D sensors. Compared with a single view, the data from multiple RGB-D sensors provide more constraints and lead to more consistent results. However, there exist a number of challenges in estimating the pose and shape of human in RGB-D data due to self-occlusion and motion complexity. To cope with the time-varying articulated human shape, we propose a new approach that combines a Gaussian Mixture Model (GMM) based fitting approach with a morphable model learned from range scans. Without any user input, this approach can automatically account for the variations in pose and shape, and enable different types of reshaping by changing body attributes such as height, weight or other physical features. Experimental results are provided to demonstrate the effectiveness of our system in manipulation of human body shapes.