Dance motion capture and composition using multiple RGB and depth sensors

Yejin Kim

doi:10.1177/1550147717696083

Abstract

Dynamic human movements such as dance are difficult to capture without using external markers due to the high complexity of a dancer’s body. This article introduces a marker-free motion capture and composition system for dance motion that uses multiple RGB and depth sensors. Our motion capture system utilizes a set of high-speed RGB and depth sensors to generate skeletal motion data from an expert dancer. During the motion acquisition process, a skeleton tracking method based on a particle filter is provided to estimate the motion parameters for each frame from a sequence of color images and depth features retrieved from the sensors. The expert motion data become archived in a database. The authoring methods in our composition system automate most of the motion editing processes for general users by providing an online motion search with an input posture and then performing motion synthesis on an arbitrary motion path. Using the proposed system, we demonstrate that various dance performances can be composed in an intuitive and efficient way on client devices such as tablets and kiosk PCs.

Highlights

Performing arts such as dance and theater are in the domain of intangible cultural heritage that risk becoming buried in oblivion if not archived in a proper record
We demonstrated the applicability of our system by capturing various dance movements from expert dancers and providing general users the captured motion data for dance motion composition
We have introduced a motion capture system that can track dynamic movements in dance motions without using external markers

Summary

Introduction

Performing arts such as dance and theater are in the domain of intangible cultural heritage that risk becoming buried in oblivion if not archived in a proper record. While systems with a single Kinect sensor suffer from the body’s occlusion and rotation problems,[3] systems with multiple sensors can capture a dancer’s posture at different angles and combine these views into continuous motion data.[4,5,6,7] These approaches have mainly targeted the capturing of relatively simple steps and dance gestures, which are not suitable for complicated and dynamic movements in ballet, modern, and K-pop dances. We propose a marker-free motion capture and composition system that can generate motion capture data from expert dancers and compose new dance performances for general users using the captured data. During the motion acquisition process, our capture system tracks the bodily movements of an expert dancer based on the input data, RGB and depth (RGB-D) images retrieved from multiple RGB-D sensors and constructs a sequence of three-dimensional (3D) skeletal postures from the input data. The experimental results are demonstrated in section ‘‘Experimental results.’’ We conclude this article with a discussion of future improvements in section ‘‘Conclusion.’’

Related work

Experimental results

Conclusion