Action database for categorizing and inferring human poses from video sequences

Abstract

One of the difficulties in automated recognition of human activities is classifying a video into a specific action class by selecting among a large number of human actions. Technology for understanding complex and varied human actions is necessary for automated surveillance, sports training, computer games, and human–robot interactions. The difficulty of classification comes from a dearth of datasets of human actions that are manually categorized and suitable for use as training data for designing action classifiers. A marker-based motion capture system enables precise measurement of human actions for the purpose of analysis. This type of capture system has several drawbacks, however; in particular, marker-based systems are expensive, intrusive, and complex to use. Despite this, the intensive use of a motion capture system can provide large datasets of human actions, and the datasets can be used to facilitate handling the variety of actions to be classified. Large datasets of human actions measured by motion capture systems are expected to be suitable for use in classifying video segments into the correct human action category, selecting from among a large number of action categories, and for inferring human postures from video. This paper proposes a new concept for a database of human whole body actions and an application to understanding human actions from video. The database contains action configurations, such as positions of body parts, pose descriptors from silhouette images, a stochastic model encoding each sequence of the pose descriptors, and a regression model for predicting the configuration from the pose descriptor. The action configurations are recorded in advance of use by measuring many human actions with a marker-based motion capture system, and silhouette images are created from these configurations. We tested the action database on action classification tasks and human body posture inference tasks. The experimental results show that the action database is suitable for use in both action classification and posture inference.