Depth-based action recognition using multiscale sub-actions depth motion maps and local auto-correlation of space-time gradients

Abstract

This paper presents a method for human action recognition from depth sequence. First, we subdivided the normalized motion energy vector into a set of segments, whose corresponding frame indices are used to partition a video. Then each sub-action is represented by three Depth Motion Maps (DMMs) to capture motion cues in three orthogonal projection views. Multi-scale Histogram of Oriented Gradients (HOG) descriptors are then computed from the DMMs for capturing the appearance cues. In order to cope with the temporal information loss in the DMMs generation, one complementary feature, a 3D motion feature descriptor, is extracted from the depth video utilizing local space-time auto-correlation of gradients (STACOG). Discriminative Multiple Canonical Correlation Analysis (DM-CCA) is then adopted to analyze DMMs-based feature and STACOG, and the two sets of features are fused into a more complete and discriminative representation of the information embedded in the dataset. l 2 -regularized Collaborative Representation Classification (l 2 -CRC) is applied to classify the proposed descriptors. Evaluations on MSR Action3D and MSRGesture3D Datasets demonstrate the effectiveness of the proposed method.