Robust human action recognition based on spatio-temporal descriptors and motion temporal templates

Abstract

Human action recognition is a challenging area of research because of its various potential applications in visual surveillance. Integrating motion temporal templates with spatio-temporal interest points based appearance descriptor for action recognition is proposed in this paper. First, we model the background in a scene using Gaussian mixture model and extract the foreground objects and binary mask of region of interest in a scene. Then, for the binary mask of object, we constructed motion temporal templates using the motion history image (MHI) and motion energy image (MEI); for the foreground objects, we performed spatio-temporal interest points (STIPs) detector. Thirdly, human action representation is combined with three-dimensional Scale Invariant Feature Transform descriptor (3D SIFT) on STIPs and Hu moments extracted from MHI and MEI. Generalized Multiple Kernel Learning (GMKL) is adapted to classification the actions. To validate the proposed descriptor, we have conducted extensive experiments on the KTH and Weizmann action datasets. The results prove that the 3D SIFT offers complementary information to motion temporal templates, showing its effectiveness for action representation; moreover, the proposed method yields comparable results with the state-of-the-art methods.