Combining appearance and structural features for human action recognition

Abstract

In this paper, we propose to integrate structural information with appearance features for human action recognition. In local representations based on detected spatio-temporal interest points (STIPs), the layout of STIPs carries important cues of motion structures in video sequences, and is assumed to contain complementary information to appearance features. We aim to incorporate structures into the description of STIPs by combing with appearance features for action representation.Based on the previous work of the 3D shape context, we present an optimised version of 3D shape context to encode the layout information of STIPs. By combining the proposed optimised 3D shape context with appearance descriptors, e.g., HOG3D and 3D gradients, we provide a more informative and discriminative description of STIPs for action classification. To validate the proposed descriptor, we have conducted extensive experiments on the KTH and the UCF YouTube datasets. The results prove that the optimised 3D shape context offers complementary information to appearance features, showing its effectiveness for action representation; moreover, the proposed descriptor yields comparable results with the state-of-the-art methods.