Human action recognition with DeepAction Kernel Gaussian Process

Abstract

Human action recognition is a challenging vision task due to the complex action patterns in the real-world videos. In this work, we propose a DeepAction Kernel Gaussian Process, which takes advantage of Gaussian process (GP) and deep learning, to capture the distinctive action characteristics. Specifically, we design a unified, deep and non-adjacent kernel structure within Gaussian process to classify different actions. First, we design an adaptive GMM kernel (adGMMK) to encode the low-level features of different actions in a specific manner. Second, inspired by the theoretical connections between neural network and GP, we transform our shallow kernels into deep kernels to learn actions with their high-level representations. Finally, we propose a novel non-adjacent kernel framework to leverage the benefits from both shallow and deep kernels for action classification. Our experimental results on two benchmark data sets (HMDB51 and UCF101) show the superior performance of our approach, in comparison with several relevant works.