Deep recursive and hierarchical conditional random fields for human action recognition

Abstract

The linear-chain CRFs is one of the most popular discriminative models for human action recognition, as it can achieve good prediction performance in temporal sequential labeling by capturing the one-or few-timestep interactions of the target states. However, existing CRFs formulations have limited capabilities to capture deeper intermediate representations within the target states and higher order dependence between the given states, which are potentially useful and significant in the modeling of complex action recognition scenarios. To address these issues, we formulate a deep recursive and hierarchical conditional random fields (DR-HCRFs) model in an infinite-order dependencies framework. The DR-HCRFs model is able to capture richer contextual information in the target states, and infinite-order temporal-dependencies between the given states. Moreover, we derive a mean-field-like approximation of the model marginal likelihood to efficiently facilitate the model inference. The parameters of the predefined model are learnt with the block-coordinate primal-dual Frank-Wolfe algorithm in a structured support vector machine framework. Experimental results on the CAD-120 benchmark dataset demonstrate that the proposed approach can achieve high scalability and perform better than other state-of-the-art methods in terms of the evaluation criteria.