Human action recognition based on action relevance weighted encoding

Abstract

Human action recognition in videos is one of the most important and active topics in computer vision, and building more discriminative video representation is crucial for action classification. In this paper, we propose a new video representation which not only captures temporal dynamics in a video but also takes account of the importance of different trajectories and video frames in action recognition. More specifically, first, we propose a new trajectory descriptor based on traditionally used local descriptors including histograms of oriented gradients, histograms of optical flow and motion boundary histogram, and it takes the correlation between trajectories and target action into consideration. Secondly, a new method to compute saliency map based on optical flow is introduced to highlight regions of foreground motion. Thirdly, to identify discriminating trajectories and frames in a video, trajectory action relevance and frame action relevance are defined and used as weights during encoding. Finally, two video representations which describe a video at different levels of abstraction are proposed, they are complementary to each other and combined to generate more expressive and discriminative representation. Experimental results on KTH, JHMDB and HMDB51 demonstrate the effectiveness and excellent performance of the proposed approach.