Predictability Analyzing: Deep Reinforcement Learning for Early Action Recognition

Abstract

Early action recognition aims at inferring ongoing activities from partial videos as early as possible, whereas conventional action recognition relies on fully observed activities. Observations show that the predictability of different activity subsequences vary wildly, however most existing work failing to fully exploit this phenomenon. We define the predictability of activity subsequences as its capacity to perform recognition early and accurately. A predictability-based early action recognition framework(PEAR) is established to utilize predictability information to achieve early recognition. It consists of a predictability evaluator and a classifier. Due to lacking of fine-grained supervision, we develop a reinforcement-learning-based strategy to optimize the evaluator encouraged by a recognizability reward and an early reward. With the predictability estimated by the evaluator, the classifier learns discriminative representation of subsequences to perform early action recognition without sacrificing much accuracy. Experiments on two benchmark datasets demonstrate the proposed approach outperforms existing methods significantly.