Long term activity prediction in first person viewpoint

Abstract

Predicting future activities from an egocentric viewpoint is of particular interest for assisted living arena. More importantly, knowing how future activities evolve with time (i.e., what will happen first, second, and etc.) enables lots of potential applications, e.g. usage in intelligent robot to forecast humans next movement and occurrence time to prepare for assisting human being with the movement in advance. However, state-of-the-art techniques are mostly NOT capable of predictive tasks, as their synchronous processing architecture performs poorly in either modeling long-term event dependency or pruning temporal redundant features. In this work, we propose an asynchronous gaze-event driven activity evolving curve prediction network. This network features two modules (sub-networks): 1) a gaze-event extraction module which models the asynchronous occurrence/ending of a certain activity; and 2) an asynchronous reasoning module which generates event-modulated activity occurring probability functions via deriving long-temporal dependency from various events. Extensive experimental results on several egocentric video benchmarks well demonstrate the effectiveness of the proposed method in long-term activity evolving curve prediction.