GA-Net: A Guidance Aware Network for Skeleton-Based Early Activity Recognition

Abstract

Early activity prediction, which aims to recognize class labels before actions are fully performed, is a very challenging task since partially observed action sequences contain insufficient class-discrimination information, and thus, many partial action sequences belonging to different categories may look very similar. Therefore, in this paper, we propose a novel <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">guidance aware network (GA-Net)</i> to boost the ability to distinguish different activities in diversified partially observed action sequences via metric learning. To mitigate the similarity problem of action segments at very early stages, the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">guided metric learning module (GMLM)</i> is able to encourage the feature extractor to mine class-discriminative information given partially observed sequences. Specifically, the GMLM is able to minimize the intraclass distance with a full-length guided direction approach and maximize the difference between interclass categories with different observation ratios. To enhance the similarities between the partial- and full-length sequences in the same action categories, we further introduce a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">distribution alignment module (DAM)</i> that employs full-length guidance to pull the partially observed features closer to the global features. We evaluate our proposed method on three public human activity datasets and achieve competitive results compared with the state-of-the-art approaches.