LgNet: A Local-Global Network for Action Recognition and Beyond

Abstract

This work addresses the task of action recognition in video sequences. In real world applications, this task is quite challenging due to the complex background of video content, the similarities between different types of actions, the dependence on a large amount of annotated data, and so on. Most of the existing methods fail to distinguish similar actions with the same static appearance and motion pattern. We attempt to address this issue from the perspective of a local-global view, considering videos as combinations of a set of action units (local semantic information) and their relations along temporal dimension (global relation information). To achieve this end, we propose a novel Local-global Networks (LgNet) to enhance recognition of similar action. Besides, we propose an end-to-end training method to decrease the reliance on annotated data. It combines self-supervised learning and supervised learning, which not only enables the model to learn video representations from a large number unannotated data but also avoids subsequent finetuning. The proposed training method can be flexibly equipped to a wide array of vision tasks. Experiments on several benchmark datasets show that our proposed model and training method achieve state-of-the-art performance.