Knowledge guided learning: Open world egocentric action recognition with zero supervision

Abstract

• Current approaches are driven by data-hungry deep learning algorithms which require large amounts of annotated training data. • Deep learning models are inductive learners where the vocabulary is fixed and do not generalize beyond their training domain. • We address the problem of open world action recognition (i.e., unknown vocabulary) with Pattern Theory and Concept-Net. • Extensive experiments show our competitive performance for open world egocentric action recognition and object detection. Advances in deep learning have enabled the development of models that have exhibited a remarkable tendency to recognize and even localize actions in videos. However, they tend to experience errors when faced with scenes or examples beyond their initial training environment. Hence, they fail to adapt to new domains without significant retraining with large amounts of annotated data. In this paper, we propose to overcome these limitations by moving to an open-world setting by decoupling the ideas of recognition and reasoning. Building upon the compositional representation offered by Grenander’s Pattern Theory formalism, we show that attention and commonsense knowledge can be used to enable the self-supervised discovery of novel actions in egocentric videos in an open-world setting, where data from the observed environment (the target domain) is open i.e., the vocabulary is partially known and training examples (both labeled and unlabeled) are not available. We show that our approach can infer and learn novel classes for open vocabulary classification in egocentric videos and novel object detection with zero supervision . Extensive experiments show its competitive performance on two publicly available egocentric action recognition datasets (GTEA Gaze and GTEA Gaze+) under open-world conditions.