Language-enhanced object reasoning networks for video moment retrieval with text query

Abstract

Video moment retrieval with text query aims to retrieve the most relevant video segment from the untrimmed video based on the given text query. Most existing works take advantage of pre-trained video understanding networks, which greatly limit its expressiveness and may ignore the fine-grained interaction between objects in the video. Motivated by this issue, we propose a novel model termed Language-enhanced Object Reasoning Networks (LEORN) which uses the object features extracted by the object detection network and incorporates the language features to model the relation between the objects within the frame. Besides, we design a new temporal shift mechanism to model the temporal relationship of objects while reducing the noise from irrelevant objects in the adjacent frames. The experiments on two challenging datasets, show that our proposed LEORN method with object-level features is a good complement to existing pre-trained video features and outperforms state-of-the-art methods by a simple combination.