Adaptively Attending to Visual Attributes and Linguistic Knowledge for Captioning

Abstract

Visual content description has been attracting broad research attention in multimedia community because it deeply uncovers intrinsic semantic facet of data. Most existing approaches formulate captioning as machine translation task (i.e., from vision to language) via a top-down paradigm with global attention, which ignores to distinguish and non-visual parts during word generation. In this work, we propose a novel adaptive attention strategy for captioning, which can selectively attend to salient content based on linguistic knowledge. Specifically, we design a key control unit, termed gate, to adaptively decide when and what the language generator attend to during the word generation process. We map all the preceding outputs of language generator into a latent space to derive the representation of sentence structures, which assists the visual gate to choose appropriate attention timing. Meanwhile, we employ a bottom-up workflow to learn a pool of semantic attributes for serving as the propositional attention resources. We evaluate the proposed approach on two commonly-used benchmarks, i.e., MSCOCO and MSVD. The experimental results demonstrate the superiority of our proposed approach compared to several state-of-the-art methods.