Abstract

Paragraph-style image captions describe diverse aspects of an image as opposed to the more common single-sentence captions that only provide an abstract description of the image. These paragraph captions can hence contain substantial information of the image for tasks such as visual question answering. Moreover, this textual information is complementary with visual information present in the image because it can discuss both more abstract concepts and more explicit, intermediate symbolic information about objects, events, and scenes that can directly be matched with the textual question and copied into the textual answer (i.e., via easier modality match). Hence, we propose a combined Visual and Textual Question Answering (VTQA) model which takes as input a paragraph caption as well as the corresponding image, and answers the given question based on both inputs. In our model, the inputs are fused to extract related information by cross-attention (early fusion), then fused again in the form of consensus (late fusion), and finally expected answers are given an extra score to enhance the chance of selection (later fusion). Empirical results show that paragraph captions, even when automatically generated (via an RL-based encoder-decoder model), help correctly answer more visual questions. Overall, our joint model, when trained on the Visual Genome dataset, significantly improves the VQA performance over a strong baseline model.

Highlights

  • Understanding visual information along with natural language have been studied in different ways

  • In visual question answering (VQA) (Antol et al, 2015; Goyal et al, 2017; Lu et al, 2016; Fukui et al, 2016; Xu and Saenko, 2016; Yang et al, 2016; Zhu et al, 2016; Anderson et al, 2018), models are trained to choose the correct answer given a question about an image

  • Similar to the VQA task, image captioning models should learn the relationship between partial areas in an image and the generated words or phrases

Read more

Summary

Introduction

Understanding visual information along with natural language have been studied in different ways. Similar to the VQA task, image captioning models should learn the relationship between partial areas in an image and the generated words or phrases. While these two tasks seem to have different directions, they have the same purpose: understanding visual information with language. Paragraph-style descriptive captions can more explicitly (via intermediate symbolic representations) explain what objects are in the image and their relationships, and VQA questions can be answered more by matching the textual information with the questions. C 2019 Association for Computational Linguistics these paragraph captions and attribute sentences as input in addition to the standard input image features.

Related Work
Paragraph Captioning Model
Features
Three Fusion Levels
VQA baseline
TextQA with GenP
Conclusion
Findings
A Attention Visualization
Full Text
Published version (Free)

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call