Towards Open Ended and Free Form Visual Question Answering: Modeling VQA as a Factoid Question Answering Problem

Abstract

Visual Question Answering (VQA) is a multi disciplinary challenging problem involving various fields such as Natural Language Processing, Computer Vision, Deep Learning and Knowledge Representation, which has garnered much interest among researchers, especially with the recent advancements in machine perception. The problem at hand not only involves reasoning over visual elements present in the image or natural language understanding of the input query, but also may involve outside world knowledge in order to infer the answer. In this paper, we convert the VQA problem to a factoid question answering task over a set of natural language facts extracted from images in the Visual Genome Dataset [1]. Recent literatures in textual question answering have established the effectiveness of End to End Memory Networks (MemN2N) over the standard LSTMs. Inspired by the approaches incorporated by researchers in this direction, as a first step to create an explainable VQA model, this paper proposes the incorporation of MemN2N with soft attention for inferring the answer from a set of regional facts extracted from the image. We also experiment with the addition of a Bayesian Neural layer for posterior reasoning of the answer from a fixed vocabulary, as proposed in [2] which shows a significant improvement in the accuracy score compared to the other models tested.