Visual Question Answer System Based on Bidirectional Recurrent Networks

Abstract

Visual Question Answer (VQA) system is the task of automatically answering natural language questions based on the content of reference image. A commonly approach for VQA is to extract image feature and question feature by convolution neural network (CNN) and long short-term memory network (LSTM) respectively, and then combine them to infer the answer through attention mechanism such as the stacked attention networks (SAN). However, the CNN ignores the information between adjacent image regions and the LSTM just memorizes the past contextual information of the question. In this paper, we propose a model based on two bidirectional recurrent networks (BiSRU and BiLSTM) to improve the accuracy of feature extraction. The BiSRU is used to allow the adjacent local region vectors of the image to maintain information each other. The BiLSTM is used to encode the question feature, which obtains past and future contextual information meanwhile when the question is very complex. The feature of image and question obtained by bidirectional recurrent networks is used to predict the answer precisely. Experiment result shows that our model get better performance on four datasets.