On the role of question encoder sequence model in robust visual question answering

Abstract

Generalizing beyond the experiences has a significant role in developing robust and practical machine learning systems. It has been shown that current Visual Question Answering (VQA) models are over-dependent on the language-priors (spurious correlations between question-types and their most frequent answers) from the train set and pose poor performance on Out-of-Distribution (OOD) test sets. This conduct negatively affects the robustness of VQA models and restricts them from being utilized in real-world situations. This paper shows that the sequence model architecture used in the question-encoder has a significant role in the OOD performance of VQA models. To demonstrate this, we performed a detailed analysis of various existing RNN-based and Transformer-based question-encoders, and along, we proposed a novel Graph attention network (GAT)-based question-encoder. Our study found that a better choice of sequence model in the question-encoder reduces the over-fit to language biases and improves OOD performance in VQA even without using any additional relatively complex bias-mitigation approaches.