Textual-Visual Reference-Aware Attention Network for Visual Dialog

Abstract

Visual dialog is a challenging task in multimedia understanding, which requires the dialog agent to answer a series of questions that are based on an input image. The critical issue to produce an exact answer is how to model the mutual semantic interaction among feature representations of the image, question-answer history, and current question. In this study, we propose a textual-visual Reference-Aware Attention Network (RAA-Net), which aims to effectively fuse Q (question), H (history), Vl (local vision), and Vg (global vision) to infer the exact answer. In the multimodal feature flows, RAA-Net first learns the textual context through multi-head attention between Q and H and then guides the textual reference semantics to the image to capture visual reference semantics by self-and cross-reference-aware attention in and between Vl and Vg. In the proposed RAA-Net, we exploit the two-stage (intraand inter-) visual reasoning mechanism on Vl and Vg. Extensive experiments on the VisDial v0.9 and v1.0 datasets show that RAA-Net achieves state-of-the-art performance. Visualization results on both visual and textual attention maps further validate the remarkable interpretability achieved by our solution.