Heterogeneous Excitation-and-Squeeze Network for visual dialog

Abstract

Visual dialog is a challenging vision-language task, which requires an agent to answer a series of questions about an image. To answer correctly, the agent should explore question-relevant clues from heterogeneous information, such as resolve visual coreference using dialog history, or locate the referred object in the image. Previous methods usually capture question-relevant factors from heterogeneous information in a sequential manner, that is, first enrich the question context with relevant dialog history, and then use the enriched question to perform visual grounding. However, this sequential way may bring unexpected information mix, leading to wrong answers. In this paper, we propose Heterogeneous Excitation-and-Squeeze Network (HESNet), which can handle with heterogeneous information in a parallel and adaptive way. HESNet contains two key modules: Heterogeneous Excitation Module (HE-Module) and Heterogeneous Squeeze Module (HS-Module). HE-Module excavates question-related clues in heterogeneous information, namely image, dialog history, and candidate answer parallelly by conducting bi-directional excitation operations. Then, the HS-Module squeezes the enriched question representation and multiple enhanced heterogeneous representations adaptively for final answer prediction. Experimental results on VisDial v0.9 and v1.0 datasets show the superiority of the proposed approach over the state-of-the-art methods. The visualization analysis further shows that the proposed HESNet alleviates the information mix by collecting the heterogeneous clues parallelly and adaptively.