Abstract

Visual dialog, which aims to hold a meaningful conversation with humans about a given image, is a challenging task that requires models to reason the complex dependencies among visual content, dialog history, and current questions. Graph neural networks are recently applied to model the implicit relations between objects in an image or dialog. However, they neglect the importance of 1) coreference relations among dialog history and dependency relations between words for the question representation; and 2) the representation of the image based on the fully represented question. Therefore, we propose a novel relation-aware graph-over-graph network (GoG) for visual dialog. Specifically, GoG consists of three sequential graphs: 1) H-Graph, which aims to capture coreference relations among dialog history; 2) History-aware Q-Graph, which aims to fully understand the question through capturing dependency relations between words based on coreference resolution on the dialog history; and 3) Question-aware I-Graph, which aims to capture the relations between objects in an image based on fully question representation. As an additional feature representation module, we add GoG to the existing visual dialogue model. Experimental results show that our model outperforms the strong baseline in both generative and discriminative settings by a significant margin.

Highlights

  • IntroductionH0: a man in a suit and tie standing next to a woman with glasses H1: what color is the man 's suit ?

  • We model explicit complex relations within and among visual content, dialog history and the current question and design a graph-over-graph structure which are different from graph-based models mentioned above

  • We present a relation-aware graphover-graph network (GoG), a novel framework for visual dialog, which models and reasons the explicit complex relations among visual content, dialog history, and the current question

Read more

Summary

Introduction

H0: a man in a suit and tie standing next to a woman with glasses H1: what color is the man 's suit ? Li et al, 2019; Huang et al, 2020) and visual dialog (Das et al, 2017; Kottur et al, 2018; Agarwal et al, 2020; Wang et al, 2020; Qi et al, 2020). Relations in these tasks are significant for reasoning and understanding the textual and visual information. Visual dialog, which aims to hold a meaningful conversation with a human about a given image, is a challenging task that requires models to reason complex relations among visual content, dialog history, and current questions

Methods
Results
Conclusion
Full Text
Published version (Free)

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call