A Multiview Text Imagination Network Based on Latent Alignment for Image-Text Matching

Abstract

In image-text matching field, one of the keys to improving performance is to extract features with more semantic information. Existing works demonstrate that semantic enrichment through knowledge expansion can improve the performance. Most of them expand image features. However, the shortage of semantic information in text modality and the unilateral character of the view are often bottlenecks that limit the performance of image-text matching models. To solve the two problems, we aggregate knowledge from multiple views and propose a Word Imagination Graph (WIG). WIG can be used to expand textual semantic information by imagination based on input images. Then, utilizing WIG, we construct a novel Multi-View Text Imagination Network (MTIN). MTIN enables latent alignment of images and texts on tags which can assist matching on a semantic level. Results on Flickr30K and MS-COCO datasets can demonstrate the effectiveness of our method.