MiC: Image-text Matching in Circles with cross-modal generative knowledge enhancement

Abstract

Image-text matching is a challenging task due to vast discrepancies between the visual and textual modalities. Existing solutions tend to focus on a limited set of strongly aligned or misaligned elements, failing to fully bridge the inherent gap between two modalities. In this work, we introduce a collaborative framework that enables simultaneous learning of the matching task alongside two generative processes designed to generate textual and visual knowledge from cross-modal inputs. In such a way, the overall cross-modal consistency is estimated in a dual-view manner by estimating the similarities not only between the original textual and visual data (cross-modal), but also across the input and the generative knowledge (uni-modal). With the incorporation of generative components, the entire matching pipeline forms two closed circles, introducing a novel paradigm of Matching in Circles (MiC). Comprehensive experiments on two public datasets (Flickr30K and MS-COCO) demonstrate the superiority of the proposed method in both image-to-text and text-to-image retrieval scenarios.