Multi-level Symmetric Semantic Alignment Network for image–text matching

Abstract

Image–text matching has attracted much attention as one of the visual-linguistic tasks. Most of the existing methods tend to concentrate on single-level semantic similarity by global embeddings or local alignment, which fail to distinguish the ambiguous image–text instances with highly similar contexts due to insufficient interactions of cross-modal features. To mitigate this problem, in this paper, we propose a novel Multi-level Symmetric Semantic Alignment Network (MSSAN), which fully exploits the multi-level semantic features to capture the complicated correlations between images and text. Specifically, global–global-level alignment is first performed based on visual–textual global representations. Then, considering the local semantic consistency, local–local-level alignment is carried out through inter-modal bidirectional cross-attention module to model fine-grained region–word relations. Moreover, a Multi-granularity Feature Fusion Module (MFFM) is constructed to learn intra-modal distribution of significance and incorporate global concepts into local features to obtain more comprehensive semantic representations, thus achieving global–local-level alignment. Finally, in order to achieve a strong separation of semantically similar samples, we develop a novel generic triplet ranking loss with adaptively updatable margins to train our model. Extensive experimental results on Flickr8K, MSCOCO and Flickr30K datasets demonstrate that our proposed MSSAN is superior to other state-of-the-art methods by a considerable margin.