Multi-modal anchor adaptation learning for multi-modal summarization

Abstract

In this paper, we focus on analyzing the relationship between the input of source text and source image, and then through the integration and generalization of the multi-modal information (e.g., texts and images), outputs the multi-modal summarization including text and image. However, existing multi-modal summarization methods face several challenges: (1) Different modalities exist in different semantic spaces, and expressing the multi-modal information of source modalities in a similar semantic representation space is crucial. (2) For the result of text summarization, it is crucial to explore how to capture both the differences and similarities within the source modalities, which can reduce redundancy to improve the quality of text summarization. In order to overcome the challenge above, Multi-Modal Anchor Adaptation Learning for Multi-Modal Summarization (MA-Sum) has been proposed. Specifically, MA-Sum employs a novel and highly efficient image anchor selection method, which selects the object sample containing the richest image information as the image anchor. Simultaneously, it carefully selects the text sentence closely intertwined with the image semantics to serve as the language anchor. Therefore, the multi-modal anchor can be seen as a bridge for multi-modal alignment to alleviate the semantic gap between textual and visual. Moreover, based on the distance between the anchors and the semantic information in the respective modal, the positive and negative semantic information of each modal will be distinguished. Based on negative semantic information, the counterfactual learning mechanism is constructed to optimize the result of multi-modal summarization. Finally, the process of multi-modal features interaction is optimized by image summary which is chosen by using multi-modal anchors. According to the experimental results, compared with the state-of-art multi-modal summarization, our proposed MA-Sum can be optimized in terms of summarization consistency and completeness, so as to obtain the optimal multi-modal summarization metric.