MTCA: A Multimodal Summarization Model Based on Two-Stream Cross Attention

Abstract

The combination of vision and natural language patterns has become an important topic in the field of computer vision and natural language processing. With the rapid growth of multimodal information, multimodal summarization has received unprecedented attention. In this paper, we propose MTCA which composed of a pre-trained feature extractor, a text encoder, an image encoder, a two-stream cross attention fusion module and a summary decoder to complete the multimodal summarization task. This framework integrates three sub tasks: extractive summarization, abstractive summarization and image selection. At the same time, we introduce a residual network to alleviate the problem of modality-bias. Experiments show that the model is better than the baseline model and the performance is better than the text summarization method which ignores visual modality.