Multi-granularity Text Representation and Transformer-Based Fusion Method for Visual Question Answering

Abstract

The Visual Questions and Answers (VQA) task is to obtain the answer to a question by combining the image information with the image related textual question information. On the one hand, the representation of problem features in traditional models is mostly the extraction of coarse-grained information, resulting in the inability to effectively utilize the deep semantics of text information. On the other hand, the traditional fusion of text and image features mostly adopts a series fusion method to simply splice the feature vectors of different modalities. This simple fusion method cannot deal with the problem of information redundancy and conflict between different modal features, which ultimately affects the accuracy of Visual Question Answering. To solve the above problems, this paper proposes a text feature representation module (MGT) for multi-granularity information. It mainly uses the method of hierarchical expansion convolution to preserve text information from multiple levels and multiple angles to improve the utilization of text features. This paper also proposes a Transformer-based multimodal fusion module (BTMF). It mainly uses the internal correlation between the picture and text modal features to dynamically adjust the weight of each modal feature. Simultaneously learn different modal information while keeping the contextual information unchanged. This paper verifies the effectiveness of the method on the VQA2.0 dataset.