Switching Text-Based Image Encoders for Captioning Images With Text

Abstract

Visual understanding, such as image caption generation, has received extensive attention. Describing images with textual information is one way to help people achieve barrier-free visibility. This study focuses on the text-based image captioning (TextCaps) task. The TextCaps task is more complex than the traditional image captioning task because it depends on optical character recognition (OCR) and the textual information that appears in the image. It also requires consideration of the relationship between recognized objects and OCR’s linguistic part in the image. In this study, we propose maximizing the use of multiple modalities in an image to improve TextCaps performance. We enrich the image and OCR linguistic features using pre-trained Contrastive Language-Image Pre-training (CLIP) models. We then introduce using two additional attention models in a transformer architecture to strengthen the representation of the image modality. The experimental results demonstrate that our proposed method, which introduces a multimodal transformer with four image-related modalities, outperforms existing methods for the TextCaps dataset.