The effectiveness of T5, GPT-2, and BERT on text-to-image generation task

Abstract

Text-to-image (T2I) generation, which involves synthesizing an image from a textual description, has emerged as a popular research topic in computer vision. Meanwhile, transformer-based models, such as BERT, GPT-2, and T5, have demonstrated promising results in various natural language processing tasks, including text generation and translation. However, their application to T2I generation remains largely unexplored. Therefore, a comparative study investigating the performance of BERT, GPT-2, and T5 in T2I generation is of significant importance. Such a study would shed light on the strengths and weaknesses of each model, facilitating the identification of the most suitable approach for this task. In this paper, we propose three architectures to conduct a comparative study of T5, GPT-2, and BERT in T2I generation tasks. We fine-tune these models to generate text vectors and then transform the textual information into images using the affine transformation into the DF-GAN generator. Subsequently, we evaluate the quality, diversity, and the models’ ability to recognize the words. Our experiments on the challenging CUB and Oxford-102 flower datasets demonstrate that T5 exhibits promising potential for T2I generation. It has the capability to generate visually appealing and semantically coherent images from textual descriptions.