Poetry4painting: Diversified poetry generation for large-size ancient paintings based on data augmentation

Abstract

Chinese painting poetry is an extraordinary art form, which not only describes the painting contexts but also grasps the sentiment of the painters. In this paper, we propose an automatic poetry generation method Poetry4painting, which enhances the poetry diversity for large-size ancient paintings. The basic framework is based on multiple modern sentences, that are first captioned from the ancient painting and then used to generate a poem using CNN and LSTM. To solve the repeatability issue of this framework, four kinds of data augmentation are employed during online processing, including quantity, shape, surrounding, and object augmentation. In offline training, data augmentation is also used to create an image caption dataset with over 1500 painting images and 7500 captions. Through ablation studies, evaluations of poetry qualities and diversities, and comparisons with other methods, we demonstrate the validity of the proposed method.