Poetic Expression Through Scenery: Sentimental Chinese Classical Poetry Generation from Images

Abstract

Most Chinese poetry generation methods only accept texts or user-specified words as input, which contradicts with the fact that ancient Chinese wrote poems inspired by visions, hearings and feelings. This paper proposes a method to generate sentimental Chinese classical poetry automatically from images based on convolutional neural networks and the language model. First, our method extracts visual information from the image and maps it to initial keywords by two parallel image classification models, then filters and extends these keywords to form a keywords set which is finally input into the poetry generation model to generate poems of different genres. A bi-directional generation algorithm and two fluency checkers are proposed to ensure the diversity and quality of generated poems, respectively. Besides, we constrain the range of optional keywords and define three sentiment-related keywords dictionary to avoid modern words that lead to incoherent content as well as ensure the emotional consistency with given images. Both human and automatic evaluation results demonstrate that our method can reach a better performance on quality and diversity of generated poems.