How can generative adversarial networks impact computer generated art? Insights from poetry to melody conversion

Abstract

Recent advances in deep learning and generative adversarial networks (GANs), in particular, has enabled interesting applications including photorealistic image generation, image translation, and automatic caption generation. This has opened up possibilities for many cross-domain applications in computer generated arts and literature. Although there are existing software-based approaches for generating musical accompaniment of a given poetry, there are no existing implementation using GANs. This work proposes a novel poetry to melody generation conditioned on poem emotion using GANs. A dataset containing pairs of poetry and melody based on three emotion categories is introduced. Furthermore, various GAN architectures including SpecGAN and WaveGAN were explored for automatic melody synthesis for a given class of poetry. Conditional SpecGAN produced the best melodies according to quantitative metrics. Melodies produced by SpecGAN were evaluated by volunteers who deemed the quality to be above average.