AI Color Organ: Piano Music Visualization using Onset Detection and HistoGAN

Abstract

The music visualization algorithm described in this study allows users to construct piano audio files using imported image files. This paper contributes to previous studies and designs of sonification by highlighting the effectiveness of utilizing onset detection in creating intuitive sonic changes. The audio-visual correspondences employed in this study could be expanded to many other syntheses and sample manipulation techniques. Translating visual information into sonic changes could yield many creative applications in music production, as it offers musicians a simultaneously optical and auditory production experience. This approach to audio manipulation also increases the unpredictability of the sound output, which could be appealing to experimental musicians seeking to control sounds with the visual structure of artworks that they enjoy, as opposed to precise parameters. It is looking forward to seeing creative implementations of the techniques in audio-visual artworks, music production tools, and interactive multimedia systems. These results shed light on guiding further exploration of AI composing.