Real-time concatenative synthesis for networked musical interactions

Abstract

The recent proliferation of Networked Music Performances has led to the investigation of low-latency, low-bitrate musical encoding schemes, including audio codecs and control protocols that specifically address the requirements of live musical interactions across the Internet. This work presents an alternative perspective inspired by the “synthesis by analysis” approach, tightly constrained in terms of processing latencies and rendering quality. The entire process is fully automated and involves an offline processing phase (that takes place prior to performance) and an online real-time analysis-synthesis phase. The offline phase involves processing a solo recording of each musician’s part so as to acquire (a) audio segments corresponding to each note in the performance, and (b) a trained Hidden Markov Model to be later used for online analysis. During live performance, the online analysis process encodes the position of the performance on a music score and re-synthesizes the audio waveform by concatenating the audio segments of the offline phase. Although the synthesized waveform originates from an offline solo recording, it is synchronized to the live performance at note level, so as to allow for rendering a wide range of musical tempi as well as their expressive variations. The paper presents the complete methodology and reports on implementation details and preliminary evaluation results.