Real-time computational audio scene analysis, transformation, and synthesis for active music listening.

Abstract

Spectral models attempt to parametrize sound at the basilar membrane of the ear, thus permitting transformations closely linked to the perception. However, for high-quality real-time applications, these models require methods for a precise analysis and an efficient synthesis. When dealing with musical sound, that is, a polyphonic mix of non-stationary complex sounds, the main challenge is to extract these different sounds present in the musical mix. This can be done using (psycho) acoustical knowledge about the sound sources (computational auditory scene analysis approach). However, the quality is often not sufficient. When access to the compositional process is given, another option is to use some bits of this ground truth as additional information (informed analysis approach). This more precise analysis allows deeper transformations, and using psycho-acoustical considerations, efficient data structures, and algorithms, it is possible to re-synthesize the sounds from the model parameters in real time and with a high quality. This opens up new impressive applications, such as “active listening”, enabling the listener to interact with the sound while it is played. The musical parameters (e.g., loudness or spatial location) of the sound sources present in the musical mix can now be changed interactively.