Sonification of Complex Spectral Structures.

Mattias Sköld,Roberto Bresin

doi:10.3389/fnins.2022.832265

Mattias Sköld, Roberto Bresin

Open Access

https://doi.org/10.3389/fnins.2022.832265

Copy DOI

Abstract

In this article, we present our work on the sonification of notated complex spectral structures. It is part of a larger research project about the design of a new notation system for representing sound-based musical structures. Complex spectral structures are notated with special symbols in the scores, which can be digitally rendered so that the user can hear key aspects of what has been notated. This hearing of the notated data is significantly different from reading the same data, and reveals the complexity hidden in its simplified notation. The digitally played score is not the music itself but can provide essential information about the music in ways that can only be obtained in sounding form. The playback needs to be designed so that the user can make relevant sonic readings of the sonified data. The sound notation system used here is an adaptation of Thoresen and Hedman's spectromorphological analysis notation. Symbols originally developed by Lasse Thoresen from Pierre Schaeffer's typo-morphology have in this system been adapted to display measurable spectral features of timbrel structure for the composition and transcription of sound-based musical structures. Spectrum category symbols are placed over a spectral grand-staff that combines indications of pitch and frequency values for the combined display of music related to pitch-based and spectral values. Spectral features of a musical structure such as spectral width and density are represented as graphical symbols and sonically rendered. In perceptual experiments we have verified that users can identify spectral notation parameters based on their sonification. This confirms the main principle of sonification that is that the data/dimensions relations in one domain, in our case notated representation of spectral features, are transformed in perceived relations in the audio domain, and back.

Highlights

The work and ideas presented here relate to an ongoing research project concerning the development of sound notation from Thoresen’s spectromorphological analysis symbols (Thoresen and Hedman, 2007; Sköld, 2020)
We are presenting a framework for working with playback of sound notation as sonification of notated data, using a data format similar to MIDI acting as a bridge between notation and sound production
We are showing through listening tests that the notated data for four timbre-related notation parameters can be identified based on their sonification

Summary

Introduction

The work and ideas presented here relate to an ongoing research project concerning the development of sound notation from Thoresen’s spectromorphological analysis symbols (Thoresen and Hedman, 2007; Sköld, 2020). During the 1980s, Mark of the Unicorn were influential in music notation on computers with their Composer’s Mosaic and Performer (Belkin, 1993) before the introduction of Finale in 1988 and Sibelius five years later. The latter two would dominate the market for over a decade (Strawn and Shockley, 2014) and are still widely used. Digital playback of notation can be misleading for young composers orchestrating their work (Deutsch, 2016), and composition teachers warn their students not to rely on it in their work We think it can be valuable if one appreciates it for what is—the sonification of the notation data

Methods

Results

Discussion

Conclusion