It’s about Time: Adopting Theoretical Constructs from Visualization for Sonification

Abstract

Both sonification and visualization convey information about data by effectively using our human perceptual system, but their ways to transform the data could not be more different. The sonification community has demanded a holistic perspective on data representation, including audio-visual analysis, several times during the past 30 years. A design theory of audio-visual analysis could be a first step in this direction. An indispensable foundation for this undertaking is a terminology that describes the combined design space. To build a bridge between the domains, we adopt two of the established theoretical constructs from visualization theory for the field of sonification. The two constructs are the spatial substrate and the visual mark. In our model, we choose time to be the temporal substrate of sonification. Auditory marks are then positioned in time, such as visual marks are positioned in space. The proposed definitions allow discussing visualization and sonification designs as well as multi-modal designs based on a common terminology. While the identified terminology can support audio-visual analytics research, it also provides a new perspective on sonification theory itself.