Parameter Mapping Sonification Research Articles

Innovative communication of molecular evolution through sound: a biological sonification concert

BackgroundA major challenge of evolutionary biology is making underlying concepts accessible to wide audiences. One method for doing so is to utilise multi-media formats that have potential to engage and inform through entertainment. This pilot study outlines and discusses a sonification concert that integrated musical performance with a range of evolutionary concepts and ideas fundamental to an understanding of evolution, such as protein sequences. We aimed to showcase sound-art objects and live-coding performances created using sonification as a mechanism for presenting complex biological processes to both researcher and non-researchers. We sought to evaluate the effectiveness of this art-adjacent practice for public engagement with evolutionary biology research, and also to gather feedback to guide future events. Toward this end, we held a live concert showcasing biologically-based algorithmic music exploring links between evolutionary biology research, sound art, and musical performance. The event had three main acts: a generative audio-visual piece giving an artistic representation of SARS coronavirus based on a parameter-mapping sonification of protein sequence of the replicase polyprotein; a pre-recorded string ensemble demonstrating the effects of codon selection on translation speed using parameter-mapping sonification; and a live-coded music piece interactively sonifying protein structures.ResultsOur event attracted 90 attendees. We evaluated success using direct observation and written feedback forms with a 58% response rate: 95% of respondents stated they had enjoyed the event and 63% indicated they were inspired by it.ConclusionsPresenting the sonic outputs of sonification research in a concert format showed good potential for the pursuit of public engagement with evolutionary biology research, demonstrating the ability to engage curiosity and inspire an audience while also conveying scientific content alongside the nuanced and complex world of modern evolutionary biology.

Spatial Audio and Parameter Mapping Experiments for the Auditory Display of Coral Bleaching Data

In this article, data relating to Hawaii’s 2019 coral bleaching are auditorily displayed using parameter mapping sonification and ambisonics. Although primarily an explorative endeavor, this undertaking is conceptually rooted in ecological sound art and neutrally positioned on an established Cartesian framework known as Aesthetic Perspective Space. Through iterative design, different versions are implemented using sound surrogates of coral reefs’ natural soundscapes derived from either (a) real undersea recordings of reef environments or (b) modeling via means of sound synthesis. These audio surrogates correspond to data clusters aggregated by geographical location and, after being represented as sound sources on an ambisonic sphere, all contribute to a final sonic environment in which each cluster is progressively altered as the corresponding coral location undergoes bleaching. To assess both the perceived aesthetic placement of these experiments and their potential for public engagement in the discourse around climate change, evaluation studies are carried out. Results align with the aesthetic goals of the author, while consecutive versions manage to improve on critical fronts relating to climate awareness, providing further motivation toward more immersive implementations in future editions to come.

Parameter Mapping Sonification of Human Olfactory Thresholds.

An objective of chemical ecology is to understand the chemical diversity across and within species, as well as the bioactivity of chemical compounds. We previously studied defensive volatiles from phytophagous insects that were subjected to parameter mapping sonification. The created sounds contained information about the repellent bioactivity of the volatiles, such as the repellence from the volatiles themselves when tested against live predators. Here, we applied a similar sonification process to data about human olfactory thresholds. Randomized mapping conditions were used and a peak sound pressure, Lpeak, was calculated from each audio file. The results indicate that Lpeak values were significantly correlated with the olfactory threshold values (e.g., rS = 0.72, t = 10.19, p < 0.001, Spearman rank-order correlation; standardized olfactory thresholds of 100 volatiles). Furthermore, multiple linear regressions used the olfactory threshold as a dependent variable. The regressions revealed that the molecular weight, the number of carbon and oxygen atoms, as well as the functional groups aldehyde, acid, and (remaining) double bond were significant determinants of the bioactivity, while the functional groups ester, ketone, and alcohol were not. We conclude that the presented sonification methodology that converts chemicals into sound data allows for the study of their bioactivities by integrating compound characteristics that are easily accessible.

Using sound to understand protein sequence data: new sonification algorithms for protein sequences and multiple sequence alignments

BackgroundThe use of sound to represent sequence data—sonification—has great potential as an alternative and complement to visual representation, exploiting features of human psychoacoustic intuitions to convey nuance more effectively. We have created five parameter-mapping sonification algorithms that aim to improve knowledge discovery from protein sequences and small protein multiple sequence alignments. For two of these algorithms, we investigated their effectiveness at conveying information. To do this we focussed on subjective assessments of user experience. This entailed a focus group session and survey research by questionnaire of individuals engaged in bioinformatics research.ResultsFor single protein sequences, the success of our sonifications for conveying features was supported by both the survey and focus group findings. For protein multiple sequence alignments, there was limited evidence that the sonifications successfully conveyed information. Additional work is required to identify effective algorithms to render multiple sequence alignment sonification useful to researchers. Feedback from both our survey and focus groups suggests future directions for sonification of multiple alignments: animated visualisation indicating the column in the multiple alignment as the sonification progresses, user control of sequence navigation, and customisation of the sound parameters.ConclusionsSonification approaches undertaken in this work have shown some success in conveying information from protein sequence data. Feedback points out future directions to build on the sonification approaches outlined in this paper. The effectiveness assessment process implemented in this work proved useful, giving detailed feedback and key approaches for improvement based on end-user input. The uptake of similar user experience focussed effectiveness assessments could also help with other areas of bioinformatics, for example in visualisation.

ECG sonification to support the diagnosis and monitoring of myocardial infarction

This paper presents the design and evaluation of four sonification methods to support monitoring and diagnosis in Electrocardiography (ECG). In particular we focus on an ECG abnormality called ST-elevation which is an important indicator of a myocardial infarction. Since myocardial infarction represents a life-threatening condition it is of essential value to detect an ST-elevation as early as possible. As part of the evaluated sound designs, we propose two novel sonifications: (i) Polarity sonification, a continuous parameter-mapping sonification using a formant synthesizer and (ii) Stethoscope sonification, a combination of the ECG signal and a stethoscope recording. The other two designs, (iii) the water ambience sonification and the (iv) morph sonification, were presented in our previous work about ECG sonification (Aldana Blanco AL, Steffen G, Thomas H (2016) In: Proceedings of Interactive Sonification Workshop (ISon). Bielefeld, Germany). The study evaluates three components across the proposed sonifications (1) detection performance, meaning if participants are able to detect a transition from healthy to unhealthy states, (2) classification accuracy, that evaluates if participants can accurately classify the severity of the pathology, and (3) aesthetics and usability (pleasantness, informativeness and long-term listening). The study results show that the polarity design had the highest accuracy rates in the detection task whereas the stethoscope sonification obtained the better score in the classification assignment. Concerning aesthetics, the water ambience sonification was regarded as the most pleasant. Furthermore, we found a significant difference between sound/music experts and non-experts in terms of the error rates obtained in the detection task using the morph sonification and also in the classification task using the stethoscope sonification. Overall, the group of experts obtained lower error rates than the group of non-experts, which means that further training could improve accuracy rates and, particularly for designs that rely mainly on pitch variations, additional training is needed in the non-experts group.

Exploring maps by sounds: using parameter mapping sonification to make digital elevation models audible

ABSTRACTThis study empirically investigates the potential of auditory displays for spatial data exploration, as an additional means to broaden the accessibility and dissemination of geographic information for a diverse body of users. In a mixed factorial experiment, three parameter mapping sonification methods are empirically evaluated to interactively explore discrete and continuous digital elevation models by auditory means. Contrasting prior sonification research, this study’s unique empirical evidence suggests that participants can indeed successfully interpret sonified displays containing continuous spatial data. Specifically, the auditory variable pitch leads to significantly better response accuracy, compared to the sound variable duration. Background and training has a weak effect on data interpretation performance with the auditory display. The more immersive the experienced soundscape, the better participants can interpret the sonified terrain. These encouraging empirical results indeed suggest that interactive auditory displays might offer additional means to disseminate spatial information, and to increase the accessibility to spatial data, beyond the currently dominant visual paradigm.

Auditory display as feedback for a novel eye-tracking system for sterile operating room interaction.

The growing number of technical systems in the operating room has increased attention on developing touchless interaction methods for sterile conditions. However, touchless interaction paradigms lack the tactile feedback found in common input devices such as mice and keyboards. We propose a novel touchless eye-tracking interaction system with auditory display as a feedback method for completing typical operating room tasks. Auditory display provides feedback concerning the selected input into the eye-tracking system as well as a confirmation of the system response. An eye-tracking system with a novel auditory display using both earcons and parameter-mapping sonification was developed to allow touchless interaction for six typical scrub nurse tasks. An evaluation with novice participants compared auditory display with visual display with respect to reaction time and a series of subjective measures. When using auditory display to substitute for the lost tactile feedback during eye-tracking interaction, participants exhibit reduced reaction time compared to using visual-only display. In addition, the auditory feedback led to lower subjective workload and higher usefulness and system acceptance ratings. Due to the absence of tactile feedback for eye-tracking and other touchless interaction methods, auditory display is shown to be a useful and necessary addition to new interaction concepts for the sterile operating room, reducing reaction times while improving subjective measures, including usefulness, user satisfaction, and cognitive workload.

Towards an application of interactive sonification for the forces applied on the pedals during cycling on the Wattbike ergometer

Cyclists aim to improve pedaling technique to increase pedal force effectiveness. Therefore, biomechanical feedback about how the force is applied on the pedal stroke throughout the cycle is valuable. This information is typically presented visually on a display mounted on the handlebars. However, challenges associated with visual information presentation motivate movement researchers to display the information acoustically via parameter-mapping sonification that has shown potential for improving motor skills. This paper describes initial considerations for an application of interactive sonification for the forces applied on the pedals during cycling on the Wattbike ergometer. It was aimed to examine if the characteristics of the pedal stroke are represented in the sound to create congruency between sound, action and reaction. The primary focus is on describing the design followed by a brief summary of evaluation and practical effectiveness which was conducted on twenty-four master students (25.9±3.0years) and tested in a pilot study with four recreational cyclists (35.3±0.9years).The results of both requests indicate that sonification of the Wattbike data as an acoustic presentation of the force applied throughout the pedal stroke cycle is directly and intuitively understandable. Individual statements from testing revealed that participants became aware of characteristics within the cycling movement which they have not explicitly noticed before. The feeling for specific periods throughout the cycle was improved (pushing and pulling), and the differences in applying force on the legs (symmetry) became evident. Listening to the sound allowed cyclists to hear fluctuations in the forces applied on the pedals and thus try to adapt their muscle activation pattern. Thus, the approach presented here could serve as a helpful training tool for cyclists to optimize their cycling movement.

Interactive Spatial Sonification of Multidimensional Data for Composition and Auditory Display

This article presents a new approach to interactive spatial sonification of multidimensional data as a tool for spatial sound synthesis, for composing temporal–spatial musical materials, and as an auditory display for scientists to analyze multidimensional data sets in time and space. The approach applies parameter-mapping sonification and is currently implemented in an application called Cheddar, which was programmed in Max/MSP. Cheddar sonifies data in real time, where the user can modify a wide variety of temporal, spatial, and sonic parameters during the listening process, and thus more easily uncover patterns and processes in the data than when applying non-real-time, noninteractive techniques. The design draws on existing literature concerning perception and acoustics, and it applies the author's practical experience in acousmatic composition, spectromorphology, and sound semantics, while addressing accuracy, flexibility, and ease of use. Although previous sonification applications have addressed some degree of real-time control and spatialization, this approach integrates space and sound in an interactive framework. Spatial information is sonified in high-order 3-D ambisonics, where the user can interactively move the virtual listening position to reveal details easily missed from fixed or noninteractive spatial views. Sounds used as input to the sonification take advantage of the rich spectra and extramusical attributes of acoustic sources, which, although previously theorized, are investigated here in a practical context thoroughly tested alongside acoustic and psychoacoustic considerations. Furthermore, when using Cheddar, no specialized knowledge of programming, acoustics, or psychoacoustics is required. These approaches position Cheddar at the junction between science and art. With one application serving both disciplines, the patterns and processes of science are more fluently appropriated into music or sound art, and vice versa for scientific research, science public outreach, and education.

Towards a systematic approach to real-time sonification design for surface electromyography

Surface electromyography (sEMG) is a technique for measuring the electrical activity of muscles and is often used as a biofeedback tool. However, challenges associated with the typically visual display of sEMG data have motivated researchers to find non-visual ways of displaying sEMG data, and parameter-mapping sonification has been explored in order to present sEMG data acoustically. Parameter-mapping sonification is a technique that involves mapping values in a data set to acoustic properties of sound. Sonification of EMG data has shown potential for identifying musculoskeletal disease and improving athletic and exercise performance. However, many sonification designs to date have not been systematically evaluated and there have been few quantitative approaches to objective comparisons of sonification paradigms. In this study, we performed a quantitative comparison of different sonification designs in order to test our hypothesis that different sonification designs may be better suited to different tasks. Thirty-six participants (ages 18–31, 14 male) who volunteered to listen to the sEMG sonifications created for this study were asked to identify two different features of the data: muscle activation time and muscle exertion level. Their responses were analyzed in order to determine the effect of sonification design on listener performance. Results indicated that having the sonifications spatialized resulted in the best performance for both tasks. However, different sonification designs resulted in the best performance for the muscle activation time estimation task (Pitch and Loudness mapped redundantly) and the muscle exertion level estimation task (Pitch, Loudness, and Attack mapped redundantly). Further, for the time estimation task, the use of the Attack mapping appeared to reliably inhibit performance. These findings strongly suggest that sonification designs for sEMG need to be designed differently based on the task the user is performing.

Comparative study on the effect of Parameter Mapping Sonification on perceived instabilities, efficiency, and accuracy in real-time interactive exploration of noisy data streams

This paper presents a comparative study on common Parameter Mapping Sonifications (PMSons) designed to reduce the perception of instabilities during the real-time interactive exploration of noisy data streams. The objective of this study is to evaluate the capacity of these PMSons in reducing the perceived fluctuations while preserving both efficiency and accuracy of participants’ estimations during the exploration task. Based on the real application of beacon localization for rescue operations, an abstraction of the task was developed which simulates the basic concepts involved, i.e. the estimation of local maxima during the exploration of a 1D data topography. Three different PMSons were selected, based on pitch, pitch averaged over time, and tempo, evaluated for three different noise levels applied to the data. Subjects explored the auditory graph using a pen tablet to define their auditory viewpoint (i.e. position) on the topography. Evaluations were based on subjects’ ability to quickly appraise the underlying data topography along with aesthetic considerations regarding the impact of noise on the exploration task. Results showed that both tempo and pitch averaged PMSon reduced the perceived instabilities compared to pitch, while tempo preserved the response time of the sonification feedback.

Can Micro-Gestural Inflections Be Used to Improve the Soniculatory Effectiveness of Parameter Mapping Sonifications?

Parameter mapping sonification is the most widely used technique for representing multi-dimensional data in sound. However, it is known to be unreliable when used for detecting information in some types of data. This is generally thought to be the result of the co-dependency of the psychoacoustic dimensions used in the mapping.Positing its perceptual basis in a theory of embodied cognition, the most common approach to overcoming this limitation involves techniques that afford the interactive exploration of the data using gross body gestures. In some circumstances, such exploration is not possible and, even when it is, it may be neither necessary nor sufficient.This article explores some other possible reasons for the unreliability of parameter mapping sonification and, drawing from the experience of expressive musical performance, suggests that the problem lies not in the parametric approachper se, nor in the lack of interactivity, but in the extent to which the parameters employed contribute to coherent gestalts. A method for how this might be achieved that relies on the use of micro-gestural information is proposed. While this is speculative, the use of such gestural inflections is well known in music performance, is supported by findings in neuroscience and lends itself to empirical testing.

Order-related acoustic characterization of production data

The conductor of an orchestra is able to distinguish not only between different instruments, but even among dozens of string players performing on instruments with similar sound qualities. Trained human ear not only is capable to highly differentiate between pitches and colors of sound, but also to localize the position, where the sound is coming from. This chapter presents a parameter mapping sonification approach on production data, which is based on these human perceptual skills. Representatively for other logistic parameters, throughput times of orders are sonified and allocated in a sonic space. Additionally to auditory representations of the established resource and order oriented views in logistics, a third perspective is introduced, which displays the complete workflow of an order simultaneously as a multi-pitched spatial sound. Thus, causes and impacts of high throughput times in the data set example could be identified.