Sonification Approaches Research Articles

Interactive multimodal Integral Field Spectroscopy

Abstract Using sonification on scientific data analysis provides additional dimensions to visualization, potentially increasing researchers’ analytical capabilities and fostering inclusion and accessibility. This research explores the potential of multimodal Integral Field Spectroscopy (IFS) applied to galaxy analysis through the development and evaluation of a tool that complements the visualization of datacubes with sound. The proposed application, ViewCube, provides interactive visualizations and sonifications of spectral information across a two-dimensional field-of-view, and its architecture is designed to incorporate future sonification approaches. The first sonification implementation described in this article uses a deep learning module to generate binaural unsupervised auditory representations. The work includes a qualitative and quantitative user study based on an online questionnaire, aimed at both specialized and non-specialized participants, focusing on the case study of datacubes of galaxies from the Calar Alto Integral Field Spectroscopy Area (CALIFA) survey. Out of 67 participants who completed the questionnaire, 42 had the opportunity to test the application in person prior to filling out the online survey. 81 per cent of these 42 participants expressed the good interactive response of the tool, 79.1 per cent of the complete sample found the application ‘Useful’, and 58.2 per cent rated its aesthetics as ‘Good’. The quantitative results suggest that all participants were able to retrieve information from the sonifications, pointing to previous experience in the analysis of sound events as more helpful than previous knowledge of the data for the proposed tasks, and highlighting the importance of training and attention to detail for the understanding of complex auditory information.

A Framework for Multimodal Medical Image Interaction.

Medical doctors rely on images of the human anatomy, such as magnetic resonance imaging (MRI), to localize regions of interest in the patient during diagnosis and treatment. Despite advances in medical imaging technology, the information conveyance remains unimodal. This visual representation fails to capture the complexity of the real, multisensory interaction with human tissue. However, perceiving multimodal information about the patient's anatomy and disease in real-time is critical for the success of medical procedures and patient outcome. We introduce a Multimodal Medical Image Interaction (MMII) framework to allow medical experts a dynamic, audiovisual interaction with human tissue in three-dimensional space. In a virtual reality environment, the user receives physically informed audiovisual feedback to improve the spatial perception of anatomical structures. MMII uses a model-based sonification approach to generate sounds derived from the geometry and physical properties of tissue, thereby eliminating the need for hand-crafted sound design. Two user studies involving 34 general and nine clinical experts were conducted to evaluate the proposed interaction framework's learnability, usability, and accuracy. Our results showed excellent learnability of audiovisual correspondence as the rate of correct associations significantly improved (p < 0.001) over the course of the study. MMII resulted in superior brain tumor localization accuracy (p < 0.05) compared to conventional medical image interaction. Our findings substantiate the potential of this novel framework to enhance interaction with medical images, for example, during surgical procedures where immediate and precise feedback is needed.

Movement Sonification Techniques to Improve Balance in Parkinson's Disease: A Pilot Randomized Controlled Trial.

Movement sonification has been recently introduced into the field of neuromotor rehabilitation alongside Neurologic Music Therapy and music-based interventions. This study introduces the use of musical auditory cues encompassing the melodic-harmonic aspect of music. Nineteen patients with Parkinson's disease were randomly assigned to the experimental (n = 10) and control (n = 9) groups and underwent thrice-weekly sessions of the same gait training program, with or without sonification. Functional and motor parameters, as well as fatigue, quality of life, and the impact of intervention on patients' well-being, were assessed at baseline (PRE), the end of treatment (POST), and at follow-up (FU). Between-group differences were assessed for each outcome measure using linear mixed-effects models. The outcome measure was entered as the dependent variable, group and time as fixed effects, and time by group as the interaction effect. Mini BESTest and Dynamic Gait Index scores significantly improved in the experimental group (p = 0.01 and p = 0.03, respectively) from PRE to FU, demonstrating a significant impact of the sonification treatment on balance. No other significant differences were observed in the outcome measures. Larger sample sizes are needed to confirm the effectiveness of sonification approaches in Parkinson's disease, as well as in other neurological disorders.

Evaluating the catalytic effect of Fe@Fe2O3-modified granulated cork as an innovative heterogeneous catalyst in electro-fenton degradation of benzoquinone in different aqueous matrices

This study investigated the potential of a novel biomass-derived cork as a suitable catalyst after its modification with Fe@Fe2O3 for in-situ application in heterogeneous electro-Fenton (HEF) process for benzoquinone (BQ) elimination from water. No attempts on the application of modified granulated cork (GC) as a suspended heterogeneous catalyst in the HEF process for water treatment have been published yet. GC was modified by sonification approach in a FeCl3 + NaBH4 solution to reduce the ferric ions to metallic iron in order to obtain Fe@Fe2O3-modified GC (Fe@Fe2O3/GC). Results clearly demonstrated that this catalyst exhibited excellent electrocatalytic properties, such as a high conductivity as well as relatively high redox current and possessed several active sites for water depollution applications. Using Fe@Fe2O3/GC as catalyst in HEF, 100% of BQ removal was achieved in synthetic solutions by applying 33.3 mA cm−2 after 120 min. Different experimental conditions were tested to determine that best possible conditions can be as follow: 50 mmol L−1 Na2SO4 and 10 mg L−1 of Fe@Fe2O3/GC catalyst using Pt/carbon-PTFE air diffusion cell by applying 33.3 mA cm−2. Nevertheless, when Fe@Fe2O3/GC was used in the HEF approach to depollute real water matrices, no complete BQ concentration was removal achieved after 300 min of treatment, achieving between 80 and 95% of effectiveness.

Evaluating the efficacy of sonification for signal detection in univariate, evenly sampled light curves using astronify

ABSTRACT Sonification is the technique of representing data with sound, with potential applications in astronomy research for aiding discovery and accessibility. Several astronomy-focused sonification tools have been developed; however, efficacy testing is extremely limited. We performed testing of astronify, a prototype tool for sonification functionality within the Barbara A. Mikulski Archive for Space Telescopes. We created synthetic light curves containing zero, one, or two transit-like signals with a range of signal-to-noise ratios (SNRs = 3–100) and applied the default mapping of brightness to pitch. We performed remote testing, asking participants to count signals when presented with light curves as a sonification, visual plot, or combination of both. We obtained 192 responses, of which 118 self-classified as experts in astronomy and data analysis. For high SNRs (=30 and 100), experts and non-experts performed well with sonified data (85–100 per cent successful signal counting). At low SNRs (=3 and 5), both groups were consistent with guessing with sonifications. At medium SNRs (=7 and 10), experts performed no better than non-experts with sonifications but significantly better (factor of ∼2–3) with visuals. We infer that sonification training, like that experienced by experts for visual data inspection, will be important if this sonification method is to be useful for moderate SNR signal detection within astronomical archives and broader research. None the less, we show that even a very simple, and non-optimized, sonification approach allows users to identify high SNR signals. A more optimized approach, for which we present ideas, would likely yield higher success for lower SNR signals.

StereoPilot: A Wearable Target Location System for Blind and Visually Impaired Using Spatial Audio Rendering.

Vision loss severely impacts object recognition and spatial cognition for limited vision individuals. It is a challenge to compensate for this using other sensory modalities, such as touch or hearing. This paper introduces StereoPilot, a wearable target location system to facilitate the spatial cognition of BVI. Through wearing a head-mounted RGB-D camera, the 3D spatial information of the environment is measured and processed into navigation cues. Leveraging spatial audio rendering (SAR) technology, it allows the navigation cues to be transmitted in a type of 3D sound from which the sound orientation can be distinguished by the sound localization instincts in humans. Three haptic and auditory display strategies were compared with SAR through experiments with three BVI and four sighted subjects. Compared with mainstream speech instructional feedback, the experimental results of the Fitts' law test showed that SAR increases the information transfer rate (ITR) by a factor of three for spatial navigation, while the positioning error is reduced by 40%. Furthermore, SAR has a lower learning effect than other sonification approaches such as vOICe. In desktop manipulation experiments, StereoPilot was able to obtain precise localization of desktop objects while reducing the completion time of target grasping tasks in half as compared to the voice instruction method. In summary, StereoPilot provides an innovative wearable target location solution that swiftly and intuitively transmits environmental information to BVI individuals in the real world.

Using sound to understand protein sequence data: new sonification algorithms for protein sequences and multiple sequence alignments

BackgroundThe use of sound to represent sequence data—sonification—has great potential as an alternative and complement to visual representation, exploiting features of human psychoacoustic intuitions to convey nuance more effectively. We have created five parameter-mapping sonification algorithms that aim to improve knowledge discovery from protein sequences and small protein multiple sequence alignments. For two of these algorithms, we investigated their effectiveness at conveying information. To do this we focussed on subjective assessments of user experience. This entailed a focus group session and survey research by questionnaire of individuals engaged in bioinformatics research.ResultsFor single protein sequences, the success of our sonifications for conveying features was supported by both the survey and focus group findings. For protein multiple sequence alignments, there was limited evidence that the sonifications successfully conveyed information. Additional work is required to identify effective algorithms to render multiple sequence alignment sonification useful to researchers. Feedback from both our survey and focus groups suggests future directions for sonification of multiple alignments: animated visualisation indicating the column in the multiple alignment as the sonification progresses, user control of sequence navigation, and customisation of the sound parameters.ConclusionsSonification approaches undertaken in this work have shown some success in conveying information from protein sequence data. Feedback points out future directions to build on the sonification approaches outlined in this paper. The effectiveness assessment process implemented in this work proved useful, giving detailed feedback and key approaches for improvement based on end-user input. The uptake of similar user experience focussed effectiveness assessments could also help with other areas of bioinformatics, for example in visualisation.

Sound of databases

Sonifications map data to auditory dimensions and offer a new audible experience to their listeners. We propose a sonification of query processing paired with a corresponding visualization both integrated in a web application. In this demonstration we show that the sonification of different types of relational operators generates different sound patterns, which can be recognized and identified by listeners increasing their understanding of the operators' functionality and supports easy remembering of requirements like merge joins work on sorted input. Furthermore, new ways of analyzing query processing are possible with the sonification approach.

Hand rehabilitation with sonification techniques in the subacute stage of stroke

After a stroke event, most survivors suffer from arm paresis, poor motor control and other disabilities that make activities of daily living difficult, severely affecting quality of life and personal independence. This randomized controlled trial aimed at evaluating the efficacy of a music-based sonification approach on upper limbs motor functions, quality of life and pain perceived during rehabilitation. The study involved 65 subacute stroke individuals during inpatient rehabilitation allocated into 2 groups which underwent usual care dayweek) respectively of standard upper extremity motor rehabilitation or upper extremity treatment with sonification techniques. The Fugl-Meyer Upper Extremity Scale, Box and Block Test and the Modified Ashworth Scale were used to perform motor assessment and the McGill Quality of Life-it and the Numerical Pain Rating Scale to assess quality of life and pain. The assessment was performed at baseline, after 2 weeks, at the end of treatment and at follow-up (1 month after the end of treatment). Total scores of the Fugl-Meyer Upper Extremity Scale (primary outcome measure) and hand and wrist sub scores, manual dexterity scores of the affected and unaffected limb in the Box and Block Test, pain scores of the Numerical Pain Rating Scale (secondary outcomes measures) significantly improved in the sonification group compared to the standard of care group (time*group interaction < 0.05). Our findings suggest that music-based sonification sessions can be considered an effective standardized intervention for the upper limb in subacute stroke rehabilitation.

Manipulating music to communicate automation reliability in conditionally automated driving: A driving simulator study

Abstract With increasing vehicle automation, drivers are likely to engage in non-driving related tasks. However, before fully autonomous vehicles can be achieved, drivers remain the fallback when automation limits are reached and/or in unexpected situations. Transition from automated to manual control could be particularly difficult for drivers who are “out-of-the-loop”. To support safe and smooth control transition, we adopted the blended sonification approach in manipulating background music to convey information about the reliability level of the automation. A driving simulator study consisting of 36 participants was conducted to investigate the effectiveness of sonification feedback—three levels of system reliability was mapped to background music pitch levels—on takeover events. Participants were assigned to one of three groups that received continuous sonification feedback, intermittent sonification feedback, or no feedback about automation reliability. The proposed blended music sonification was found effective in increasing monitoring behavior and decreasing visual response times to takeover requests during automated driving. Steering control in a takeover scenario that required substantial amount of maneuvering also benefitted from having sonification feedback. There was some evidence that sonification feedback provided continuously was more effective than when it was provided intermittently, possibly related to observed differences in the associated glance patterns. In conclusion, blended sonification may be an effective approach in helping drivers stay on the loop during automated driving, although the study was limited to the choice of music examined. Our future work will investigate how best to combine automation reliability information and traffic situational information to better support situation awareness and facilitate task-switching in automated driving.

Sonic Bodies

This article examines the use of sonification, and movement sonification, as a tool for engagement with artworks in galleries and museums. It focuses on the relationship between artwork and sighted, partially sighted and non-sighted visitors and the way sonification, by offering an audio-centred mode of engagement, complicates the aesthetic experience. By drawing on Bruno Latour’s discussion on hybrids (1993) as well as with reference to examples of the use of sonification in gallery spaces, the article argues that rather than assuming the artwork to be a finished and stable object, movement sonification approaches the artwork as a blueprint of proximal, conceptual and material relationships which are animated by a moving-listening subject. As such, the article concludes, movement sonification can have radical implications for the way artworks are conceived as well as the way the aesthetic experience is configured.

Music of brain and music on brain: a novel EEG sonification approach.

Can we hear the sound of our brain? Is there any technique which can enable us to hear the neuro-electrical impulses originating from the different lobes of brain? The answer to all these questions is YES. In this paper we present a novel method with which we can sonify the electroencephalogram (EEG) data recorded in"control" state as well as under the influence of a simple acoustical stimuli-a tanpura drone. The tanpura has a very simple construction yet the tanpura drone exhibitsvery complex acoustic features, which is generally used for creation of an ambience during a musical performance. Hence, for this pilot project we chose to study the nonlinear correlations between musical stimulus (tanpura drone as well as music clips) and sonified EEG data. Till date, there have been no study which deals with the direct correlation between a bio-signal and its acoustic counterpart and also tries to see how that correlation varies under the influence of different types of stimuli. This study tries to bridge this gap and looks for a direct correlation between music signal and EEG data using a robust mathematical microscope called Multifractal Detrended Cross Correlation Analysis (MFDXA). For this, we took EEG data of 10 participants in 2min "control condition" (i.e. with white noise) and in 2min 'tanpura drone' (musical stimulus) listening condition. The same experimental paradigm was repeated for two emotional music, "Chayanat" and "Darbari Kanada". These are well known Hindustani classical ragas which conventionally portray contrast emotional attributes, also verified from human response data. Next, the EEG signals from different electrodes were sonified and MFDXA technique was used to assess the degree of correlation (or the cross correlation coefficient γx) between the EEG signals and the music clips. The variation of γx for different lobes of brain during the course of the experiment provides interesting new information regarding the extraordinary ability of music stimuli to engage several areas of the brain significantly unlike anyother stimuli (which engages specific domains only).

On the Auditory-Proprioception Substitution Hypothesis: Movement Sonification in Two Deafferented Subjects Learning to Write New Characters.

The aim of this study was to evaluate the compensatory effects of real-time auditory feedback on two proprioceptively deafferented subjects. The real-time auditory feedback was based on a movement sonification approach, consisting of translating some movement variables into synthetic sounds to make them audible. The two deafferented subjects and 16 age-matched control participants were asked to learn four new characters. The characters were learned under two different conditions, one without sonification and one with sonification, respecting a within-subject protocol. The results revealed that characters learned with sonification were reproduced more quickly and more fluently than characters learned without and that the effects of sonification were larger in deafferented than in control subjects. Secondly, whereas control subjects were able to learn the characters without sounds the deafferented subjects were able to learn them only when they were trained with sonification. Thirdly, although the improvement was still present in controls, the performance of deafferented subjects came back to the pre-test level 2 h after the training with sounds. Finally, the two deafferented subjects performed differently from each other, highlighting the importance of studying at least two subjects to better understand the loss of proprioception and its impact on motor control and learning. To conclude, movement sonification may compensate for a lack of proprioception, supporting the auditory-proprioception substitution hypothesis. However, sonification would act as a “sensory prosthesis” helping deafferented subjects to better feel their movements, without permanently modifying their motor performance once the prosthesis is removed. Potential clinical applications for motor rehabilitation are numerous: people with a limb prosthesis, with a stroke, or with some peripheral nerve injury may potentially be interested.

Sonification: Review of Auditory Display Solutions in Electronic Travel Aids for the Blind

AbstractSonification is defined as presentation of information by means of non-speech audio. In assistive technologies for the blind, sonification is most often used in electronic travel aids (ETAs) - devices which aid in independent mobility through obstacle detection or help in orientation and navigation. The presented review contains an authored classification of various sonification schemes implemented in the most widely known ETAs. The review covers both those commercially available and those in various stages of research, according to the input used, level of signal processing algorithm used and sonification methods. Additionally, a sonification approach developed in the Naviton project is presented. The prototype utilizes stereovision scene reconstruction, obstacle and surface segmentation and spatial HRTF filtered audio with discrete musical sounds and was successfully tested in a pilot study with blind volunteers in a controlled environment, allowing to localize and navigate around obstacles.

Design and evaluation of the effectiveness of a sonification technique for real time heart-rate data

This article is motivated by the question “Can a sonification system that provides continuous auditory heart rate feedback help stabilize an athlete’s heart rate at a given target heart rate while exercising?” The sonification system uses a Polar H7 heart rate sensor to measure the heart rate of the athlete and an iOS device for its processing and display. We implemented several sonification approaches, of which two were tested in both a unimodal and an audiovisual context in comparison to a purely visual feedback and to not having any feedback. The system’s objective performance and multiple subjective usability aspects were evaluated in an experiment with 16 subjects. The experiment has to be considered a pilot study because the exercising conditions were artificial. The subjects were exercising on an indoor cycle and could focus their visual sense on the visual display all the time. It was found that all of the feedback methods could convey information to the athlete and were therefore clearly superior to not having any feedback. The failure of showing a supremacy of the multimodal methods over the purely visual one can be reasoned by the fact that the testing conditions were artificial and could therefore not show the advantages of auditory/audiovisual feedback due to limited bandwidth of the visual channel. The conclusions we make about the design and evaluation of such sonification systems can be considered a useful starting point for further work in this field.

Sonifying the location of an object: A comparison of three methods

Auditory displays are promising for informing operators about hazards or objects in the environment. However, it remains to be investigated how to map distance information to a sound dimension. In this research, three sonification approaches were tested: Beep Repetition Rate (BRR) in which beep time and inter-beep time were a linear function of distance, Sound Intensity (SI) in which the digital sound volume was a linear function of distance, and Sound Fundamental Frequency (SFF) in which the sound frequency was a linear function of distance. Participants (N = 29) were presented with a sound by means of headphones and subsequently clicked on the screen to estimate the distance to the object with respect to the bottom of the screen (Experiment 1), or the distance and azimuth angle to the object (Experiment 2). The azimuth angle in Experiment 2 was sonified by the volume difference between the left and right ears. In an additional Experiment 3, reaction times to directional audio-visual feedback were compared with directional visual feedback. Participants performed three sessions (BRR, SI, SFF) in Experiments 1 and 2 and two sessions (visual, audio-visual) in Experiment 3, 10 trials per session. After each trial, participants received knowledge-of-results feedback. The results showed that the three proposed methods yielded an overall similar mean absolute distance error, but in Experiment 2 the error for BRR was significantly smaller than for SI. The mean absolute distance errors were significantly greater in Experiment 2 than in Experiment 1. In Experiment 3, there was no statistically significant difference in reaction time between the visual and audio-visual conditions. The results are interpreted in light of the Weber-Fechner law, and suggest that humans have the ability to accurately interpret artificial sounds on an artificial distance scale.

Flexible touch sensor for evaluating geometric properties of virtual shapes through sound

This paper describes the design and implementation of a system for rendering virtual shape through vision, haptic and sound. The system consists of a haptic strip that physically renders virtual curves. A flexible capacitive touch sensor (FCTS) is integrated with the haptic strip, and allows the system to track the position of the user's fingers on the strip. According to the position, the system renders curve properties such as curve shape, inflexion points and curvature through sound metaphors. The goal of this sonification approach is to strengthen the user's understanding of the shape of a virtual prototype, and to inform the user about geometrical attributes that could otherwise remain unseen. Such unseen attributes may either be a result of limitations in the visual and haptic display hardware or a result of limitations in human perception.

Hearing the Buildings: Smart Monitoring through Advanced Sonification Approaches

Risk prevention of historical buildings through sensor network monitoring represents a challenging and together promising task towards an optimized buildings’ protection plan. Due to the significant character that historical buildings have within the history and identity of local and consequently national communities, the monitoring approach above is nowadays considered a fundamental component of cultural preservation. Modern trends on sensor networks nowadays allow the installation of wireless monitoring equipment, able to collect a large volume and variety of data that can be further processed in order to support prevention modelling techniques and strategies. Despite the valuable post-processing outcome of the above modelling techniques, in several real-time monitoring cases, an aspect that may significant impact the data analysis accuracy from the monitoring engineer is to create an audiovisual representation of the gathered data that are easily perceived and directly associated to well-known environmental conditions. Towards this aim, in this work an audio-based representation of the collected data is introduced that achieves high monitoring performance in terms of the perceived nature of the building vibrations themselves. The proposed audio-based representation technique is based on a number of advanced sonification methods, combined with specific human-hearing simulation mechanisms that allow an effective increase of the represented information volume in real-time. A number of subjective tests that were performed demonstrate a significant improvement on the way that the monitoring personnel can efficiently perceive the origin and / or nature of the acquired vibrations, rendering the proposed technique a strong supplement towards efficient historical building risk prevention.

SEISMOTOOL: An Easy Way to See, Listen, and Analyze Seismograms

Online Material: Computer code (.exe) for processing seismic records; example seismograms. Using sonification for data analysis is an intensively developing area for scientists working in various fields from astronomy and seismology to psychology (Zwicker and Fastl, 1999; Rubinstein et al. , 2007; Last and Gorelik, 2008; Hermann, 2009; Feder, 2012; www.sonification.de; last accessed May 2014). According to popular definition, “Sonification is the use of non‐speech audio to convey information or perceptualize data. Auditory perception has advantages in temporal, amplitude, and frequency resolution that open possibilities as an alternative or complement to visualization techniques” (Wikipedia). The sonification approach is promising in filtering large amounts of data, monitoring data streams, and communication (Feder, 2012). Our ears can serve as the processor of time series not only in popularization applications, but also as a serious scientific tool, because it allows detecting small changes in dynamical patterns of signal. The simplest way to listen to seismic records is to compress record time by changing the sampling rate. Examples of this technique were presented in the 2012 issues of Seismological Research Letters (Kilb et al. , 2012; Peng et al. , 2012); authors suggested an automated method that allows effective communication with audiences having different backgrounds. In this paper, we present a different nonautomated approach using the popular Sony SoundForge package that employs more advanced features, such as iteratively applying different filters, speed‐up factors, selecting some parts of a record or gluing different records before sonification as well as to analyze long data files. ![Graphic][1] This new tool, which we call SEISMOTOOL (available in the electronic supplement to this paper), can be used for both data analysis and data sonification and is a convenient way to watch, listen, process, and even analyze seismic records in an easy way. The advantage of the … [1]: /embed/inline-graphic-1.gif

Order-related acoustic characterization of production data

The conductor of an orchestra is able to distinguish not only between different instruments, but even among dozens of string players performing on instruments with similar sound qualities. Trained human ear not only is capable to highly differentiate between pitches and colors of sound, but also to localize the position, where the sound is coming from. This chapter presents a parameter mapping sonification approach on production data, which is based on these human perceptual skills. Representatively for other logistic parameters, throughput times of orders are sonified and allocated in a sonic space. Additionally to auditory representations of the established resource and order oriented views in logistics, a third perspective is introduced, which displays the complete workflow of an order simultaneously as a multi-pitched spatial sound. Thus, causes and impacts of high throughput times in the data set example could be identified.