Speech-based Mark for Data Sonification

The 2019 WSGC Elijah High-Altitude Balloon Payload Fellowship focused on 4 high-altitude phenomena: Modular Payload Design, Applications of Air Turbulence (power generation and visualization), Electronic Behavior at Altitude, and Sonification of Atmospheric Data. Modular payload design focused on user-accessibility by creating friction-reducing rings in-between insulation and the instrumentation capsule. Height-adjustable, modular shelving was also constructed. Turbulence and Electronics project both suffered data loss during flight. However, post-flight lab analysis showed the power generation apparatus produced 96J – 120J and turbulence visualization’s potential to assist in calculating Eddie dissipation rates. Additionally, Electronic Behavior observed corona discharges across large electrical gaps near-vacuum pressures. Sonification of Data used computer algorithms to transcribe data relationships into music. The intent was to allow users to perceive data relationships and patterns aurally.

The COVID-19 pandemic has significantly impacted nursing students in urban settings. Understanding stress and coping in this population is essential for developing effective interventions for future pandemics and post-pandemic nursing education. This study assesses the interactions between demographic variables and stress responses, demonstrating these relationships through data sonification. A cross-sectional analytical study was conducted at an urban nursing college, with 195 participants completing modified versions of the Perceived Stress Scale, Physio-Psycho-Social Response Scale, and Coping Behavior Inventory. Structural equation modeling and data sonification were employed for analysis. Nursing students reported moderate stress from academic demands and clinical responsibilities, utilizing coping strategies such as problem-solving and optimism. Structural equation modeling indicated a significant negative relationship between coping skills and physio-psycho-social stress (β = -0.301, p < .0001) and a negative association between age and perceived stress (β = -0.160, p = 0.035). Data sonification revealed synchronization between perceived stress and stress responses, with coping efforts peaking slightly after stress levels. While students employ effective coping strategies, targeted interventions are needed to enhance stress management, especially for younger students. Stress and coping among nursing students in urban settings are complex, with data sonification providing deeper insights. J. Med. Invest. 72 : 102-116, February, 2025.

Starting from the observation that symbolic language is not the only channel for human communication, this chapter examines ‘data sonification’, a means of understanding, reasoning about, and communicating meaning that extends beyond that which can be conveyed by symbolic language alone. Data sonification is a mapping from data generated by a model, captured in an experiment, or otherwise gathered through observation to one or more parameters of an audio signal or sound synthesis model for the purpose of better understanding, communicating, or reasoning about the original model, experiment, or system. Although data sonification shares techniques and materials with data-driven music, it is in the interests of the practitioners of both music composition and data sonification to maintain a distinction between the two fields.

Data sonification is a means of representing data through sound and has been utilized in a variety of applications. Crime against women has been a rising concern in India. We explore the potential of data sonification to provide an immersive engagement with sensitive data on crime against women in Indian states. The data for nine crime categories covering thirty-five Indian states over a period of twelve years is acquired from National records. Sonification techniques of parameter mapping and auditory icons are adopted: sound parameters such as frequencies, amplitudes and timbres are incorporated to represent the crime data, and audio sounds of women screams are employed as auditory icons to emphasize the traumatic experience. Higher crime rates are assigned higher frequencies, harsher scream textures and larger amplitudes. A user-friendly interface is developed with multiple options for sequential and comparative data sonification. Through the interface, a user can evaluate and compare the extent of crime against women in different states, years or crime categories. Sound spatialization is used to immerse the listener in the sound and further intensify the sonification experience. To assess and validate effectiveness, a user study on twenty participants is conducted with feedback obtained through questionnaires. The responses indicate that the participants could comprehend trends in the data easily and found the data sonification experience impactful. Sonification may therefore prove to be a valuable tool for data representation in fields related to social and human studies.

: Rapidly growing data rates and operational complexity require new approaches to providing situation awareness to military analysts, planners, and decision makers. Representation of complex information through sound, or Data Sonification (DS), is one such promising approach that remains relatively unexploited in both military and non-military information systems. The goal of the Phase I effort was to investigate and demonstrate the feasibility of a new approach to DS applications, including: (1) Methods for identifying potentially worthwhile sonification display functions in the work environment, (2) Analytical methods for decomposing and characterizing DS design problems in target systems, (3) Guidance and principles for generating, implementing and evaluating DS options, including a DS grammar for structuring symbolic aural representations, (4) Computer-based tools for assisting and improving design and evaluation tasks, and (5) Information technology appropriate to the representational demands of DS applications.

Data is predominantly conveyed and analysed in a visual manner. Data sonification provides an alternative approach to data analysis, has a broader application (e.g., peripheral monitoring) and offers accessibility to an alternative cohort (e.g., the visually impaired). While there are countless data visualization tools available, producing data sonifications typically requires in-depth knowledge of audio software and/or computer programming. Research was undertaken into existing data sonification tools, and subsequently a Data Sonification Toolkit was developed using Abelton Live and Max for Live. The Toolkit was developed in particular as an alternative means of data analysis for use by multiple disciplines across a built environment firm. The key aims for the toolkit were that (1) it should be user-friendly and accessible to people without an in-depth knowledge of either Ableton or Max, (2) the sonifications produced be accurate and true representations of input data sets, and (3) the Toolkit should have the capability and flexibility to be expanded and customized by those with the expertise to do so. The Toolkit was used, in collaboration with astrophysicist Chris Harrison, to develop sonifications of astronomical phenomena for the visually impaired. These sonifications were presented at the British Science Festival in 2019.

The spectro-temporal structure of a sound determines its timbre, and carries musically interesting information such as instrument type and performance expressions. Using timbre in data sonification can be viewed as an inverse transform of this process: Expressing data with timbre is equivalent to designing the spectro-temporal structure of a sound. Taking that into account, timbre is most effectively used in sonification by projecting time-series data onto the spectro-temporal structure of a sound. The temporal structure of the data often differs from the archetypal spectro-temporal structure of traditional instrumental sounds. But this discrepancy contributes to novel musical expression, based on novel timbre design. From this perspective, some sonification works are presented, such as ones produced by sonification of dynamic motion of genetically-modified worms and dynamic transitions of brain-wave data. Based on these examples, methods for effective and expressive use of timbre in data sonification will be presented.

Much discussion surrounding data sonification for musical and artistic purposes focuses on seeming contradictions that arise from the ways in which this practice differs from that of data sonification as a scientific tool. Over the past 30 years, this debate has become a rabbit hole of questions and arguments regarding the nature of music/sound-art and data sonification, and of their relationships with one another. In the following article, the authors identify three areas/classifications of artistic sonification, using a “strata” metaphor with the hope of bringing clarity to this discussion, as well as enabling reflection on the nature of science-art collaborations using this approach.

The “Accessible Oceans” pilot project aims to inclusively design auditory displays that support perception and understanding of ocean data in informal learning environments (ILEs). The project’s multi-disciplinary team includes expertise from all related fields — ocean scientists, dataset experts, a sound designer with specialization in data sonification, and a learning sciences researcher. In addition, the PI is blind and provides a crucial perspective in our research. We describe the sound design of informative sonifications and respective auditory displays based on iterative design with user input at each stage, including from blind and low-vision (BLV) students, their teachers, and subject-matter experts. We discuss the importance of framing data sonifications through an auditory presentation of contextual information. We also report on our latest auditory display evaluation using Auditory Interface UX Scale (BUZZ) surveys at three ILE test sites. These responses further affirm our auditory display design developments. We include access to the auditory displays media and lessons learned over the course of this multi-year NSF-funded Advancing Informal Stem Learning (AISL) grant https://accessibleoceans.whoi.edu/

Ocean currents play an important role in regulating the Earth’s climate by redistributing ocean heat around the planet. One important climate phenomenon that includes several large ocean currents is the Atlantic Meridional Overturning Circulation, which is the process of warm surface currents in the North Atlantic Ocean moving to the Arctic to ultimately form cold currents near the seafloor. This process relieves the tropics of excess ocean heat, which helps reduce the intensity of Atlantic hurricanes. Scientists are concerned that the Atlantic Meridional Overturning Circulation may slow, or even collapse, because of anthropogenic climate change. In this activity, students will use selected portions of a four-year dataset of observations of the Atlantic Meridional Overturning Circulation to learn about the importance of long-term consistent sampling and data biases. The students will plot their data, calculate the slope between the first and last points, listen to sonified versions of the data, and discuss trends in the data. This activity will also give students the opportunity to build a mooring with craft supplies representing the anchor, cable, scientific instruments, and flotation. There are modifications for the visually impaired via the graphing format, data sonification, and with the tactile nature of the mooring construction.

This paper falls in the field of playing analytics. It deals with an empirical work dedicated to explore the potential of data sonification (i.e. the conversion of data into sound that reflects their objective properties or relations). Data sonification is proposed as an alternative to data visualization. We applied data sonification for the analysis of gameplays and players’ strategies during a session dedicated to game-based learning. The data of our study (digital traces) was collected from 200 pre-service teachers who played Tamagocours, an online collaborative multiplayer game dedicated to learn the rules (i.e. copyright) that comply with the policies for the use of digital resources in an educational context. For one typical individual (parangon) for each of the 5 categories of players, the collected digital traces were converted into an audio format so that the actions that they performed become listenable. A specific software, SOnification of DAta for Learning Analytics (SODA4LA), was developed for this purpose. The first results show that different features of the data can be recognized from data listening. These results also enable for the identification of different parameters that should be taken into account for the sonification of diachronic data. We consider that this study open new perspectives for playing analytics. Thus we advocate for new research aiming at exploring the potential of data sonification for the analysis of complex and diachronic datasets in the field of educational sciences.

Research into the origins of music and language can shed new light on musical representation, including program music and more recent incarnations such as data sonification. Although sonification and program music have different aims — one scientific explication, the other artistic expression — similar techniques, relying on human and animal biology, cognition, and culture, underlie both. Examples include Western composers such as Beethoven and Berlioz, to more recent figures like Messiaen, Stockhausen and Tom Johnson, as well as music theory, semiotics, biology, and data sonifications by myself and others. The common thread connecting these diverse examples is the use of human musicality, in the bio- musicological sense, for representation. Links between musicality and representation — dimensions like high/low, long/short, near/far, etc., bridging the real and abstract — can prove useful for researchers, sound designers, and composers.

The idea that sound can convey information predates the modern era, and certainly the computational present. Data sonification can be broadly described as the creation, study and use of the non-speech aural representation of information to convey information. As a field of contemporary enquiry and practice, data sonification is young, interdisciplinary and evolving; existing in parallel to the field of data visualization. Drawing on older practices such as auditing, and the use of information messaging in music, this paper provides an historical understanding of how sound and its representational deployment in communicating information has changed. In doing so, it aims to encourage a critical awareness of some of the sociocultural as well as technical assumptions often adopted in sonifying data, especially those that have been developed in the context of Western music of the last half-century or so.

Computational music analysis investigates the relevant features required for the detection and classification of musical content, features which do not always directly overlap with musical composition concepts. Human perception of music is also an active area of research, with existing work considering the role of perceptual schema in musical pattern recognition. Data sonification investigates the use of non-speech audio to convey information, and it is in this context that some potential guidelines for human pattern recognition are presented for discussion in this paper. Previous research into the role of musical contour (shape) in data sonification shows that it has a significant impact on pattern recognition performance, whilst investigation in the area of rhythmic parsing made a significant difference in performance when used to build structures in data sonifications. The paper presents these previous experimental results as the basis for a discussion around the potential for inclusion of schema- based classifiers in computational music analysis, considering where shape and rhythm classification may be employed at both the segmental and supra-segmental levels to better mimic the human process of perception.

This article reflects on recent notions about data sonification within sound-based experimental and artistic practices. The intention is not to survey the current state of data sonification methods and techniques as such, but rather to suggest a number of selected points of critique for addressing specific assumptions about processes and discourses related to what we may broadly refer to as sonification. Furthermore, these issues will be addressed by critically asking what we understand by “data” in the first place, as something susceptible to be turned into actual sounding material. Considering how specific discourses and cultural understandings frame contemporary notions of data, the article also includes different examples of alternative, exploratory practices. Thus, one of the aims will partly be to open up a transdisciplinary discussion about the critical affordances and potential pitfalls of data sonification seen both as an aesthetic and a knowledge-producing practice. This involves not only attention toward strictly academic and scientific settings, but also relates to how data sonification ventures are being communicated within broader societal, cultural and art institutional contexts.