Use Of Sonification Research Articles

Skin Cancer Detection in Diverse Skin Tones by Machine Learning Combining Audio and Visual Convolutional Neural Networks

Introduction: Skin cancer (SC) is common in fair skin (FS) at a 1:5 lifetime incidence for nonmelanoma skin cancer. In order to assist clinicians’ decisions, a risk intervention technology was developed, which combines a dual-mode machine learning of visual and sonified (pixel to sound) data. The addition of an audio technology enhances malignant features of lesions, increases sensitivity and was previously validated under a prospective clinical setting in FS. In dark skin (DS), although rare by a 10–30 factor, skin cancer is diagnosed at more advanced stages resulting in a delayed diagnosis and affecting life quality and expectancy. It is known as well that SC diagnostic accuracy by machine learning in DS is decreased as compared to FS. The present study tests the use of sonification aided by artificial intelligence algorithms to compare diagnostics of different skin tones. Methodology: Biopsy-validated smartphone images were diagnosed in a retrospective study by a dual audio-visual convoluted neural network. A total of 60 Fitzpatrick I-III were compared to 72 Fitzpatrick IV-VI. A dichotomous diagnostic output, either malignant or benign, was assessed for sensitivity, specificity and area under the curves (AUCs) for the receiver operating characteristic (ROC) curve. Results: ROC curve analytics indicated an AUC of 0.858 (95% CI: 0.795–0.921) and 0.856 (95% CI: 0.759–0.953) for fair and DS (p = NS). Sensitivity of Fitzpatrick I-III skin and Fitzpatrick IV-VI were 84.4% (71.8–96.9) and 79.6% (63.4–93.8), respectively (p = NS). Specificity of Fitzpatrick I-III skin and Fitzpatrick IV-VI were 84.2% (72.6–95.8) and 85.3% (73.4–97.2), respectively (p = NS). The positive predictive and negative predictive values as well as accuracy (0.817 vs. 0.847) were all within the same range (p = NS). Conclusions: The results demonstrate that the dual-modality classifier identifies skin cancer of FS and DS similarly well. Sonification of malignant signs of a skin lesion demonstrates promising results, even with smartphone images, which should be considered as a tool to achieve more effective and accessible healthcare.

The design of Datascapes: toward a design framework for sonification for anomaly detection in AI-supported networked environments

There is a growing need for solutions that can improve the communication between anomaly detection algorithms and human operators. In the context of real-time monitoring of networked systems, it is crucial that new solutions do not increase the burden on an already overloaded visual channel. Sonification can be leveraged as a peripheral monitoring tool that complements current visualization systems. We conceptualized, designed, and prototyped Datascapes, a framework project that explores the potential of sound-based applications for the monitoring of cyber-attacks on AI-supported networked environments. Within Datascapes, two Design Actions were realized that applied sonification on the monitoring and detection of anomalies in (1) water distribution networks and (2) Internet networks. Two series of prototypes were implemented and evaluated in a real-world environment with eight experts in network management and cybersecurity. This paper presents experimental results on the use of sonification to disclose anomalous behavior and assess both its gravity and the location within the network. Furthermore, we define and present a design methodology and evaluation protocol that, albeit grounded in sonification for anomaly detection, can support designers in the definition, development, and validation of real-world sonification applications.

PAINTING GRAPHS WITH SOUNDS: COSMONIC SONIFICATION PROJECT

CosMonic (COSmos harMONIC) is a sonification project with a triple purpose: analysis (by means of sounds) of any type of data, source of inspiration for artistic creations, and pedagogical and dissemination purposes. In this contribution we present the work recently produced by CosMonic in the latter field, creating specific cases for the inclusive astronomy dissemination project Astroaccesible for blind and partially sighted people, but also aimed at a general public that wants to understand astrophysics in an alternative format. For this project, CosMonic seeks to create simple astronomical cases in their acoustic dimension in order to be easily understood. CosMonic's philosophy for these sonifications can be summarized in a simple metaphor: painting graphs with sounds. Sonification is a powerful tool that helps to enhance visual information. Therefore, CosMonic accompanies its audios with animations, using complementary methods to reach a general public. In addition to providing some cases created by CosMonic for inclusive astronomy, we also share our experience with different audiences, as well as suggest some ideas for a better use of sonification in (global) inclusive outreach.

Measurement and sonification of construction site noise and particle pollution data

PurposeThe noise and dust particles caused by the construction transport are by most stakeholders experienced as disturbing. The purpose of this study is to explore how sonification can support visualization in construction planning to decrease construction transport disturbances.Design/methodology/approachThis paper presents an interdisciplinary research project, combining research on construction logistics, internet of things and sonification. First, a data recording device, including sound, particle, temperature and humidity sensors, was implemented and deployed in a development project. Second, the collected data were used in a sonification design, which was, third, evaluated with potential users.FindingsThe results showed that the low-cost sensors used could capture “good enough” data, and that the use of sonification for representing these data is interesting and a possible useful tool in urban and construction transport planning.Research limitations/implicationsThere is a need to further evolve the sonification design and better communicate the aim of the sounds used to potential users. Further testing is also needed.Practical implicationsThis study introduces new ideas of how to support visualization with sonification planning the construction work and its impact on the vicinity of the site. Currently, urban planning and construction planning focus on visualizing the final result, with little focus on how to handle disturbances during the construction process.Originality/valueShowing the potentials of using low-cost sensor data in sonification, and using sonification together with visualization, is the result of a novel interdisciplinary research area combination.

Sonification of Emotion in Social Media: Affect and Accessibility in Facebook Reactions

Facebook Reactions are a collection of animated icons that enable users to share and express their emotions when interacting with Facebook content. The current design of Facebook Reactions utilizes visual stimuli (animated graphics and text) to convey affective information, which presents usability and accessibility barriers for visually-impaired Facebook users. In this paper, we investigate the use of sonification as a universally-accessible modality to aid in the conveyance of affect for blind and sighted social media users. We discuss the design and evaluation of 48 sonifications, leveraging Facebook Reactions as a conceptual framework. We conducted an online sound-matching study with 75 participants (11 blind, 64 sighted) to evaluate the performance of these sonifications. We found that sonification is an effective tool for conveying emotion for blind and sighted participants, and we highlight sonification design strategies that contribute to improved efficacy. Finally, we contextualize these findings and discuss the implications of this research with respect to HCI and the accessibility of online communities and platforms.

The Use of Sonification for the Analysis and Teaching of Interpretive Auditory Nuances.

The integrative powers of human auditory perception in masterful performing artists can create nuances of sound from the interpretation of notes on a written page that often seem beyond one’s grasp as a listener. It is important to consider what type of feedback can provide a clearer understanding of nuances in sound to guide motor learning for the acquisition of new skills for expression. Master pianists’ performance has been used as a model for imitation. However, to ensure the accuracy of imitation and clear understanding, sonification was examined for its effectiveness in providing a more immediate understanding of individual interpretations in terms of inter-onset timing and velocity of the notes. Three master concert pianists volunteered to record a performance of the Chopin Nocturne Opus 15 No. 1 on a Yamaha Disklavier Pro MIDI (music instrument digital interface) grand piano. Logic software was used to analyze and compare MIDI data from each performance from the perspective of phrase-by-phrase, note onset timings, and corresponding data from the other pianists. The study objectives were to examine commonalities and differences in timing and dynamics among the performances using MIDI measurements; to probe whether listening to comparative performance data assisted with feedback from sonification would enhance music students’ understanding of the interpretive nuances through imitation; and to determine whether auditory-assisted sonification feedback could be used as a tool to expand students’ interpretive choices and enhance performance. Participants imitated selected phrases of each of the master pianists, first with only music listening, and then with feedback from the sonified, comparative performance data. Results showed limited success in attempts to imitate the model with auditory feedback alone. Auditory-assisted sonification feedback significantly enhanced the participants’ abilities to imitate the model faster and with greater accuracy in the final imitation experiments. The data gathered by the study provide insights on this kind of sonification as an effective feedback tool for heightening participants’ auditory understanding of nuances in sound, as well as providing an effective teaching tool for imitation exercises.

Warning users about cyber threats through sounds

This paper reports a formative evaluation of auditory representations of cyber security threat indicators and cues, referred to as sonifications, to warn users about cyber threats. Most Internet browsers provide visual cues and textual warnings to help users identify when they are at risk. Although these alarming mechanisms are very effective in informing users, there are certain situations and circumstances where these alarming techniques are unsuccessful in drawing the user’s attention: (1) security warnings and features (e.g., blocking out malicious Websites) might overwhelm a typical Internet user and thus the users may overlook or ignore visual and textual warnings and, as a result, they might be targeted, (2) these visual cues are inaccessible to certain users such as those with visual impairments. This work is motivated by our previous work of the use of sonification of security warnings to users who are visually impaired. To investigate the usefulness of sonification in general security settings, this work uses real Websites instead of simulated Web applications with sighted participants. The study targets sonification for three different types of security threats: (1) phishing, (2) malware downloading, and (3) form filling. The results show that on average 58% of the participants were able to correctly remember what the sonification conveyed. Additionally, about 73% of the participants were able to correctly identify the threat that the sonification represented while performing tasks using real Websites. Furthermore, the paper introduces “CyberWarner”, a sonification sandbox that can be installed on the Google Chrome browser to enable auditory representations of certain security threats and cues that are designed based on several URL heuristics.Article highlightsIt is feasible to develop sonified cyber security threat indicators that users intuitively understand with minimal experience and training.Users are more cautious about malicious activities in general. However, when navigating real Websites, they are less informed. This might be due to the appearance of the navigating Websites or the overwhelming issues when performing tasks.Participants’ qualitative responses indicate that even when they did not remember what the sonification conveyed, the sonification was able to capture the user’s attention and take safe actions in response.

A Breathing Sonification System to Reduce Stress During the COVID-19 Pandemic.

Since sound and music are powerful forces and drivers of human behavior and physiology, we propose the use of sonification to activate healthy breathing patterns in participants to induce relaxation. Sonification is often used in the context of biofeedback as it can represent an informational, non-invasive and real-time stimulus to monitor, motivate or modify human behavior. The first goal of this study is the proposal and evaluation of a distance-based biofeedback system using a tempo- and phase-aligned sonification strategy to adapt breathing patterns and induce states of relaxation. A second goal is the evaluation of several sonification stimuli on 18 participants that were recruited online and of which we analyzed psychometric and behavioral data using, respectively questionnaires and respiration rate and ratio. Sonification stimuli consisted of filtered noise mimicking a breathing sound, nature environmental sounds and a musical phrase. Preliminary results indicated the nature stimulus as most pleasant and as leading to the most prominent decrease of respiration rate. The noise sonification had the most beneficial effect on respiration ratio. While further research is needed to generalize these findings, this study and its methodological underpinnings suggest the potential of the proposed biofeedback system to perform ecologically valid experiments at participants' homes during the COVID-19 pandemic.

Sonic Bodies

This article examines the use of sonification, and movement sonification, as a tool for engagement with artworks in galleries and museums. It focuses on the relationship between artwork and sighted, partially sighted and non-sighted visitors and the way sonification, by offering an audio-centred mode of engagement, complicates the aesthetic experience. By drawing on Bruno Latour’s discussion on hybrids (1993) as well as with reference to examples of the use of sonification in gallery spaces, the article argues that rather than assuming the artwork to be a finished and stable object, movement sonification approaches the artwork as a blueprint of proximal, conceptual and material relationships which are animated by a moving-listening subject. As such, the article concludes, movement sonification can have radical implications for the way artworks are conceived as well as the way the aesthetic experience is configured.

The use of sonification for physiotherapy in human movement tasks: A scoping review

Summary Objectives This review aims to: (1) map the use of sonification in human movement tasks for physical therapy; (2) identify methods of data capture, tasks and its effects on human subjects; (3) suggest future research directions. News Sonification can be described as a technique to translate data into sound. It has been used for human motion analysis tasks even if it is not part of most physical therapist's lexicon. Prospects and projects Identify and analyze publications where sonification was used as an audio-feedback technique for physical therapy. Thirty-five papers were included, 13 randomized-control-trials. Thirteen papers reported an investigation on a specific dysfunction, while upper limb movements were investigated in fifteen papers. Inertial measurement units were the most commonly used technology to capture human movement, 10 papers reported improvements in motor control and/or movement quality. Gaps in the literature were identified: (1) absence of sonification framework for rehabilitation, (2) no long-term comparison with gold-standard interventions for specific populations, (3) approaches for cardio-respiratory physical therapy and injury prevention were absent. Conclusion Sonification has the potential to support rehabilitation for physical therapy. Effects of sonification were varied and ranged from improvements in movement quality/control, increased movement and body-awareness and improvements in performance when compared with activities with audio-visual or non-specific audio-feedback among others. Data for sonification was mainly captured using inertial measurement units, smartphones and optical tracking devices but others are also commonly used. Well-designed clinical trials supported by current promising results need to be developed. We recommend testing different sonification techniques in common physical therapy disfunctions using significant outcome measures to understand and maximize its effects on motor learning and control while scoping for further benefits.

Gait Training in Orthopedic Rehabilitation after Joint Replacement - Back to Normal Gait with Sonification?

Abstract Even several years after total hip (THR) and total knee replacement (TKR) surgery patients frequently show deficient gait patterns leading to overloads and relieving postures on the contralateral side or in the spine. Gait training is, in these cases, an essential part of rehabilitation. The aim of this study was to compare different feedback methods during gait training after THR and TKR focusing, in particular, on auditory feedback via sonification. A total of 240 patients after THR and TKR were tested in a pre-post-test design during a 3-week rehabilitation period. Even though sonification did not show, statistically, a clear advantage over other feedback methods, it was well accepted by the patients and seemed to significantly change gait pattern during training. A sudden absence of sonification during training led to a rapid relapse into previous movement patterns, which highlights its effectiveness in breaking highly automated gait patterns. A frequent use of sonification during and after rehabilitation could, hence, reduce overloading after THR and TKR. This may soon be viable, since new technologies, such as inertial measurement units, allow for wearable joint angle measurement devices. Back to normal gait with sonification seems possible.

Musical sonification supports visual discrimination of color intensity

ABSTRACTVisual representations of data introduce several possible challenges for the human visual perception system in perceiving brightness levels. Overcoming these challenges might be simplified by adding sound to the representation. This is called sonification. As sonification provides additional information to the visual information, sonification could be useful in supporting the visual perception. In the present study, usefulness (in terms of accuracy and response time) of sonification was investigated with an interactive sonification test. In the test, participants were asked to identify the highest brightness level in a monochrome visual representation. The task was performed in four conditions, one with no sonification and three with different sonification settings. The results show that sonification is useful, as measured by higher task accuracy, and that the participant's musicality facilitates the use of sonification with better performance when sonification was used. The results were also supported by subjective measurements, where participants reported an experienced benefit of sonification.

Sonification to Support the Monitoring Tasks of Security Operations Centres

Sonification (the representation of data as sound) may offer a solution to some of the network-security monitoring challenges faced in security operations centres (SOCs). Prior work has shown that sonification can present network-security information to humans effectively, and indicated that security practitioners foresee potential for sonification to aid in scenarios related to their work. The use of sonification by security practitioners in tasks relevant to SOCs has not been examined, however. To address this gap, we assessed the use of sonification by security practitioners in network-security monitoring tasks in an experimental setting. We report on the results of a study in which we compared the performance of security practitioners using a Security Information and Event Management (SIEM) tool with their performance using a SIEM tool that incorporated sonification, in a primary and a non-primary monitoring task. In both tasks, a number of aspects of the monitoring performance of participants were significantly improved when sonification was used. Our results support the potential for sonification to aid in SOC tasks, and indicate a need to validate the utility of sonification systems by running them in operational SOCs.

The sound of migration: exploring data sonification as a means of interpreting multivariate salmon movement datasets

The migration of Pacific salmon is an important part of functioning freshwater ecosystems, but as populations have decreased and ecological conditions have changed, so have migration patterns. Understanding how the environment, and human impacts, change salmon migration behavior requires observing migration at small temporal and spatial scales across large geographic areas. Studying these detailed fish movements is particularly important for one threatened population of Chinook salmon in the Snake River of Idaho whose juvenile behavior may be rapidly evolving in response to dams and anthropogenic impacts. However, exploring movement data sets of large numbers of salmon can present challenges due to the difficulty of visualizing the multivariate, time-series datasets. Previous research indicates that sonification, representing data using sound, has the potential to enhance exploration of multivariate, time-series datasets. We developed sonifications of individual fish movements using a large dataset of salmon otolith microchemistry from Snake River Fall Chinook salmon. Otoliths, a balance and hearing organ in fish, provide a detailed chemical record of fish movements recorded in the tree-like rings they deposit each day the fish is alive. This data represents a scalable, multivariate dataset of salmon movement ideal for sonification. We tested independent listener responses to validate the effectiveness of the sonification tool and mapping methods. The sonifications were presented in a survey to untrained listeners to identify salmon movements with increasingly more fish, with and without visualizations. Our results showed that untrained listeners were most sensitive to transitions mapped to pitch and timbre. Accuracy results were non-intuitive; in aggregate, respondents clearly identified important transitions, but individual accuracy was low. This aggregate effect has potential implications for the use of sonification in the context of crowd-sourced data exploration. The addition of more fish, and visuals, to the sonification increased response time in identifying transitions.

Interactive Sonification in Rowing: Acoustic Feedback for On-Water Training

Feedback systems used in elite sport mainly provide visual information. Another approach displays information audibly via sonification to create coherency between action and reaction. This article describes the use of sonification as acoustic feedback (AF) in on-water rowing training with elite athletes. On the theoretical basis of an ecological dynamics approach, the audio-motor relationship was elucidated for understanding expertise and skill acquisition in sport. Results gained from athlete surveys between 2009 and 2013 were presented to determine if AF reflects specific sections within the rowing-motion comprehensible for the athletes and if the information is provided in a way that is useful for technique training. The final aim was to provide criteria and recommendations for the development of sonification-based applications within a moving context for sports and rehabilitation.

Living with alarms: the audio environment in an intensive care unit

This article treats the use of sonification in Percy Military Training Hospital’s intensive care unit, through an interview with Anaesthetist Professor Bruno Debien. It starts with a description of the environment completed by some technical information concerning the equipment. This is followed by a commented transcription of the interview with Bruno Debien and concludes with reflections on the nature of audio alarms and their relation to different modes of listening.

The effects of sonification and TiO 2 deposition on the micro-characteristics of the thermally treated SiO 2/TiO 2 spherical core–shell particles for photo-catalysis of methyl orange

SiO 2/TiO 2 core–shell particles were prepared to decompose methyl orange (MO) in a laboratory scale study prior to its use in the treatment of waste containing azo dyes that might potentially pose adverse environmental consequences and health hazard to human. The preparation of silica cores produced perfectly spherical and smooth surfaces, but when TiO 2 shell/film was deposited using titanium butoxide (TBOT) precursor at concentration >0.24 mol L −1, the resulting shell was rougher and loose, giving rise to Ostwald ripening that caused the core–shell particles to agglomerate. The use of sonification improved the dispersion at the expense of thin shell. When multi-step shell coating was applied to increase the shell thickness, loose titania shells were observed despite increasing surface coverage at higher number of coating step. The isoelectric point of the core–shell particles was equivalent to that of pure titania (4.5) due to the complete encapsulation of silica cores by titania. The silica core crystals exhibited highly amorphous structures, whereas titania covered silica cores demonstrated pure anatase phase that grew in size upon heat treatment at 900 °C. Rutile phase started to develop at 5 coatings, suggesting that anatase crystallite size was attainable for as long as the critical crystalline size smaller than ca. 30 nm was sustained or equivalently, 3 coatings were applied. Complete decomposition of MO was achieved in less than 370 min using core–shell particles prepared from triple coating.

Sounds sequential: sonification in the social sciences

This article discusses the use of sound for auditory information display, and in particular its application for exploration of scientific data, known as sonification. Sonification can be defined as the use of sound to display data of scientific interest in order to investigate structures, trends or patterns in the data. Background is provided from several perspectives: the use of the senses in the history of science, the strengths of human hearing, the recent technological availability of auditory interfaces, the development of sonification itself, and differentiation of sonification from musical practices. In Western science, as in Western culture, the eye has become the predominant organ of sense; using the ear consciously in research thus implicitly questions the implications of the eye's predominance. As practical examples, two applications of sonification to data-sets from the social sciences are discussed in detail. We argue that the most promising areas of application of sonification within the social sciences are in the exploration of sequential data. Our two examples both concern sets of sequential data, one temporal, the other spatial (geographical). Discussion of these examples is followed by consideration of the practical and cultural implications of working with sonification. We thus hope to further the use of sonification in the social sciences, not as an alternative to visualisation or statistical approaches, but as a complementary tool of data analysis and exploration.