Sonic Feedback Research Articles

Inducing Flow In Board Games Through Augmented Audio

Games provide high levels of enjoyment and fun through closed systems with concrete objectives. These characteristics enhance a state of concentration known as flow. Many modern board games provide good gaming experiences. However, it is possible to use various resources to further enhance the experience. This study proposes that augmented audio technology can be used to induce flow during a board game playthrough. For this purpose, a prototype was developed to be tested with the board game Rising Sun. The main concept behind this approach was to deliver to the players real time sonic feedback through an augmented sound environment mapped to in game actions, by triggering spatialized contextual sound events related to the user’s in-game actions and game phases. A study was conducted using the aforementioned prototype to validate the proposal, and the results have shown that augmented audio may induce the state of flow.

Adding visual contextual information to continuous sonification feedback about low-reliability situations in conditionally automated driving: A driving simulator study

In conditionally automated driving, control transitions where drivers are asked to take over manual control quickly are a known challenge as drivers are likely “out-of-the-loop” due to automated driving. We previously demonstrated a useful sonification feedback design to support drivers for takeover events by manipulating background music to convey real-time changes in the automation reliability level. The present research examines two types of auxiliary information—explanations about the potential sources for reduced reliability and recommended actions to prepare for a potential takeover event—to communicate event-specific context about the low-reliability situations indicated by the sonification feedback. In a driving simulator with SAE level 3 automation, forty participants were assigned to one of four interface conditions: (i) sonification feedback only, (ii) sonification feedback and visual auxiliary information providing explanations, (iii) sonification feedback and visual auxiliary information providing action recommendations, or (iv) sonification feedback and both explanatory and action-recommending auxiliary information. Participants experienced six low-reliability events that belonged to three types of situations that may eventually lead to a takeover event: operational problems, system limitations, and unexpected events. Two events became system-requested takeover events. Our results showed that auxiliary information impacted visual information processing but not the takeover decisions or response times to takeover requests. The situation type—characterized by the availability of cues and presence of hazards in the environment—also played a significant role in both participants’ takeover decisions and visual information processing behaviors. These findings have implications for designing effective feedback for not only Level 3 automation, which we studied, but also for Level 4 dual-mode vehicles that could allow transitions to manual driving when exiting their operational design domains and for Level 2 automation, where drivers may need more support to monitor the road effectively and make more timely takeover decisions.

Sofigait-A Wireless Inertial Sensor-Based Gait Sonification System.

In this study, a prototype of an inertial sensor-based gait sonification system was tested for the purpose of providing real-time gait feedback on the knee angle. The study consisted of two parts: (1) a comparison of the knee angle measurement to a marker-based 3D optical capturing system (Vicon, Oxford, UK) with N = 24 participants and (2) an evaluation four different sonification feedback versions in an accentuation × pitch (2 × 2) design on a sample of N = 28 participants. For the measurement system comparison, the RMSE was 7.6° ± 2.6° for the left and 6.9° ± 3.1° for the right side. Measurement agreement with bias up to -7.5° ± 6.2° (for maximum knee flexion) was indicated by the Bland-Altmann Method. The SPM revealed significant differences between both measurement systems for the area 45-90% (p < 0.001) (left) and the area between 45% and 80% (p = 0.007) (right). For the sonification perception, the variation of pitch had a significant effect on the perception of pleasantness of the sound. No effect was found for the accentuation of the swing or stance phase.

Motor retraining by real-time sonic feedback: understanding strategies of low impact running (PhD Academy Award)

Motor retraining by real-time sonic feedback: understanding strategies of low impact running (PhD Academy Award)

Sonic Ecologies

Sonic Ecologies

The Use of Sonification for the Analysis and Teaching of Interpretive Auditory Nuances.

The integrative powers of human auditory perception in masterful performing artists can create nuances of sound from the interpretation of notes on a written page that often seem beyond one’s grasp as a listener. It is important to consider what type of feedback can provide a clearer understanding of nuances in sound to guide motor learning for the acquisition of new skills for expression. Master pianists’ performance has been used as a model for imitation. However, to ensure the accuracy of imitation and clear understanding, sonification was examined for its effectiveness in providing a more immediate understanding of individual interpretations in terms of inter-onset timing and velocity of the notes. Three master concert pianists volunteered to record a performance of the Chopin Nocturne Opus 15 No. 1 on a Yamaha Disklavier Pro MIDI (music instrument digital interface) grand piano. Logic software was used to analyze and compare MIDI data from each performance from the perspective of phrase-by-phrase, note onset timings, and corresponding data from the other pianists. The study objectives were to examine commonalities and differences in timing and dynamics among the performances using MIDI measurements; to probe whether listening to comparative performance data assisted with feedback from sonification would enhance music students’ understanding of the interpretive nuances through imitation; and to determine whether auditory-assisted sonification feedback could be used as a tool to expand students’ interpretive choices and enhance performance. Participants imitated selected phrases of each of the master pianists, first with only music listening, and then with feedback from the sonified, comparative performance data. Results showed limited success in attempts to imitate the model with auditory feedback alone. Auditory-assisted sonification feedback significantly enhanced the participants’ abilities to imitate the model faster and with greater accuracy in the final imitation experiments. The data gathered by the study provide insights on this kind of sonification as an effective feedback tool for heightening participants’ auditory understanding of nuances in sound, as well as providing an effective teaching tool for imitation exercises.

Haptic Feedback to Assist Blind People in Indoor Environment Using Vibration Patterns.

Feedback is one of the significant factors for the mental mapping of an environment. It is the communication of spatial information to blind people to perceive the surroundings. The assistive smartphone technologies deliver feedback for different activities using several feedback mediums, including voice, sonification and vibration. Researchers 0have proposed various solutions for conveying feedback messages to blind people using these mediums. Voice and sonification feedback are effective solutions to convey information. However, these solutions are not applicable in a noisy environment and may occupy the most important auditory sense. The privacy of a blind user can also be compromised with speech feedback. The vibration feedback could effectively be used as an alternative approach to these mediums. This paper proposes a real-time feedback system specifically designed for blind people to convey information to them based on vibration patterns. The proposed solution has been evaluated through an empirical study by collecting data from 24 blind people through a mixed-mode survey using a questionnaire. Results show the average recognition accuracy for 10 different vibration patterns are 90%, 82%, 75%, 87%, 65%, and 70%.

Visual Displays Augmenting Sonification Feedback to Support Drivers in Conditionally Automated Driving

Visual Displays Augmenting Sonification Feedback to Support Drivers in Conditionally Automated Driving

Interactive Vibrotactile Feedback Enhances the Perceived Quality of a Surface for Musical Expression and the Playing Experience.

An advanced multi-touch sensor surface aimed at musical expression was recently equipped by the authors with interactive multi-point localized vibrotactile feedback. Using such interface, a subjective assessment was conducted that measured how the presence and type of vibration affect the perceived quality of the device and various attributes related to the playing experience. Two clearly distinct sound settings each with three vibrotactile feedback strategies were tested. At each trial, the task was to play freely while comparing two related setups which used the same sound setting and differed only in the presence/absence of vibration. Independent of the sound setting, as compared to the respective non-vibrating setups, vibrations conveying frequency and amplitude dynamics cues coherent with the player's gesture and/or sonic feedback had the most positive effect. Vibrotactile feedback especially improved the enjoyment of playing and the perceived potential for musical expressivity.

Manipulating music to communicate automation reliability in conditionally automated driving: A driving simulator study

Abstract With increasing vehicle automation, drivers are likely to engage in non-driving related tasks. However, before fully autonomous vehicles can be achieved, drivers remain the fallback when automation limits are reached and/or in unexpected situations. Transition from automated to manual control could be particularly difficult for drivers who are “out-of-the-loop”. To support safe and smooth control transition, we adopted the blended sonification approach in manipulating background music to convey information about the reliability level of the automation. A driving simulator study consisting of 36 participants was conducted to investigate the effectiveness of sonification feedback—three levels of system reliability was mapped to background music pitch levels—on takeover events. Participants were assigned to one of three groups that received continuous sonification feedback, intermittent sonification feedback, or no feedback about automation reliability. The proposed blended music sonification was found effective in increasing monitoring behavior and decreasing visual response times to takeover requests during automated driving. Steering control in a takeover scenario that required substantial amount of maneuvering also benefitted from having sonification feedback. There was some evidence that sonification feedback provided continuously was more effective than when it was provided intermittently, possibly related to observed differences in the associated glance patterns. In conclusion, blended sonification may be an effective approach in helping drivers stay on the loop during automated driving, although the study was limited to the choice of music examined. Our future work will investigate how best to combine automation reliability information and traffic situational information to better support situation awareness and facilitate task-switching in automated driving.

Assisting Home-Based Resistance Training for Normotensive and Prehypertensive Individuals Using Ambient Lighting and Sonification Feedback: Sensor-Based System Evaluation.

BackgroundPhysical exercise is an effective lifestyle intervention to improve blood pressure. Although aerobic sports can be performed anywhere, resistance exercises are traditionally performed at the gym; extending the latter to the home setting may promote an increase in the number of practitioners.ObjectiveThis study aims to evaluate a sensor-based system that guides resistance exercises through ambient lighting and sonification (A/S) feedback in a home setting in 34 study participants who were normotensive and prehypertensive.MethodsParticipants took part in a 1.5-hour exercise session in which they experienced the A/S feedback (ie, experimental condition) as well as a control condition (ie, no feedback) and a reference condition (ie, verbal feedback through a human remote coach). The system was evaluated for improving exercise form (range of motion, timing, and breathing patterns) as well as psychophysiological experience (perceived exertion, attentional focus, competence, and motivation).ResultsA/S feedback was significantly better than the control for concentric (mean 2.48, SD 0.75 seconds; P<.001) and eccentric (mean 2.92, SD 1.05 seconds; P<.001) contraction times, concentric range of motion consistency (mean 15.64, SD 8.31 cm vs mean 17.94, SD 9.75 cm; P<.001), and perceived exertion (mean 3.37, SD 0.78 vs mean 3.64, SD 0.76; P<.001). However, A/S feedback did not outperform verbal feedback on any of these measures. The breathing technique was best in the control condition (ie, without any feedback). Participants did not show more positive changes in perceived competence with A/S feedback or verbal feedback.ConclusionsThe system seemed to improve resistance exercise execution and perception in comparison with the control, but did not outperform a human tele-coach. Further research is warranted to improve the breathing technique.

Sound Stories

Abstract The paper ‘Sound Stories: A Context-Based Study of Everyday Listening to Augmented Soundscapes’ by Milena Droumeva and Iain McGregor published in Interacting with Computers, Volume 31, Issue 3, May 2019, explores how soundscapes are transforming to accommodate additional sonic feedback.

Sound Stories: A Context-Based Study of Everyday Listening to Augmented Soundscapes

Abstract With an increasing number of everyday operations and communications becoming both automated and autonomous, ambient intelligent soundscapes are transforming to accommodate additional sonic feedback, and with it, new frameworks of listening. While this type of research and design of audio augmented technology is not new, the impact pre-existing acoustic environments upon listeners’ sense-making activities is rarely considered holistically. Much of the study into the design of effective auditory displays focuses on perceptual acuity and correct source identification, often at the expense of understanding the context of meaning-making. This paper presents a study involving 70 participants who listened to unidentified audio recordings of two archetypal everyday urban sound environments naturally containing artificial signals as well as typical sounds. Using a ThinkAloud protocol we investigated listeners’ approaches to meaning-making in both semantic and temporal dimensions. Through a semantic content analysis, we articulate five aspects of sonic meaning-making: spatial, descriptive, experiential, associational and narrative. We further analyse the use of these perceptual elements on a temporal plane, in order to investigate how listeners construct a narrative of what they hear in real-time, naturally evolving as each subsequent sound event is interpreted. Results suggest that while listeners attend to sound events and spatial characteristics of a sound environment at the beginning of a new listening situation, as the soundscape unfolds they utilize associations and familiarity in order to place individual sounds into increasingly coherent narratives. Finally, we suggest that this approach could provide sound designers and human–computer interaction specialists with a model for investigating the context aspects of a soundscape more holistically, allowing them to evaluate the effect of any new designed sounds prior to introduction into real-world environments.

Assisting Dumbbell Curls via Interactive Sonification of Arm Movement and Muscular Signals

We developed an interactive sonification system that lets users listen to their own hand/arm movement and bicep muscle signals as sonic feedback during biceps curls exercise. The system aims to improve the exercise quality and user motivation. Based on two studies, we found that the sonification was more effective in providing temporal cues to slow down the repetition but not as effective as hoped in extending the vertical movement range and increasing repetition amount. The studies provide some design guidelines for multivariate sonification and also reflect a wider potential for applications that include general fitness, physiotherapy, and sports training.

The first impression in the urban sonic experience: transitions, attention, and attunement

ABSTRACTThe objective of this study is to understand the phenomenology of moments of transition between sonic environments in the urban context and conceptualize the role they play in the attunement of the body to urban spaces. It has been argued that transitions between different environments tend to be moments when sonic perception and feedback are enhanced and, as moments of interruption, they play a significant role in how we attune to urban places. Our study contributes to a greater understanding of these moments. We present the concept of sonic first impression to address the affective nuances of these moments, and the concept of sonic attunement to address how individuals tune in to urban places through sound. In order to explore how these concepts apply in practice, we have undertaken a participatory methodology with 12 volunteers, which included sound diaries and go-alongs. These methods allowed us to gather in-depth data about spatial and temporal transitions and issues related to the angle of arrival of the body. Our results unveil the role that sound plays in attunement to place, a role that encompasses both meaning and affect, active and passive hearing, consciousness and unconsciousness, mind and body, psychology and physiology.

Sensori-Motor Learning with Movement Sonification: Perspectives from Recent Interdisciplinary Studies.

This article reports on an interdisciplinary research project on movement sonification for sensori-motor learning. First, we describe different research fields which have contributed to movement sonification, from music technology including gesture-controlled sound synthesis, sonic interaction design, to research on sensori-motor learning with auditory-feedback. In particular, we propose to distinguish between sound-oriented tasks and movement-oriented tasks in experiments involving interactive sound feedback. We describe several research questions and recently published results on movement control, learning and perception. In particular, we studied the effect of the auditory feedback on movements considering several cases: from experiments on pointing and visuo-motor tracking to more complex tasks where interactive sound feedback can guide movements, or cases of sensory substitution where the auditory feedback can inform on object shapes. We also developed specific methodologies and technologies for designing the sonic feedback and movement sonification. We conclude with a discussion on key future research challenges in sensori-motor learning with movement sonification. We also point out toward promising applications such as rehabilitation, sport training or product design.

Comparative study on the effect of Parameter Mapping Sonification on perceived instabilities, efficiency, and accuracy in real-time interactive exploration of noisy data streams

This paper presents a comparative study on common Parameter Mapping Sonifications (PMSons) designed to reduce the perception of instabilities during the real-time interactive exploration of noisy data streams. The objective of this study is to evaluate the capacity of these PMSons in reducing the perceived fluctuations while preserving both efficiency and accuracy of participants’ estimations during the exploration task. Based on the real application of beacon localization for rescue operations, an abstraction of the task was developed which simulates the basic concepts involved, i.e. the estimation of local maxima during the exploration of a 1D data topography. Three different PMSons were selected, based on pitch, pitch averaged over time, and tempo, evaluated for three different noise levels applied to the data. Subjects explored the auditory graph using a pen tablet to define their auditory viewpoint (i.e. position) on the topography. Evaluations were based on subjects’ ability to quickly appraise the underlying data topography along with aesthetic considerations regarding the impact of noise on the exploration task. Results showed that both tempo and pitch averaged PMSon reduced the perceived instabilities compared to pitch, while tempo preserved the response time of the sonification feedback.

Audio-Based Feedback Techniques for Teaching Touchscreen Gestures

While sighted users may learn to perform touchscreen gestures through observation (e.g., of other users or video tutorials), such mechanisms are inaccessible for users with visual impairments. As a result, learning to perform gestures without visual feedback can be challenging. We propose and evaluate two techniques to teach touchscreen gestures to users with visual impairments: (1) gesture sonification to generate sound based on finger touches, creating an audio representation of a gesture; and (2) corrective verbal feedback that combined automatic analysis of the user's drawn gesture with speech feedback. To refine and evaluate the techniques, we conducted three controlled laboratory studies. The first study, with 12 sighted participants, compared parameters for sonifying gestures in an eyes-free scenario. We identified pitch+stereo panning as the best combination. In the second study, ten blind and low-vision participants completed gesture replication tasks for single-stroke, multistroke, and multitouch gestures using the gesture sonification feedback. We found that multistroke gestures were more difficult to understand in sonification, but that playing each finger sound serially may improve understanding. In the third study, six blind and low-vision participants completed gesture replication tasks with both the sonification and corrective verbal feedback techniques. Subjective data and preliminary performance findings indicated that the techniques offer complementary advantages: although verbal feedback was preferred overall primarily due to the precision of its instructions, almost all participants appreciated the sonification for certain situations (e.g., to convey speed). This article extends our previous publication on gesture sonification by extending these techniques to multistroke and multitouch gestures. These findings provide a foundation for nonvisual training systems for touchscreen gestures.

A Prototype Audio-Tactile Map System with an Advanced Auditory Display

Tactile surfaces can display information in a variety of applications for all users, but can be of particular benefit to blind and visually impaired individuals. One example is the use of paper-based tactile maps as navigational aids for interior and exterior spaces; visually impaired individuals may use these to practice and learn a route prior to journeying. The addition of an interactive auditory display can enhance such interfaces by providing additional information. This article presents a prototype system which tracks the actions of a user's hands over a tactile surface and responds with sonic feedback. The initial application is an Audio-Tactile Map (ATM); the auditory display provides verbalised information as well as environmental sounds useful for navigation. Two versions of the interface are presented; a desktop version intended as a large-format information point and a mobile version which uses a tablet computer overlain with tactile paper. Details of these implementations are provided, including observations drawn from the participation of a partially-sighted individual in the design process. A usability test with five visually impaired subjects also gives a favourable assessment of the mobile version.

Naturalness influences the perceived usability and pleasantness of an interface’s sonic feedback

This study examined the influence of the naturalness of a sonic feedback on the perceived usability and pleasantness of the sounds used in a human-computer interface. The interface was the keyboard of an Automatic Teller Machine. The naturalness of the feedback was manipulated by using different kinds of relationship between a keystroke and its sonic feedback: causal, iconic, and arbitrary. Users were required to rate the naturalness, usability, and pleasantness of the sounds before and after manipulating the interface. Two kinds of interfaces were used: a normally functioning and a defective interface. The results indicated that the different relationships resulted in different levels of naturalness: causal mappings resulted in sounds perceived as natural, and arbitrary mappings in sounds perceived as non-natural, regardless of whether the sounds were recorded or synthesized. Before the subjects manipulated the interface, they rated the natural sounds as more pleasant and useful than the non-natural sounds. Manipulating the interface exaggerated these judgments for the causal and arbitrary mappings. The feedback sounds ruled by an iconic relationship between the user’s gesture and the resulting sounds were overall positively rated, but were sensitive to a potential contamination by the negative feelings created by a defective interface.