Auditory Interface Research Articles

Re-imagining the future state of the ventricular assist device controller interface through human-centered design.

Ventricular assist devices (VADs) are effective therapy for patients with end-stage heart failure. Current VAD controllers offer improved interactivity, yet limitations of the visual, tactile, and auditory interface persist that impact patient experience and quality of life (QoL). This study explores how VAD controllers can be redesigned using a human-centered design approach to enhance the emotional and functional experience of the device for patients. VAD patients (n = 21), caregivers (n = 4) and healthcare practitioners (n = 24) were interviewed to uncover design opportunities. From this, a series of realistic scenarios to design for emerged. A "design by analogy" method took inspiration from existing consumer products to ideate new functionality for the VAD wearable system, creating concepts for a controller interface and paired wearable device. An additional 15 patients and 2 caregivers were engaged to explore current VAD controller experiences and evaluate the future-state concepts. This research validated the need for increased automation and emergency functionality in VAD controllers, including remote monitoring of data, accurate communication of battery status, and automated medical alerts for critical device alarms. "Manage my health," "Feeling normal," "Social belonging," "Feeling safe," and "Sense of control" emerged as key patient concerns to be met by future VAD controller designs. The study demonstrated an innovative and relevant approach to improve usability of future VAD peripherals. By considering both emotional and functional perspectives in the design of lifesaving medical devices such as VADs, device manufacturers can uncover new opportunities to improve patient QoL through improved user experiences.

Hey, What's Going On?

Highly Automated Vehicles offer a new level of independence to people who are blind or visually impaired. However, due to their limited vision, gaining knowledge of the surrounding traffic can be challenging. To address this issue, we conducted an interactive, participatory workshop (N=4) to develop an auditory interface and OnBoard- a tactile interface with expandable elements - to convey traffic information to visually impaired people. In a user study with N=14 participants, we explored usability, situation awareness, predictability, and engagement with OnBoard and the auditory interface. Our qualitative and quantitative results show that tactile cues, similar to auditory cues, are able to convey traffic information to users. In particular, there is a trend that participants with reduced visual acuity showed increased engagement with both interfaces. However, the diversity of visual impairments and individual information needs underscores the importance of a highly tailored multimodal approach as the ideal solution.

How manoeuvre information via auditory (spatial and beep) and visual UI can enhance trust and acceptance in automated driving

In conditionally automated driving (SAE level 3), drivers may take their eyes off the road but will still need to be ready to take control and will, therefore, benefit from information on automation. This study aims to investigate the effectiveness of automation manoeuvre information provided through spatial sound, traditional notification sound (beep), and a visual interface. Spatial sounds were designed differentiating four distinct driving manoeuvres: overtaking a leading car, slowing down, turning right, and passing a roundabout. The notification sound consisted of one beep being identical for all manoeuvres. The visual interface showed the automation mode with an image and manoeuvre information with text and images. The impact of these interfaces on trust, workload, acceptance, situation awareness, and sense of control was evaluated with questionnaires and visual attention was evaluated with eye tracking while participants engaged in a visual-motor secondary task in a driving simulator. The results indicate that, with all interfaces tested, manoeuvre information enhances trust, acceptance, situation awareness, and sense of control, without significantly affecting the overall workload. These benefits were more profound, adding auditory information and differed marginally between the traditional notification and the spatial sound, as the effectiveness of the different auditory interface types varied depending on the specific manoeuvre. Findings highlight the importance of designing user interfaces for automation manoeuvre information using auditory cues to improve the user experience in automated driving.

Navigating Hierarchical Menus Through Sound

Much of the current user experience navigating digital information relies on visual displays. Providing auditory alternatives enables access for visually impaired and blind users. The current study evaluated hierarchical navigation in a novel spatialized auditory interface compared to a screen reader and visual navigation mode. Past studies suggest that spatialized audio may provide performance improvements. Sixteen participants navigated menu structures of varying depth / breadth to select a target item with three different interface styles (spatial audio, screen reader, and visual). Time-to-completion, errors, and SUS scores were compared across interfaces. Results showed that spatial audio was significantly slower, more error prone, and less usable than the other conditions. However, the problems of spatial audio might be overcome with simple changes in the interaction mode and optimization of display space. Our experience shows that designers must solve the problem of auditory clutter and spatial selection to achieve usable auditory navigation.

Localization of novel reduced-annoyance audio interface sounds

Foley et al. [JASA 151, 3189–3196 (2022)] have demonstrated that annoyance of auditory interface sounds can be reduced both by shortening the duration of upper harmonics and by applying percussive rather than flat amplitude envelopes. However, sounds in that study had a maximum frequency of 2400 Hz, which would likely affect localization based on spectral cues. In a setting with multiple interface devices (e.g., a multi-patient hospital ward), localization of interface sounds is a concern. We characterized normally hearing listeners' ability to localize a variety of candidate reduced-annoyance interface sounds (flat or percussive envelopes; durations between 360 and 1600 ms; all but one with uppermost harmonic limited to 2400 Hz) in quiet or at + 4-dB and −11-dB SNR in spatially diffuse multi-talker babble. Listeners stood at the center of a 360-degree loudspeaker array in a darkened anechoic chamber and used a head-pointing response to report the perceived location of each target. Decreasing SNR increased response variability and the rate of front/rear confusions. For all sounds with restricted bandwidth, front/rear confusions were frequent in the absence of head movements, but when head movements were initiated before target offset, confusions were substantially reduced. The results highlight the need to consider localizability when designing improved auditory interface sounds.

Influence of Multi-Modal Warning Interface on Takeover Efficiency of Autonomous High-Speed Train.

As a large-scale public transport mode, the driving safety of high-speed rail has a profound impact on public health. In this study, we determined the most efficient multi-modal warning interface for automatic driving of a high-speed train and put forward suggestions for optimization and improvement. Forty-eight participants were selected, and a simulated 350 km/h high-speed train driving experiment equipped with a multi-modal warning interface was carried out. Then, the parameters of eye movement and behavior were analyzed by independent sample Kruskal-Wallis test and one-way analysis of variance. The results showed that the current level 3 warning visual interface of a high-speed train had the most abundant warning graphic information, but it failed to increase the takeover efficiency of the driver. The visual interface of the level 2 warning was more likely to attract the attention of drivers than the visual interface of the level 1 warning, but it still needs to be optimized in terms of the relevance of and guidance between graphic-text elements. The multi-modal warning interface had a faster response efficiency than the single-modal warning interface. The auditory-visual multi-modal interface had the highest takeover efficiency and was suitable for the most urgent (level 3) high-speed train warning. The introduction of an auditory interface could increase the efficiency of a purely visual interface, but the introduction of a tactile interface did not improve the efficiency. These findings can be used as a basis for the interface design of automatic driving high-speed trains and help improve the active safety of automatic driving high-speed trains, which is of great significance to protect the health and safety of the public.

More detectable, less annoying: Temporal variation in amplitude envelope and spectral content improves auditory interface efficacy

Auditory interfaces, such as auditory alarms, are useful tools for human computer interaction. Unfortunately, poor detectability and annoyance inhibit the efficacy of many interface sounds. Here, it is shown in two ways how moving beyond the traditional simplistic temporal structures of normative interface sounds can significantly improve auditory interface efficacy. First, participants rated tones with percussive amplitude envelopes as significantly less annoying than tones with flat amplitude envelopes. Crucially, this annoyance reduction did not come with a detection cost as percussive tones were detected more often than flat tones-particularly, at relatively low listening levels. Second, it was found that reductions in the duration of a tone's harmonics significantly lowered its annoyance without a commensurate reduction in detection. Together, these findings help inform our theoretical understanding of detection and annoyance of sound. In addition, they offer promising original design considerations for auditory interfaces.

Model study of target discrimination in concurrent auditory events

With the development of automation technology in human-machine interface, the traditional visual display no longer met the needs of complicated control systems, while auditory display has become an important part of human-machine interface design. Analysing the auditory attention selection behaviours in concurrent auditory events is essential for the design and evaluation of the auditory interface. Listeners tend to pay attention to auditory target with physical attributes which were more salient and informative with deep cognitive level. Listeners tend to focus on the important information in the concurrent auditory events. By combining the two tendencies of the selective attention mechanism, the binary Logit model was built to analyse human auditory selective attention (ASA) behaviour, and the auditory target's selection can be transformed into a probability event. The continuous speech-shaped noise (SSN) was described by the concurrent auditory events scene, which is a typical target discrimination task for ASA. An experiment using the target discrimination task in SSN was conducted to verify the model. The results showed that the listener's attention behaviour was in good agreement with the proposed Logit model. The proposed ASA model is effective and reasonable, which is promising for use in behaviour analysis, auditory display optimization, and interface design.

High Time for Temporal Variation: Improving Sonic Interaction with Auditory Interfaces

Listening is a powerful, under-appreciated, mechanism for acquiring information about the world. Changes in the hum of a car engine can offer an early clue about potential problems; the rustle of leaves informs us of weather conditions even before checking the forecast; slight fluctuations in the sound of a kettle tell us the water is almost boiled. These are but a few examples of important sonic interactions we have with our environment throughout the day.

HuRAI: A brain-inspired computational model for human-robot auditory interface

The deep learning era endows immense opportunities for ubiquitous robotic applications by leveraging big data generated from widespread sensors and ever-growing computing capability. While the growing demands for natural human-robot interaction (HRI) as well as concerns for energy efficiency, real-time performance, and data security motive novel solutions. In this paper, we present a brain-inspired spiking neural network (SNN) based Human-Robot Auditory Interface, namely HuRAI. The HuRAI integrates the voice activity detection, speaker localization and voice command recognition systems into a unified framework that can be implemented on the emerging low-power neuromorphic computing (NC) devices. Our experimental results demonstrate superior modeling capabilities of SNNs, achieving accurate and rapid prediction for each task. Moreover, the energy efficiency analysis reveals a compelling prospect, with up to three orders of magnitude energy savings, over the equivalent artificial neural networks that running on the state-of-the-art Nvidia graphics processing unit (GPU). Therefore, integrating the algorithmic power of large-scale SNN models and the energy efficiency of NC devices offers an attractive solution for real-time, low-power robotic applications.

Adaptive Auditory Feedback Mechanism of Visual Impaired

There is a lot of work done in HCI research area, due to this auditory interface develops for visually impaired people that assist them to work with desktop environment easily. To enhance better design, designer faces many challenges to develop auditory feedback system. This study purposed an new method named as adaptive auditory feedback system, which is continuous switching of speech-based feedback system (i.e. sound, music) to non-speech-based feedback system (i.e. spearcone) according to user state of interaction. As a visually impaired person who frequently access desktop system does not need continuous instructions. Experiment done on sixteen visually impaired people and result shows that adaptive auditory feedback system is efficient then only one particular method (speech-based or non-speech-based). User can choose any type of instruction according to need and want that improve usability experience. Result and discussion shows highest satisfaction level of visually impaired people regarding adaptive auditory feedback. In future, same type of adaptive auditory feedback system may implement on mobile applications rather than desktop environment.

The impact of auditory continual feedback on take-overs in Level 3 automated vehicles

ObjectiveTo implement auditory continual feedback into the interface design of a Level 3 automated vehicle and to test whether gaze behavior and reaction times of drivers improved in take-over situations. BackgroundWhen required to assume manual control in take-over situations, drivers of Level 3 automated vehicles are less likely than conventional drivers to spot potential hazards, and their reaction time is longer. Therefore, it is crucial that the interface of Level 3 automated vehicles will be designed to improve drivers’ performance in take-over situations. MethodIn two experiments, participants drove a simulated route in a Level 3 automated vehicle for 35 min with one imminent take-over event. Participants’ gaze behavior and performance in an imminent take-over event were monitored under one of three auditory interface designs: (1) Continual feedback. A system that provides verbal driving-related feedback; (2) Persistent feedback. A system that provides verbal driving-related feedback and a persistent beep; and (3) Chatter feedback. A system that provides verbal non-driving-related feedback. Also, there was a control group without feedback. ResultsUnder all three auditory feedback designs, the number of drivers' on-road glances increased compared to no feedback, but none of the designs shortened reaction time to the imminent event. ConclusionIncreasing the number of on-road glances during automated driving does not necessarily improve drivers’ attention to the road and their reaction times during take-overs. ApplicationPossible implications for the effectiveness of auditory continual feedback should be considered when designing interfaces for Level 3 automated vehicles.

Re-Sounding Alarms: Designing Ergonomic Auditory Interfaces by Embracing Musical Insights.

Auditory alarms are an important component of human–computer interfaces, used in mission-critical industries such as aviation, nuclear power plants, and hospital settings. Unfortunately, problems with recognition, detection, and annoyance continue to hamper their effectiveness. Historically, they appear designed more in response to engineering constraints than principles of hearing science. Here we argue that auditory perception in general and music perception in particular hold valuable lessons for alarm designers. We also discuss ongoing research suggesting that the temporal complexity of musical tones offers promising insight into new ways of addressing widely recognized shortcomings of current alarms.

Auditory displays and auditory user interfaces: art, design, science, and research

For almost 3 decades, research on auditory displays and sonification has been well advanced. Now, the auditory display community has arrived at the stage of sonic information design with a more systematic, refined necessity, going beyond random mappings between the referents and sounds. Due to its innate transdisciplinary nature of auditory display, it would be difficult to unify the methods to study it. This special issue covers a diverse collection of approaches to auditory displays, involving art, design, science, and research. Accordingly, the works in the present special issue included new theories, frameworks, methods, and applications about auditory displays and auditory user interfaces. We hope that this special issue can provide the state of art of auditory display research and auditory user interface design, offering fresh inspiration and motivation to researchers and designers for their future works.

Auditory user interface design guidelines for control transitions in the conditional self-driving phase

Auditory user interface design guidelines for control transitions in the conditional self-driving phase

Evaluations of Different Human Machine Interfaces for Presenting Right-Turn Timing at Intersections

Several driver assistance systems have been proposed to assist drivers to avoid collisions at intersections, by informing drivers about the existence of oncoming vehicles. However, drivers still need to judge the right-turn timing by themselves. This study proposed a right-turn timing assistance system. Different human machine interfaces of the system, including haptic, visual and auditory methods, were evaluated by analyzing the influences on driver behaviors in actual vehicle experiments. It was observed that the driver’s reaction time could be significantly reduced when the right-turn timing assistance was provided with haptic and auditory interfaces alone or the combinations of haptic, auditory and visual interfaces, compared to the condition without assistance. The meaning of right-turn timing assistance was significant easier for drivers to understand when it was presented with visual interface than auditory interface. Meanwhile, the post-encroachment time value was not smaller than 5 s when the assistance was presented with the haptic, visual and auditory interfaces together, which indicated a small risk of collision with the oncoming vehicles.

A motion imitation system for humanoid robots with inference-based optimization and an auditory user interface

Humanoid robots are expected to perform various jobs replacing human beings, but it is difficult for non-expert users to control them. To control humanoid robots easily, various motion imitation systems, in which a humanoid robot does the same motion as a human teacher, have been studied. In the present study, we make two contributions to building easy-to-use imitation systems. The first contribution is a flexible learning system that enables both synchronous real-time imitation, in which the robot imitates human motion almost immediately, and batch imitation, in which the robot’s motion is optimized more accurately after longer measurement of human motion. We use an efficient inference-based optimization method to enable both learning modes. The other contribution is an auditory user interface that accepts voice commands from the user and at the same time identifies the target person to be imitated. This enables the user to easily use the imitation system even when other people are present. Experimental results show that the proposed system can be used in both imitation modes and that it can identify the target person.

"실버세대를 고려한 청각 인터페이스 (AUI: Auditory User Interface) 디자인 방법론"

"실버세대를 고려한 청각 인터페이스 (AUI: Auditory User Interface) 디자인 방법론"

Keeping the driver in the loop: Dynamic feedback to support appropriate use of imperfect vehicle control automation

Abstract Objective This study evaluates the benefits and costs associated with providing drivers continuous feedback on the limits and behavior of imperfect vehicle control automation. Background In-vehicle automated systems remove drivers from active vehicle control, often at the expense of timely interventions when failures occur. Discrete warnings, as a type of feedback to inform drivers about automated system behavior, fail to keep drivers aware of its proximity to operating limits. Method In a fixed-based simulator, 48 drivers drove using Adaptive Cruise Control (ACC)—a form of control automation that maintains a set speed, or a set headway if the vehicle encounters a slower moving vehicle. A first experiment compared ACC with discrete warnings to ACC with continuous information, which indicated moment-to-moment ACC state relative to its operating limits. Three display conditions, designed to provide non-obtrusive, ecologically-valid information, were evaluated in a second experiment: 1) a visual interface; 2) an auditory interface; and 3) a combined visual-auditory interface. Results Drivers provided with continuous feedback relied more appropriately on ACC than did those with discrete warnings. Continuous feedback increased the frequency of proactive responses to automation failures and improved system understanding. Of the three displays, the combined visual-auditory interface performed the best. Conclusion Continuous feedback helped communicate to drivers the evolving relationship between system performance and operating limits. Application Displays for increasingly automated vehicles should inform about the automation's situation-specific behavior rather than simply alert drivers to failures and/or the need to resume vehicle control in order to promote appropriate understanding and trust.

The Relationship Between Rhythm Variation and Distance Perception in Auditory In-Vehicle Interfaces

The increasing use of smartphones in parallel to driving presents great risks to drivers, who split their attention from the road to inside the vehicle. It is arguably, then, that sound may be an efficient alternative to communicating information of in-vehicle interfaces to drivers. This paper presents a study that aimed to investigate the relationship between rhythm variation in earcons and drivers’ perception of distance, as well as the impact of previous experience with in-vehicles’ auditory interfaces on the perception of this relation. The results showed that participants can establish a consistent and direct relationship between rhythm variation and distance and that participants with previous experience with auditory interfaces presented smaller mean perceived distances than inexperienced participants when exposed to alarms with the same rhythms.