Virtual Sound Sources Research Articles

Virtual sound source construction based on adaptive crossfade processing with electro-dynamic and parametric loudspeaker arrays

Among the various high-presence sound field reproduction technologies, we focus on wave field synthesis (WFS) using a loudspeaker array, as it can construct a virtual sound source (VSS) with few restrictions. A VSS using parametric array loudspeakers (PALs) can provide more accurate direction perception than VSS using electro-dynamic loudspeakers (EDLs) because PALs hold sharp directivity by utilizing ultrasounds. However, the distance perception tends to be degraded. We previously proposed a VSS construction based on WFS with hybrid control of an EDL and a PAL arrays. This method enhances distance perception by exploiting the difference in radiation characteristics between PALs and EDLs. However, the difference in frequency responses between PAL and EDL is noticeable to the listener when switching between them. In this paper, to enable more smooth transition of the EDL and PAL arrays, we propose a VSS construction method based on adaptive crossfade processing with EDL and PAL arrays. For an arbitrary VSS construction position, the coefficients of crossfade for each frequency are determined by an adaptive filter based on measured impulse responses of VSSs with EDL and PAL arrays. We demonstrated the effectiveness of the proposed method through evaluation experiments.

An end-to-end approach for blindly rendering a virtual sound source in an audio augmented reality environment

Audio augmented reality (AAR), a prominent topic in the field of audio, requires understanding the listening environment of the user for rendering an authentic virtual auditory object. Reverberation time (RT60\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$RT_{60}$$\\end{document}) is a predominant metric for the characterization of room acoustics and numerous approaches have been proposed to estimate it blindly from a reverberant speech signal. However, a single RT60\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$RT_{60}$$\\end{document} value may not be sufficient to correctly describe and render the acoustics of a room. This contribution presents a method for the estimation of multiple room acoustic parameters required to render close-to-accurate room acoustics in an unknown environment. It is shown how these parameters can be estimated blindly using an audio transformer that can be deployed on a mobile device. Furthermore, the paper also discusses the use of the estimated room acoustic parameters to find a similar room from a dataset of real BRIRs that can be further used for rendering the virtual audio source. Additionally, a novel binaural room impulse response (BRIR) augmentation technique to overcome the limitation of inadequate data is proposed. Finally, the proposed method is validated perceptually by means of a listening test.

Control of Electric Wheelchair by Brain‐Computer Interface Using Mixed Reality and Virtual Sound Source

AbstractA brain‐computer interface for operating an electric wheelchair (wheelchair BCI) has been studied to support independent living for physically disabled patients. The current wheelchair BCI systems are facing the problem that the wheelchair cannot be moved or rotated to an arbitrary place or direction in a single measurement. To solve this problem, we developed a BCI that uses audiovisual stimuli based on mixed‐reality (MR) and virtual sound sources. During the online analysis, six out of seven participants were able to move or rotate the wheelchair to the target location or direction at a rate higher than the chance level. Thus, our results indicate that audiovisual stimulation with MR and virtual sound sources may be useful for intuitive wheelchair operation. © 2024 Institute of Electrical Engineer of Japan and Wiley Periodicals LLC.

Comparing the Auditory Distance and Externalization of Virtual Sound Sources Simulated Using Nonindividualized Stimuli.

When reproducing sounds over headphones, the simulated source can be externalized (i.e., perceived outside the head) or internalized (i.e., perceived within the head). Is it because it is perceived as more or less distant? To investigate this question, 18 participants evaluated distance and externalization for three types of sound (speech, piano, helicopter) in 27 conditions using nonindividualized stimuli. Distance and externalization ratings were significantly correlated across conditions and listeners, and when averaged across listeners or conditions. However, they were also decoupled in some circumstances: (1) Sound type had different effects on distance and externalization: the helicopter was evaluated as more distant, while speech was judged as less externalized. (2) Distance estimations increased with simulated distances even for stimuli judged as internalized. (3) Diotic reverberation influenced distance but not externalization. Overall, a source was not rated as externalized as soon as and only if its perceived distance exceeded a threshold (e.g., the head radius). These results suggest that distance and externalization are correlated but might not be aspects of a single perceptual continuum. In particular, a virtual source might be judged as both internalized and with a distance. Hence, it could be important to avoid using a scale related to distance when evaluating externalization.

Sound field control in a rectangular closed space using multiple selected room modes

Sound field control using multiple loudspeakers is essential for many applications such as sound field reproduction for delivering an immersive sound experience and noise control. These sound field controls are generally required in closed spaces, such as living rooms or factories, where eigenmodes are prominent at low frequencies. Many studies on the control of a few modes in closed spaces have been conducted in the field of noise control. In this study, we attempted to reproduce arbitrary sound fields, such as plane and spherical waves, from virtual sound sources using simultaneous multiple eigenmodes based on room-mode matching (RMM). Because most rooms are generally considered rectangular at low frequencies, we expanded the target sound field with discrete cosine transform (DCT) by approximating the eigenmodes as cosine functions. We proposed a method for extracting the control modes necessary to reproduce the desired sound field considering the mode strength at a target frequency. The simulation results verified that the proposed method can reproduce propagating plane waves with loudspeaker arrays placed on both sides of the closed space while maintaining the loudspeaker filter gains at appropriate values.

Co-immersion in Audio Augmented Virtuality: the Case Study of a Static and Approximated Late Reverberation Algorithm.

In immersive Audio Augmented Reality, a virtual sound source should be indistinguishable from the existing real ones. This property can be evaluated with the co-immersion criterion, which encompasses scenes constituted by arbitrary configurations of real and virtual objects. Thus, we introduce the term Audio Augmented Virtuality (AAV) to describe a fully virtual environment consisting of auditory content captured from the real world, augmented by synthetic sound generation. We propose an experimental design in AAV investigating how simplified late reverberation (LR) affects the co-immersion of a sound source. Participants listened to simultaneous virtual speakers dynamically rendered through spatial Room Impulse Responses, and were asked to detect the presence of an impostor, i.e., a speaker rendered with one of two simplified LR conditions. Detection rates were found to be close to chance level, especially for one condition, suggesting a limited influence on co-immersion of the simplified LR in the evaluated AAV scenes. This methodology can be straightforwardly extended and applied to different acoustics scenes, complexities, i.e., the number of simultaneous speakers, and rendering parameters in order to further investigate the requirements for immersive audio technologies in AAR and AAV applications.

Sound field navigation with virtual higher-order sound sources using complex greedy pursuit algorithm

As the Virtual Reality (VR) and metaverse industries continue to expand rapidly, there is a growing need to capture and recreate real-world experiences in immersive audio-visual scenes. Sound field translation, which activates equivalent virtual sources to recreate spatial sound fields, enables users to navigate through these scenes seamlessly. However, combining recordings from different microphones and processing mixed fields of exterior and interior sources present significant challenges. This paper introduces a novel method for virtual navigation by sparsely decomposing complex sound fields using distributed virtual higher-order sound sources and an iterative complex greedy pursuit algorithm. By combining spatially separated microphone recordings, the technique identifies its sparse representation with several higher-order virtual sources. This process effectively decomposes the complex sound field into a grid of higher-order sources, leveraging the power of the complex greedy pursuit algorithm. Through extensive experimentation, we demonstrate the suitability of our method for applications such as VR navigation and sound field reproduction with binaural devices. The results showcase enhanced realism and accuracy, offering users a truly immersive audio experience in virtual environments. As the demand for realistic audio-visual scenes grows, this innovative approach holds promise for advancing VR technology and enriching user experiences in the metaverse.

Influence of background noise on spatial audio localization for helicopter pilot headsets

A helicopter pilot is subjected to high workloads. A high noise level in the cockpit further increases stress. This stress can be reduced through headsets for hearing protection including active noise control (ANC). An actual research topic is the improvement of the human-machine-interface by 3-D-audio systems. These systems can improve pilot’s situational awareness and, therefore, reduce stress. This paper presents a 3-D-audio system for headphones. This binaural system uses head-tracking to process spatial information about the location of the sound source. The spatial sensation is achieved by convolving the source signal with head-related transfer functions. This sensation is influenced by background noise, which can be reduced by ANC. A first listening test with physical sound sources provides information on the localizability of sound sources at different elevation angles. This is compared to a second listening test with virtual sound sources generated by the 3-D-audio system. In a third listening test, the virtual sound source is located at variable azimuth and elevation angles. The system is evaluated with respect to sound source localization and the influence of background noise and ANC. This study highlights the potential of the 3-D-audio system to improve the helicopter pilot’s situational awareness in spatial acoustic environments.

Acoustically-mediated orientation and navigation in rooms

While the visual system provides the dominating sensory input in sighted humans for awareness, orientation, and navigation, in low-light conditions, smoke, or in visually-impaired and blind individuals, auditory perception becomes important. Acoustic information about the environment includes sounds radiated from sources, as well as reflections and reverberation in enclosed spaces in response to either external sound sources or self-produced sounds (e.g., mouth clicks), referred to as echolocation. Here, we investigated orientation and navigation in typical corridors based on acoustic cues only, using sighted humans without training and real-time virtual acoustics. Virtual sound sources and echo-location with predefined sounds as well as own vocalizations were used with the goal to identify suitable techniques for an acoustically augmented reality (AAR) based mobility aid. A hand-held acoustic pointer, rendering virtual sound sources at the closest wall in the pointing direction was best suited for orientation and navigation. For identification of the room shape, a performance similar to that obtained with a reference (visual) laser pointer could be achieved with the acoustic pointer in specific conditions. In addition to the above acoustic pointer, an AAR-based echolocation pointer with highly directed sound radiation, as achievable in the ultrasonic range, was additionally suggested.

Echo threshold measurement based on headphone presentation

Echo threshold(ET), as the upper threshold of the precedence effect, refers to the critical delay value when the listener transitions from perceiving a single fused sound image to two clear and separated sound images in the presence of two relatively delayed sounds. This study investigates the ET measurement using two kinds of headphone-based virtual sound source presentation methods, i.e., interaural time difference (ITD) lateralization and virtual auditory synthesis (VAS). The “lead-lag” paradigm was adopted in the experiment and an adaptive up-down procedure was used to measure ETs of four stimuli, including noise burst, speech, music and pink noise. The results demonstrate that ET is related to the type of signal and the presentation method. Long-duration signals (i.e., speech, music and pink noise) have greater ETs than short-duration signal(noise burst), which is due to the fact that the temporal overlap between lead and lag sound of long-duration signal would suppress the identification of the lag. ETs of ITD-based presentation were significantly greater than that of VAS-based method for long-duration signal, which indicate that different spatial cues produced by different presentation methods would influence the ET of long-duration signal.

The Accuracy of Dynamic Sound Source Localization and Recognition Ability of Individual Head-Related Transfer Functions in Binaural Audio Systems with Head Tracking

The use of audio systems that employ binaural synthesis with head tracking has become increasingly popular, particularly in virtual reality gaming systems. The binaural synthesis process uses the Head-Related Transfer Functions (HRTF) as an input required to assign the directions of arrival to sounds coming from virtual sound sources in the created virtual environments. Generic HRTFs are often used for this purpose to accommodate all potential listeners. The hypothesis of the research is that the use of individual HRTF in binaural synthesis instead of generic HRTF leads to improved accuracy and quality of virtual sound source localization, thus enhancing the user experience. A novel methodology is proposed that involves the use of dynamic virtual sound sources. In the experiments, the test participants were asked to determine the direction of a dynamic virtual sound source in both the horizontal and vertical planes using both generic and individual HRTFs. The gathered data are statistically analyzed, and the accuracy of localization is assessed with respect to the type of HRTF used. The individual HRTFs of the test participants are measured using a novel and efficient method that is accessible to a broad range of users.

Real-time co-location in dynamic virtual soundscapes across multi-occupant immersive rooms

Immersive rooms, a type of virtual reality system consisting of human-scale panoramic visual and acoustic display systems and distributed sensing apparatus for occupant motion, have been increasingly adopted for dynamic and interactive applications. While these applications enable multi-user audiovisual immersion and navigation from a single physical location, they have yet to propagate along multiple homogeneous system infrastructures in a networked manner. In this work, we intend to co-locate two physically-remote immersive rooms – at EMPAC and the CRAIVE-Lab, respectively – in a single system of shared environments developed in Unity and embedded with virtual soundscapes. This system actively monitors spatial properties of both immersive rooms’ dynamic virtual footprint and their corresponding occupants. It generates virtual sound sources. both procedurally and through spatially-aware user inputs. The sound sources are rendered in real time via an algorithm synthesizing a ray-traced early reflection window and a parameterized late reverberation estimate from in-scene geometries. The co-located virtual soundscapes, displayed in individual immersive rooms through their respective multi-channel wave field synthesis loudspeaker systems, are shared as such that the user interaction in one physical location has holistic effects on the experience of virtual environments across all associated physical locations. [Work supported by NSF IIS-1909229 & CNS-1229391.]

Distance control of virtual sound source based on switching electro-dynamic and parametric loudspeaker arrays

High presence sound field reproduction technology has recently gained attention. We have previously proposed a virtual sound source (VSS) construction method based on wave field synthesis (WFS) using parametric loudspeaker array (PAL array). The previously proposed method can construct the VSS at any position without relocating the PALs. However, the previously proposed method has difficulty constructing the VSS near the PAL array. In this paper, we propose a new method to control the distance of a VSS based on WFS using an electro-dynamic loudspeaker (EDL) array and a PAL array. The EDL array is suitable to construct VSSs near the loudspeaker array. In contrast, the PAL array is suitable to construct VSSs far away from the loudspeaker array. Therefore, in this method, we place an EDL array and a PAL array parallel to each other and switch the loudspeaker array for VSSs construction depending on the desired position to realize VSS distance control. The threshold of switching loudspeaker array is determined by a subjective preliminary experiment. We demonstrated the effectiveness of the proposed method through evaluation experiments.

Virtual sound source perception challenges of binaural audio systems with head-tracking

With the recent leaps in spatial audio technology, the use of binaural head-tracking for spatial audio can revolutionize the way how music and audio are experienced. Moreover, to research noise-related perception in laboratories, binaural head-tracking is frequently used as an audio reproduction system. Although this technology is getting significantly better in recent years, signal processing of binaural audio is often experiencing problems such as non-adequate sound source externalization and unacceptably high values of system response time since it influences the usability of the technology at fast head rotation. In this paper, the results of an experiment are presented, in which test subjects are determining the direction of a virtual sound source in the horizontal plane with multiple parameter changes. The experiment is done in a controlled environment in a listening room with known acoustical properties. Parameter variation includes: hardware head-tracker variation (commercial one and head-tracker based on simple embedded system), various software solutions for real-time binaural synthesis, and variation in the used Head-Related Transfer Functions. The problem of externalization of a virtual sound source via headphones is discussed, and possible solutions to the problem are given. Results of the experiment and recorded data are presented.

Evaluation of Bone Conduction and Active-Noise-Cancellation Headsets Based on Listening Tests in a Virtual Environment

Alternative design headsets incorporating activenoise-cancellation or bone conduction were evaluated in listening tests in a virtual reality environment. Virtual sound sources in the horizontal plane had to be identified using stereo panning in the frontal hemisphere. In addition, transfer characteristics and damping effects were measured with a dummy-head. Results indicate that up to five source locations can be used in real applications with high accuracy in virtual scenarios, independent of the spectral content of the excitation signals. Furthermore, the use of noise cancellation in presence of 80 dB background noise does not improve performance. Commercially available bone conduction headsets can provide the same detection accuracy even if the subjective sound quality is lower.

Impact of wearing a head-mounted display on localization accuracy of real sound sources

For augmented reality experiences, users wear head-mounted displays (HMD) while listening to real and virtual sound sources. This paper assesses the impact of wearing an HMD on localization accuracy of real sources. Eighteen blindfolded participants completed a localization task on 32 loudspeakers while wearing either no HMD, a bulky visor HMD, or a glass visor HMD. Results demonstrate that the HMDs had a significantly impact on participants’ localization performance, increasing local great circle angle error by 0.9°, and that the glass visor HMD demonstrably increased the rate of up–down confusions in the responses by 0.9–1.1%. These results suggest that wearing an HMD has a sufficiently small impact on real source localization that it can safely be considered as an HMD-free condition in most but the most demanding AR auditory localization studies.

Steady-state auditory motion based potentials evoked by intermittent periodic virtual sound source and the effect of auditory noise on EEG enhancement

Hearing is one of the most important human perception forms, and humans can capture the movement of sound in complex environments. On the basis of this phenomenon, this study explored the possibility of eliciting a steady-state brain response in an intermittent periodic motion sound source. In this study, a novel discrete continuous and orderly change of sound source positions stimulation paradigm was designed based on virtual sound using head-related transfer functions (HRTFs). And then the auditory motion stimulation paradigms with different noise levels were designed by changing the signal-to-noise ratio (SNR). The characteristics of brain response and the effects of different noises on brain response were studied by analyzing electroencephalogram (EEG) signals evoked by the proposed stimulation. Experimental results showed that the proposed paradigm could elicit a novel steady-state auditory evoked potential (AEP), i.e., steady-state motion auditory evoked potential (SSMAEP). And moderate noise could enhance SSMAEP amplitude and corresponding brain connectivity. This study enriches the types of AEPs and provides insights into the mechanism of brain processing of motion sound sources and the impact of noise on brain processing.

Auditory orientation and distance estimation of sighted humans using virtual echolocation with artificial and self-generated sounds.

Active echolocation of sighted humans using predefined synthetic and self-emitted sounds, as habitually used by blind individuals, was investigated. Using virtual acoustics, distance estimation and directional localization of a wall in different rooms were assessed. A virtual source was attached to either the head or hand with realistic or increased source directivity. A control condition was tested with a virtual sound source located at the wall. Untrained echolocation performance comparable to performance in the control condition was achieved on an individual level. On average, the echolocation performance was considerably lower than in the control condition, however, it benefitted from increased directivity.

Asymmetric visual capture of virtual sound sources in the distance dimension.

Visual capture describes the tendency of a sound to be mislocalized to the location of a plausible visual target. This effect, also known as the ventriloquist effect, has been extensively studied in humans, but primarily for mismatches in the angular direction between auditory and visual targets. Here, visual capture was examined in the distance dimension using a single visual target (an un-energized loudspeaker) and invisible virtual sound sources presented over headphones. The sound sources were synthesized from binaural impulse-response measurements at distances ranging from 1 to 5 m (0.25 m steps) in the semi-reverberant room (7.7 × 4.2 × 2.7 m3) in which the experiment was conducted. Listeners (n = 11) were asked whether or not the auditory target appeared to be at the same distance as the visual target. Within a block of trials, the visual target was placed at a fixed distance of 1.5, 3, or 4.5 m, and the auditory target varied randomly from trial-to-trial over the sample of measurement distances. The resulting psychometric functions were generally consistent with visual capture in distance, but the capture was asymmetric: Sound sources behind the visual target were more strongly captured than sources in front of the visual target. This asymmetry is consistent with previous reports in the literature, and is shown here to be well predicted by a simple model of sensory integration and decision in which perceived auditory space is compressed logarithmically in distance and has lower resolution than perceived visual space.

Acoustically driven orientation and navigation in enclosed spaces.

Awareness of space, and subsequent orientation and navigation in rooms, is dominated by the visual system. However, humans are able to extract auditory information about their surroundings from early reflections and reverberation in enclosed spaces. To better understand orientation and navigation based on acoustic cues only, three virtual corridor layouts (I-, U-, and Z-shaped) were presented using real-time virtual acoustics in a three-dimensional 86-channel loudspeaker array. Participants were seated on a rotating chair in the center of the loudspeaker array and navigated using real rotation and virtual locomotion by "teleporting" in steps on a grid in the invisible environment. A head mounted display showed control elements and the environment in a visual reference condition. Acoustical information about the environment originated from a virtual sound source at the collision point of a virtual ray with the boundaries. In different control modes, the ray was cast either in view or hand direction or in a rotating, "radar"-like fashion in 90° steps to all sides. Time to complete, number of collisions, and movement patterns were evaluated. Navigation and orientation were possible based on the direct sound with little effect of room acoustics and control mode. Underlying acoustic cues were analyzed using an auditory model.