Binaural Signals Research Articles

The contribution of dynamic cues to externalization of virtual source in virtual auditory display

It is well known that static virtual auditory display with headphone is inclined to create the auditory perception of in-head-localization, especially for target virtual sources in the median plane. By introducing the dynamic variation of binaural pressures caused by head turning, dynamic virtual auditory display eliminates the in-head-localization effectively. The present work investigates the contributions of dynamic interaural time difference (ITD) and dynamic interaural level difference (ILD) in binaural pressures to the externalization of auditory perception. By modifying the head-related transfer functions used in a dynamic virtual auditory display, binaural signals with dynamic ITD or dynamic ILD alone are created and presented by headphone. A psychoacoustic experiment is conducted to evaluate the extent of perceived externalization. The experiment results prove that, for the target virtual source in the median plane, dynamic ITD and dynamic ILD contribute to the externalization of low and high frequency components of the stimuli respectively. For wideband stimulus with low and high frequency components, both dynamic ITD and dynamic ILD contribute externalization but dynamic ITD contributes more than that of dynamic ILD.

Evaluating on the spatial localization cues of binaural sound pressure recorded via behind-the-ear microphones in a hearing aid

Spatial hearing ability enables people with normal hearing to localize a sound source and utilizes the cocktail party effect to detect the target speech information in a noise environment. Hearing-impairment causes not only hearing loss but also the degradation of spatial auditory ability. Hearing aid is desired to improve the spatial auditory ability of some hearing-impaired people, in addition to compensating hearing loss. However, the microphones for conventional behind-the-ear (BTE) hearing aid are located behind the pinnae rather than in the entrances of ear canal, which may cause distortion in spatial cues in the recorded binaural signals. The present work comprehensively evaluates the spatial hearing cues recorded by BTE-microphones. The head-related transfer functions (HRTF) for a KEMAR artificial head and eight human subjects, at three BTE positions and the entrances of blocked ear-canal, are measured respectively. Analysis from the HRTF data indicates that BTE microphone recording basically reserves the interaural time difference and interaural level difference as lateral-direction information. It also reserves the dynamic cues, but obvious distortions appear in the spectral cues at high frequencies. However, due to the redundancy among different spatial information, it can be inferred that dynamic cues alone may provide appropriate front-back and vertical-direction information.

Benefit of Hearing-Aid Amplification and Signal Enhancement for Speech Reception in Complex Listening Situations.

A major goal of hearing-device provision is to improve communication in daily life. However, there is still a large gap between the user's daily-life aided listening experience and hearing-aid benefit as assessed with established speech reception measurements in the laboratory and clinical practice. For a more realistic assessment, hearing-aid provision needs to be tested in suitable acoustic environments. In this study, using virtual acoustics, we developed complex acoustic scenarios to measure speech-intelligibility and listening-effort benefit obtained from hearing-aid amplification and signal enhancement strategies. Measurements were conducted using the participants' own devices and a research hearing aid, the Portable Hearing Laboratory (PHL). On the PHL, in addition to amplification, a monaural and a binaural directional filter, as well as a spectral filter were employed. We assessed the benefit from different signal enhancement strategies at the group and the individual level. At the group level, signal enhancement including directional filtering provided a higher hearing-aid benefit in challenging acoustic scenarios in terms of speech intelligibility compared to amplification alone or combined with spectral filtering. However, no difference between monaural and binaural signal enhancement occurred. On an individual level, we found large differences in hearing-aid benefit between participants. While some benefitted from signal-enhancement algorithms, others benefitted from amplification alone, but additional signal enhancement had a detrimental effect. This shows the importance of an individual selection of signal enhancement strategies as a part of the hearing-aid fitting process.

Non-intrusive speech intelligibility prediction using an auditory periphery model with hearing loss

Speech intelligibility prediction methods are necessary for hearing aid development. However, many such prediction methods are categorized as intrusive metrics because they require reference speech as input, which is often unavailable in real-world situations. Additionally, the processing techniques in hearing aids may cause temporal or frequency shifts, which degrade the accuracy of intrusive speech intelligibility metrics. This paper proposes a non-intrusive auditory model for predicting speech intelligibility under hearing loss conditions. The proposed method requires binaural signals from hearing aids and audiograms representing the hearing conditions of hearing-impaired listeners. It also includes additional acoustic features to improve the method's robustness in noisy and reverberant environments. A two-dimensional convolutional neural network with neural decision forests is used to construct a speech intelligibility prediction model. An evaluation conducted with the first Clarity Prediction Challenge dataset shows that the proposed method performs better than the baseline system.

Effects of visual stimuli on auditory separation of sound images spatially split by synthesized binaural signal

In order to clarify the effect of visual stimuli on the spatially split perception of sound images by interaural differences, we conducted spatially split perception experiments under conditions in which visual stimuli were presented on a head mounted display. The auditory stimuli were synthesized binaural signals consisting of two uncorrelated pink noises convolved with the receiver's own head related impulse responses. As manipulated by the listener using the mouse scroll-wheel, the visual and auditory stimuli spread to the left and right in front of the listener, and the listener was made to answer at what point the sound images were spatially split into two. As a result, the detection limit was smaller in the conditions with split visual stimuli than in the condition without visual stimulus. These results suggest that visual stimuli that give the impression of separation reduce the detection limit of the auditorily perceived spatially split sound images.

Auditory vertical localization in the median plane with conflicting dynamic interaural time difference and other elevation cues.

Both dynamic variation of interaural time difference (ITD) and static spectral cues provide information for front-back discrimination and vertical localization. However, the contributions of the two cues are still unclear. The static spectral cue has conventionally been regarded as the dominant one. In the present work, psychoacoustic experiments were conducted to examine the contribution of dynamic ITD and static spectral cues to vertical localization in the median plane. By modifying the head-related transfer functions used in a dynamic virtual auditory display, binaural signals with conflicting dynamic ITD and spectral cues that were either static or dynamically modified according to instantaneous head position were created. The results indicated that the dynamic ITD and static spectral cues contribute to vertical localization at low and high frequencies, respectively. For full a bandwidth stimulus, conflicting dynamic ITD and static spectral cues usually result in two separated virtual sources at different elevations corresponding to the spatial information conveyed by the low- and high-frequency bands, respectively. In most cases, no fused localization occurs in the high-level cognition system. Therefore, dynamic ITD and static spectral cues contribute to vertical localization at different frequency ranges, and neither of them dominates vertical localization in the case of wideband stimuli.

Effect of Individualized Head-Related Transfer Functions on Distance Perception in Virtual Reproduction for a Nearby Sound Source

The head-related transfer function (HRTF) is dependent on the position of the sound source (both direction and distance) and is also affected by individual anatomical parameters. Individualized HRTFs have been shown to affect the perception of sound direction, but have not been considered in distance perception. This work aims to discover, by means of psychoacoustic experiments for a virtual reproduction system through a pair of in-ear headphones, the effect of individualized HRTF on auditory distance perception for a nearby sound source. The individualized HRTFs of six subjects and the non-individualized HRTFs of a mannequin at seven distances between 0.2 and 1.0 m and five lateral azimuths between 45X and 135X in the horizontal plane were processed with white noise to generate binaural signals. Further, the individualized and non-individualized HRTFs were used in the auditory distance perception experiments. Results of distance perception show that the variance of distance perception results among subjects is significant, the reason could be the stimuli are lack of dynamic cue and early reflections, or the auditory difference of distance perception among subjects. However, via the analyses of mean slope of perceptual distance and correlation between the perceptual and real distance, we find that the individualized HRTF cue has insignificant influence on distance perception.

Spatial location from running binaural signals using cepstral analysis and logarithmic filtering

This paper proposes an acoustic model for predicting the acoustical room characteristics from a running binaural signal. This is accomplished via training a convolutional neural network on a precedence effect model to extract the spatial locations of the direct sound source and its early reflections. The precedence effect model extends and modifies the BICAM algorithm with cepstral analysis [Tyler, J., Si, M., \& Braasch, J. Acoust. Soc. Am. 151] and a logarithmic filter. The logarithmic filter takes human perception into account and provides better separation at higher frequencies. A synthetic dataset of binaural signals was generated using anechoic orchestral recordings with added reflections and reverberations. The binaural model generates binaural activity maps from binaural input signals, which are then used to train a convolutional neural network. The ability to predict the traits of a direct sound source and its reflections has applications in academic areas like perceptual modeling and room acoustical analysis. It can also be applied to industrial areas such as television and movies, video games, and augmented and virtual reality, to name a few. [Work supported by the National Science Foundation: HCC-1909229.]

Inferring the basis of binaural detection with a modified autoencoder.

The binaural system utilizes interaural timing cues to improve the detection of auditory signals presented in noise. In humans, the binaural mechanisms underlying this phenomenon cannot be directly measured and hence remain contentious. As an alternative, we trained modified autoencoder networks to mimic human-like behavior in a binaural detection task. The autoencoder architecture emphasizes interpretability and, hence, we "opened it up" to see if it could infer latent mechanisms underlying binaural detection. We found that the optimal networks automatically developed artificial neurons with sensitivity to timing cues and with dynamics consistent with a cross-correlation mechanism. These computations were similar to neural dynamics reported in animal models. That these computations emerged to account for human hearing attests to their generality as a solution for binaural signal detection. This study examines the utility of explanatory-driven neural network models and how they may be used to infer mechanisms of audition.

A model framework for simulating spatial hearing of bilateral cochlear implant users

Bilateral cochlear implants (CIs) greatly improve spatial hearing acuity for CI users, but substantial gaps still exist compared to normal-hearing listeners. For example, CI users have poorer localization skills, little or no binaural unmasking, and reduced spatial release from masking. Multiple factors have been identified that limit binaural hearing with CIs. These include degradation of cues due to the various sound processing stages, the viability of the electrode-neuron interface, impaired brainstem neurons, and deterioration in connectivity between different cortical layers. To help quantify the relative importance and inter-relationship between these factors, computer models can and arguably should be employed. While models exploring single stages are often in good agreement with selected experimental data, their combination often does not yield a comprehensive and accurate simulation of perception. Here, we combine information from CI sound processing with computational auditory model stages in a modular and open-source framework, resembling an artificial bilateral CI user. The main stages are (a) binaural signal generation with optional head-related impulse response filtering, (b) generic CI sound processing not restricted to a specific manufacturer, (c) electrode-to-neuron transmission, (d) binaural interaction, and (e) a decision model. The function and the outputs of different model stages are demonstrated with examples of localization experiments. However, the model framework is not tailored to a specific dataset. It offers a selection of sound coding strategies and allows for third-party model extensions or substitutions; thus, it is possible to employ the model for a wide range of binaural applications and even for educational purposes.

Influence of first-order lateral reflections on the localization of virtual source reproduced by crosstalk cancellation system

This study elucidates the influence of first-order lateral reflections on the localization of virtual sources reproduced by a crosstalk cancellation system. First, an experimental study was conducted to examine how the reflections with a delay within the scope of the summing localization and the precedence effect influence the localization of virtual sources with different frequency bands and target azimuths. The results show that the localization of virtual sources is influenced due to the reflections with a delay below the low limit of the precedence effect (1 ms universally), and the influence degree is related to the frequency band and target azimuth. When the delay is further increased, the effect of reflected sound is almost negligible. Particularly, for a virtual source in the region outside the loudspeaker on the contralateral side of the reflecting surface (at -60° or -90°), the perceived azimuth under reflections with a small delay was more lateral than that without reflections. Then, simulations were conducted under the same conditions as the experiments. Based on the binaural signals modified by the inhibition function of the precedence effect, binaural cues, including interaural phase delay difference (ITDp) and interaural level difference, were calculated to explain the experimental results. Particularly, the average ITDp over frequency can qualitatively predict the localization results below 1.5 kHz from the subjective experiment in most cases.

Assessing the benefits of virtual speaker lateralization for binaural speech intelligibility over the Internet

Binaural speech intelligibility tests using headphones were conducted remotely through the Internet using virtual speaker lateralization at azimuth angles around the front within ±45 degrees, combined with babble noise with signal to noise ratios (SNR) from -12 to -4 dB. Monophonic speech recordings, selected from the publicly available Mozilla Common Voice speech corpus in Spanish, were binaurally processed using a database of generic Head-Related Transfer Functions (HRTF) to produce virtual speaker lateralization. A common signal of babble noise was mixed to both the left and right binaural signals, with different SNR. Speech intelligibility tests were conducted through a Google Forms questionnaire containing Internet links to YouTube videos embedding the corresponding test audio files. Symbolic text difference metrics: Levenshtein distance, and word error rate (WER), commonly used in the field of automatic speech recognition (ASR), were used to automatically calculate estimations of speech intelligibility scores, more conventionally used for (human) speech intelligibility research at the word level, together with microscopic speech intelligibility scores at the phonetic symbol level. Speech reception thresholds (SRT), and intelligibility slopes were determined, showing that the known beneficial effect of spatial release from masking (SRM) in improving speech intelligibility when the speaker is oriented at lateral azimuth angles relative to the listener, is preserved when using virtual speaker lateralization, and by the use of Internet transmitted audio using headphones. Results show that for azimuth angles of 20 and 30 degrees, left or right, speech intelligibility improves with an average unmasking benefit of 3.7±0.7 dB SRM, and an average intelligibility slope of 5.6±0.8%/dB, while also maintaining at these angles, a desirable impression of a virtual speaker with a reasonably frontal orientation. This provides a useful technique for improved Internet speech delivery.

Hybrid feedback active noise control headset based on binaural signal utilization

A standard feedback active noise control (FBANC) headset utilizes the estimate of a primary disturbance at the left-ear (right-ear) error microphone to control noise only at the left (right) ear-cup, i.e., each ear’s controller works independently. In contrast to the FBANC headset, in this paper, a binaural hybrid feedback active noise control (HFBANC) headset is designed that uses the estimate of the primary disturbances at both the left and right-ear error microphones to achieve improved noise control at both the left and right ear-cups. To further improve noise cancellation performance, the nearest Kronecker product decomposition technique is incorporated into the algorithm. The performance of the proposed HFBANC headset is compared to the standard FBANC headset under a variety of different sound field conditions. Experimental results show an improvement of 3–5 dB in the noise cancellation using the proposed algorithms, where the benefits are more prominent for noise sources originating from the side of the user (left and right).

Perceptual evaluation of approaches for binaural reproduction of non-spherical microphone array signals

Microphone arrays consisting of sensors mounted on the surface of a rigid, spherical scatterer are popular tools for the capture and binaural reproduction of spatial sound scenes. However, microphone arrays with a perfectly spherical body and uniformly distributed microphones are often impractical for the consumer sector, in which microphone arrays are generally mounted on mobile and wearable devices of arbitrary geometries. Therefore, the binaural reproduction of sound fields captured with arbitrarily shaped microphone arrays has become an important field of research. In this work, we present a comparison of methods for the binaural reproduction of sound fields captured with non-spherical microphone arrays. First, we evaluated equatorial microphone arrays (EMAs), where the microphones are distributed on an equatorial contour of a rigid, spherical 1. Second, we evaluated a microphone array with six microphones mounted on a pair of glasses. Using these two arrays, we conducted two listening experiments comparing four rendering methods based on acoustic scenes captured in different rooms2. The evaluation includes a microphone-based stereo approach (sAB stereo), a beamforming-based stereo approach (sXY stereo), beamforming-based binaural reproduction (BFBR), and BFBR with binaural signal matching (BSM). Additionally, the perceptual evaluation included binaural Ambisonics renderings, which were based on measurements with spherical microphone arrays. In the EMA experiment we included a fourth-order Ambisonics rendering, while in the glasses array experiment we included a second-order Ambisonics rendering. In both listening experiments in which participants compared all approaches with a dummy head recording we applied non-head-tracked binaural synthesis, with sound sources only in the horizontal plane. The perceived differences were rated separately for the attributes timbre and spaciousness. Results suggest that most approaches perform similarly to the Ambisonics rendering. Overall, BSM, and microphone-based stereo were rated the best for EMAs, and BFBR and microphone-based stereo for the glasses array.

An Algorithm for Generating Virtual Sources in Dynamic Virtual Auditory Display Based on Tensor Decomposition of Head-Related Impulse Responses

Dynamic virtual auditory displays (VADs) are increasingly used for generating various auditory objects and scenes in virtual and augmented reality. Dynamic VADs are required to generate virtual sources in various directions and distances by using HRTF- or HRIR-based binaural synthesis. In the present work, an algorithm for improving the efficiency and performance of binaural synthesis in dynamic VAD is proposed. Based on tensor decomposition, a full set of near-field HRIRs is decomposed as a combination of distance-, direction-, and time-related modes. Then, binaural synthesis in VAD can be implemented by a common set of time mode-related convolvers or filters associated with direction- and distance-related weights. Dynamic binaural signals are created by updating the weights rather than updating the HRIR-based convolvers, which enables the independent control of virtual source distance and direction and avoids the audible artifact caused by updating the HRIR-based convolvers. An example of implementation indicates that a set of eight common convolvers or filters for each ear is enough to synthesize the binaural signals with sufficient accuracy. The computational efficiency of simultaneously generating multiple virtual sources is improved when the number of virtual sources is larger than eight. A virtual-source localization experiment validates the algorithm.

Detection limit of sound images spatially split by synthesized binaural signals

Detection limit of sound images spatially split by synthesized binaural signals

Enhancing binaural rendering of head-worn microphone arrays through the use of adaptive spatial covariance matching.

In this article, the application of spatial covariance matching is investigated for the task of producing spatially enhanced binaural signals using head-worn microphone arrays. A two-step processing paradigm is followed, whereby an initial estimate of the binaural signals is first produced using one of three suggested binaural rendering approaches. The proposed spatial covariance matching enhancement is then applied to these estimated binaural signals with the intention of producing refined binaural signals that more closely exhibit the correct spatial cues as dictated by the employed sound-field model and associated spatial parameters. It is demonstrated, through objective and subjective evaluations, that the proposed enhancements in the majority of cases produce binaural signals that more closely resemble the spatial characteristics of simulated reference signals when the enhancement is applied to and compared against the three suggested starting binaural rendering approaches. Furthermore, it is shown that the enhancement produces spatially similar output binaural signals when using these three different approaches, thus indicating that the enhancement is general in nature and could, therefore, be employed to enhance the outputs of other similar binaural rendering algorithms.

Predicting reflection patterns from binaural activity maps using deep neural networks

A new model architecture is presented to predict room acoustical parameters from a running binaural signal. For this purpose, a deep neural network architecture is combined with a precedence effect model to extract the spatial and temporal locations of the direct signal and early reflections. The precedence effect model builds on the modified BICAM algorithm [Braasch, J. Acoust. Soc. Am. 140, EL143], for which the 1st layer auto-/cross correlation functions are replaced with a Cepstrum method. The latter allows a better separation of features relating to the source signal's early reflections and harmonic structure. The precedence effect model is used to create binaural activity maps that are analyzed by the neural network for pattern recognition. Anechoic orchestral recordings were reverberated by adding four early reflections and late reverberation to test the model. Head-related transfer functions were used to spatialize direct sound and early reflections. The model can identify the main reflection characteristics of a room, offering applications in numerous fields, including room acoustical assessment, acoustical analysis for virtual-reality applications, and modeling of human perception. [Work supported by the National Science Foundation under Grant No. IIS-1909229.]

Predicting the Colouration between Binaural Signals

Although the difference between the fast Fourier transforms of two audio signals is often used as a basic measure of predicting perceived colouration, these signal measures do not provide information on how relevant the results are from a perceptual point of view. This paper presents a perceptually motivated loudness calculation for predicting the colouration between binaural signals which incorporates equal loudness frequency contouring, relative subjective loudness weighting, cochlea frequency modelling, and an iterative normalisation of input signals. The validation compares the presented model to three other colouration calculations in two ways: using test signals designed to evaluate specific elements of the model, and against the results of a listening test on degraded binaural audio signals. Results demonstrate the presented model is appropriate for predicting the colouration between binaural signals.

Spatial Audio Scene Characterization (SASC): Automatic Localization of Front-, Back-, Up-, and Down-Positioned Music Ensembles in Binaural Recordings

The automatic localization of audio sources distributed symmetrically with respect to coronal or transverse planes using binaural signals still poses a challenging task, due to the front–back and up–down confusion effects. This paper demonstrates that the convolutional neural network (CNN) can be used to automatically localize music ensembles panned to the front, back, up, or down positions. The network was developed using the repository of the binaural excerpts obtained by the convolution of multi-track music recordings with the selected sets of head-related transfer functions (HRTFs). They were generated in such a way that a music ensemble (of circular shape in terms of its boundaries) was positioned in one of the following four locations with respect to the listener: front, back, up, and down. According to the obtained results, CNN identified the location of the ensembles with the average accuracy levels of 90.7% and 71.4% when tested under the HRTF-dependent and HRTF-independent conditions, respectively. For HRTF-dependent tests, the accuracy decreased monotonically with the increase in the ensemble size. A modified image occlusion sensitivity technique revealed selected frequency bands as being particularly important in terms of the localization process. These frequency bands are largely in accordance with the psychoacoustical literature.