Binaural System Research Articles

Towards a comprehensive model of binaural processing

Most classic models of binaural processing are partly built on assumptions that have been challenged or even contradicted by physiologic or psychoacoustic data. For example, that delay lines encode interaural time difference (ITD), that “binaural filters” are wider than auditory filters, or that the binaural system is sluggish and cannot encode rapidly changing ITDs. While these assumptions have been motivated by psychoacoustic data, they have also been contradicted by other psychoacoustic data. The present study revisits and challenges the respective reasonings. One result is that the psychoacoustic proof of delay lines contained an unmet assumption. Instead of “sometimes wider filters” we propose a spectral incoherence interference mechanism that appears to resolve the impasse of inconsistent filter bandwidth requirements. In the temporal domain, we model fast ITD encoding and binaural sluggishness is realized by slow reformation of the interaural statistics of auditory objects. The resulting model still has some unproven assumptions; however, it does not need to change its parameters depending on the type of stimulus or task. As the binaural filter bandwidths and processing speeds are now equivalent to their monaural counterparts and consistent with physiological findings, the model may be instrumental in moving to a more unified understanding of auditory processing.

Influence of background noise on spatial audio localization for helicopter pilot headsets

A helicopter pilot is subjected to high workloads. A high noise level in the cockpit further increases stress. This stress can be reduced through headsets for hearing protection including active noise control (ANC). An actual research topic is the improvement of the human-machine-interface by 3-D-audio systems. These systems can improve pilot’s situational awareness and, therefore, reduce stress. This paper presents a 3-D-audio system for headphones. This binaural system uses head-tracking to process spatial information about the location of the sound source. The spatial sensation is achieved by convolving the source signal with head-related transfer functions. This sensation is influenced by background noise, which can be reduced by ANC. A first listening test with physical sound sources provides information on the localizability of sound sources at different elevation angles. This is compared to a second listening test with virtual sound sources generated by the 3-D-audio system. In a third listening test, the virtual sound source is located at variable azimuth and elevation angles. The system is evaluated with respect to sound source localization and the influence of background noise and ANC. This study highlights the potential of the 3-D-audio system to improve the helicopter pilot’s situational awareness in spatial acoustic environments.

Cortical temporal integration can account for limits of temporal perception: investigations in the binaural system

The auditory system has exquisite temporal coding in the periphery which is transformed into a rate-based code in central auditory structures, like auditory cortex. However, the cortex is still able to synchronize, albeit at lower modulation rates, to acoustic fluctuations. The perceptual significance of this cortical synchronization is unknown. We estimated physiological synchronization limits of cortex (in humans with electroencephalography) and brainstem neurons (in chinchillas) to dynamic binaural cues using a novel system-identification technique, along with parallel perceptual measurements. We find that cortex can synchronize to dynamic binaural cues up to approximately 10 Hz, which aligns well with our measured limits of perceiving dynamic spatial information and utilizing dynamic binaural cues for spatial unmasking, i.e. measures of binaural sluggishness. We also find that the tracking limit for frequency modulation (FM) is similar to the limit for spatial tracking, demonstrating that this sluggish tracking is a more general perceptual limit that can be accounted for by cortical temporal integration limits.

Remote Binaural System (RBS) for Noise Acoustic Monitoring

The recent emergence of advanced information technologies such as cloud computing, artificial intelligence, and data science has improved and optimized various processes in acoustics with potential real-world applications. Noise monitoring tasks on large terrains can be captured using an array of sound level meters. However, current monitoring systems only rely on the knowledge of a singular measured value related to the acoustic energy of the captured signal, leaving aside spatial aspects that complement the perception of noise by the human being. This project presents a system that performs binaural measurements according to subjective human perception. The acoustic characterization in an anechoic chamber is presented, as well as acoustic indicators obtained in the field initially for a short period of time. The main contribution of this work is the construction of a binaural prototype that resembles the human head and which transmits and processes acoustical data on the cloud. The above allows noise level monitoring via binaural hearing rather than a singular capturing device. Likewise, it can be highlighted that the system allows for obtaining spatial acoustic indicators based on the interaural cross-correlation function (IACF), as well as detecting the location of the source on the azimuthal plane.

A Clinically Viable Medial Olivocochlear Reflex Assay Using Transient-Evoked Otoacoustic Emissions.

The contralateral medial olivocochlear reflex (MOCR) strength may indicate various auditory conditions in humans, but a clinically viable assay and equipment are needed for quick, accurate, and reliable measurements. The first experiment compared an earlier version of the assay, which used a nonlinear-mode chirp stimulus, with a new assay using a linear-mode click stimulus, designed to give reliable MOCR measurements in most normal-hearing ears. The second experiment extended the improved assay on a purpose-built binaural hardware platform that used forward-pressure level (FPL) calibration for both the stimulus and the contralateral MOCR elicitor. Transient-evoked otoacoustic emission (TEOAE) tests were measured with and without a 60-dB SPL MOCR-evoking contralateral broadband noise. The normalized MOCR strength (MOCR%) was derived from the TEOAE responses for each trial pair using the complex pressure difference weighted by the TEOAE magnitude. Experiment 1 compared MOCR% within-subject and across-day using two TEOAE stimuli: nonlinear-mode chirps (50 dB SPL, bandpass 1-5 kHz, 14 ms window delayed by 2 ms) and linear-mode clicks (50 dB SPL, bandpass 0.5-2.5 kHz, 13 ms window delayed by 5 ms). TEOAE responses were analyzed in the 0.5 to 2.5 kHz band. Thirty adult participants with normal hearing (30 ears) completed the study. The TEOAE stimulus was calibrated in situ using spectral flattening, and the contralateral noise was calibrated in a coupler. Twelve TEOAE trial pairs were collected for each participant and condition. Experiment 2 used a purpose-built binaural system. The TEOAE stimuli were linear-mode clicks (50 dB SPL, bandpass 1-3 kHz, 13 ms window delayed by 5 ms), analyzed in the 1 to 3 kHz band over ~12 trial pairs. After a probe refit, an additional trial pair was collected for the two early-stopping signal-to-noise ratio criteria (15 and 20 dB). They were evaluated for single-trial reliability and test time. Nineteen adult participants with normal hearing (38 ears) completed the study. The TEOAE clicks and contralateral elicitor noise were calibrated in situ using FPL and delivered with automated timing. MOCR% for linear-mode clicks was distinguishable from measurement variability in 98% to 100% of participants' ears (both experiments), compared with only 73% for the nonlinear-mode chirp (experiment 1). MOCR detectability was assessed using the MOCR% across-subject/within-subject variance ratio. The ratio in experiment 1 for linear-mode clicks was higher (8.0) than for nonlinear-mode chirps (6.4). The ratio for linear-mode clicks (8.9) in experiment 2 was slightly higher than for the comparable linear-mode stimulus (8.0) in experiment 1. TEOAEs showed excellent reliability with high signal-to-noise ratios in both experiments, but reliability was higher for linear-mode clicks than nonlinear-mode chirps. MOCR reliability for the two stimuli was comparable. The FPL pressure response retest reliability derived from the SPL at the microphone was higher than the SPL retest reliability across 0.4 to 8 kHz. Stable results required 2 to 3 trial pairs for the linear-mode click (experiments 1 and 2) and three for the nonlinear-mode chirp (experiment 1), taking around 2 min on average. The linear-mode click assay produced measurable, reliable, and stable TEOAE and MOCR results on both hardware platforms in around 2 min per ear. The stimulus design and response window ensured that any stimulus artifact in linear mode was unlikely to confound the results. The refined assay is ready to produce high-quality data quickly for clinical and field studies to develop population norms, recognize diagnostic patterns, and determine risk profiles.

Playing With Others Using Headphones: Musicians Prefer Binaural Audio With Head Tracking Over Stereo

Immersive listening systems have grown significantly over the past decade and are now an established area of scientific, artistic, and industrial research. However, scarce research has been conducted on musicians' preferences for playing through headphones over binaural spatialization systems with the addition of head tracking, as opposed to classical stereophonic systems. This comparison is essential to optimally support the playing experience with others for cases of remote collaborative playing, individual instrumental practice, individual recreational music-making using backing tracks, and studio recording sessions. In this article, we study the preferences of playing musicians for a stereophonic system versus a binaural head-tracking system composed of Ambisonics technology and binaural synthesis with generalized head-related transfer functions. We conducted two experiments, each with 30 expert musicians, where participants were asked to rate and compare the 2 listening conditions while playing their instrument either seated or standing. Overall, the quantitative and qualitative results indicated a generalized preference for the binaural system with head tracking over the stereophonic system, with higher ratings for localization, immersion, social presence, realism, and connection with other musicians. Moreover, participants moved their heads significantly more in the binaural conditions. This phenomenon may be explained by the higher engagement and arousal due to the improved auditory experience, or alternatively by the presence of embodied music cognition mechanisms that cause a higher degree of exploration to better understand the action–perception loop. These findings highlight the need for progressing current commercial hardware and software systems used by musicians while playing over headphones.

Inferring the basis of binaural detection with a modified autoencoder.

The binaural system utilizes interaural timing cues to improve the detection of auditory signals presented in noise. In humans, the binaural mechanisms underlying this phenomenon cannot be directly measured and hence remain contentious. As an alternative, we trained modified autoencoder networks to mimic human-like behavior in a binaural detection task. The autoencoder architecture emphasizes interpretability and, hence, we "opened it up" to see if it could infer latent mechanisms underlying binaural detection. We found that the optimal networks automatically developed artificial neurons with sensitivity to timing cues and with dynamics consistent with a cross-correlation mechanism. These computations were similar to neural dynamics reported in animal models. That these computations emerged to account for human hearing attests to their generality as a solution for binaural signal detection. This study examines the utility of explanatory-driven neural network models and how they may be used to infer mechanisms of audition.

Spatial auditory presentation of a partner\u2019s presence induces the social Simon effect

Social presence is crucial for smooth communications in virtual reality (VR). Current telecommunication systems rarely submit spatial auditory information originating from remote people. However, such information may enhance social presence in VR. In this study, we constructed a dynamic binaural synthesis system and investigated the effect of spatial auditory information of a remote partner on a participant’s behavior using the social Simon effect (SSE). The SSE is a spatial stimulus–response compatibility effect between two persons. The SSE occurs when one perceives that their partner is present. Several studies have confirmed the SSE in actual environments. We presented partner sounds diotically (i.e., without spatial information) to one group or binaurally (i.e., with spatial information) to another group through headphones without providing visual information about the partner. The results showed that the SSE was induced only in the binaural group in the current auditory VR (Experiment 1), whereas both groups exhibited the SSE in an actual environment (Experiment 2). These results suggest that the auditory spatial information of remote people is sufficient to induce the SSE and has a potential to enhance social presence.

Aging effects on binaural listening and other auditory system assessments

Older listeners are known to have decreased sound localization and hearing in noise abilities compared to young listeners, even when exhibiting clinically normal hearing. We hypothesize that these deficits may be related to decreased temporal precision of neural activity in the sound localization processing pathway, and/or other central auditory system deficits involving more complex functions such as speech recognition. Presented here are preliminary results from a study underway on the impact of aging on auditory functioning, and in particular, binaural processing. Subjects range from 21 to 89 years of age, have normal or near normal-hearing sensitivity, and are being evaluated on assessments of auditory system integrity and behavioral performance. Primary outcome measures include speech understanding in noise, auditory brainstem responses (ABRs) and the calculated ABR binaural interaction component, behavioral measures of spatial acuity, and a subjective questionnaire of hearing handicap. Secondary measures include temporal fine structure and spectro-temporal modulation sensitivity tests, otoacoustic emissions, and electrocochleography. In addition, extended-high-frequency hearing thresholds are measured and working memory assessed. Early results show an aging effect across decades of life on several of these measures. Greatest aging effects are seen in the more adverse listening conditions from behavioral testing. [Support: R01-DC017924.]

Automatic discrimination between front and back ensemble locations in HRTF-convolved binaural recordings of music

One of the greatest challenges in the development of binaural machine audition systems is the disambiguation between front and back audio sources, particularly in complex spatial audio scenes. The goal of this work was to develop a method for discriminating between front and back located ensembles in binaural recordings of music. To this end, 22, 496 binaural excerpts, representing either front or back located ensembles, were synthesized by convolving multi-track music recordings with 74 sets of head-related transfer functions (HRTF). The discrimination method was developed based on the traditional approach, involving hand-engineering of features, as well as using a deep learning technique incorporating the convolutional neural network (CNN). According to the results obtained under HRTF-dependent test conditions, CNN showed a very high discrimination accuracy (99.4%), slightly outperforming the traditional method. However, under the HRTF-independent test scenario, CNN performed worse than the traditional algorithm, highlighting the importance of testing the algorithms under HRTF-independent conditions and indicating that the traditional method might be more generalizable than CNN. A minimum of 20 HRTFs are required to achieve a satisfactory generalization performance for the traditional algorithm and 30 HRTFs for CNN. The minimum duration of audio excerpts required by both the traditional and CNN-based methods was assessed as 3 s. Feature importance analysis, based on a gradient attribution mapping technique, revealed that for both the traditional and the deep learning methods, a frequency band between 5 and 6 kHz is particularly important in terms of the discrimination between front and back ensemble locations. Linear-frequency cepstral coefficients, interaural level differences, and audio bandwidth were identified as the key descriptors facilitating the discrimination process using the traditional approach.

Prediction and Improvement of Aircraft Cabin Acoustics using Statistical Energy Analysis and Sound Quality Evaluation

Aircraft interior acoustic design is a key influencer for cabin comfort. An essential part of the design is optimization of acoustic insulation systems under weight restrictions to create a pleasant environment for human ear. Considering the complexity of aircraft geometry, noise sources, and transfer paths, computational prediction techniques become invaluable tools for increasing the accuracy in material selection while reducing design time and costs. In this study, a procedure that integrates sound quality evaluation with Statistical Energy Analysis (SEA) to design aircraft acoustic insulation systems is described. SEA is employed to predict the cabin sound pressure levels of a narrow body aircraft insulated with sound absorption and vibration damping materials. Aircraft cabin including under-floor sections is modelled based on 3D airframe and VIP style interior design and the model is validated with flight test data. Transfer functions obtained from SEA model for selected transfer paths are utilized to filter the noise signal recorded with a binaural recording system during flight. Sound quality metrics are computed in order to map perceptive response. An iterative process is introduced to improve acoustic design by investigating the effects of different sound insulation systems and room absorption values on noise levels and sound quality metrics.

The Application of Binaural System on Virtual Reality Technology Through the Animation of Wayang Kulit Yogyakarta Performance

Binaural system is an audio technology which relatively easy to apply these days. This was due to the growth in technology developments of the audio recording system. The application of this system is not well known in Indonesia. This was aligned with public assumption where Wayang Kulit performance are one of many unappreciated forms of arts in Indonesia. The main problem that occurred is that whether the binaural system conjoined with the Virtual Reality (VR) System can make the public more appreciate the art form of Indonesian Wayang Kulit. By using the theory of perception conjoin with the binaural technology, it can be assumed that the audience can have a new experience from the performance through the VR thus elevating their appreciation for the art itself. This research will be divided into three stages. The first stage focused on the application of the binaural system on Wayang Kulit Animation through VR system. The second stage will focus on the product assessment whether it can elevate the appreciation of the audience for the art itself. The third stage will focus on applying the binaural system on other Indonesian art form. This year, the research will begin at the first stage.

Binaural Audio-Visual Localization

Localizing sound sources in a visual scene has many important applications and quite a few traditional or learning-based methods have been proposed for this task. Humans have the ability to roughly localize sound sources within or beyond the range of the vision using their binaural system. However most existing methods use monaural audio, instead of binaural audio, as a modality to help the localization. In addition, prior works usually localize sound sources in the form of object-level bounding boxes in images or videos and evaluate the localization accuracy by examining the overlap between the ground-truth and predicted bounding boxes. This is too rough since a real sound source is often only a part of an object. In this paper, we propose a deep learning method for pixel-level sound source localization by leveraging both binaural recordings and the corresponding videos. Specifically, we design a novel Binaural Audio-Visual Network (BAVNet), which concurrently extracts and integrates features from binaural recordings and videos. We also propose a point-annotation strategy to construct pixel-level ground truth for network training and performance evaluation. Experimental results on Fair-Play and YT-Music datasets demonstrate the effectiveness of the proposed method and show that binaural audio can greatly improve the performance of localizing the sound sources, especially when the quality of the visual information is limited.

Impaired Binaural Hearing in Adults: A Selected Review of the Literature.

Despite over 100 years of study, there are still many fundamental questions about binaural hearing that remain unanswered, including how impairments of binaural function are related to the mechanisms of binaural hearing. This review focuses on a number of studies that are fundamental to understanding what is known about the effects of peripheral hearing loss, aging, traumatic brain injury, strokes, brain tumors, and multiple sclerosis (MS) on binaural function. The literature reviewed makes clear that while each of these conditions has the potential to impair the binaural system, the specific abilities of a given patient cannot be known without performing multiple behavioral and/or neurophysiological measurements of binaural sensitivity. Future work in this area has the potential to bring awareness of binaural dysfunction to patients and clinicians as well as a deeper understanding of the mechanisms of binaural hearing, but it will require the integration of clinical research with animal and computational modeling approaches.

Binaural speaker identification using the equalization-cancelation technique

In real applications, environmental effects such as additive noise and room reverberation lead to a mismatch between training and testing signals that substantially reduces the performance of far-field speaker identification. As a solution to this mismatch problem, in this paper, a new binaural speaker identification system is proposed which employs the well-known equalization-cancelation technique in its structure. The equalization-cancelation algorithm is employed to enhance the input test speech and alleviate the detrimental effects of noise and reverberation in the speaker identification system. The performance of the proposed speaker identification system is compared with unprocessed identification systems and a traditional binaural speaker identification system from the literature. The proposed system is evaluated in both anechoic and reverberant conditions using different types of noise at various azimuthal positions. Simulation results show the superiority of the proposed method in all experimental conditions.

Temporal quantization deteriorates the discrimination of interaural time differences.

Cochlear implants (CIs) often replace acoustic temporal fine structure by a fixed-rate pulse train. If the pulse timing is arbitrary (that is, not based on the phase information of the acoustic signal), temporal information is quantized by the pulse period. This temporal quantization is probably imperceptible with current clinical devices. However, it could result in large temporal jitter for strategies that aim to improve bilateral and bimodal CI users' perception of interaural time differences (ITDs), such as envelope enhancement. In an experiment with 16 normal-hearing listeners, it is shown that such jitter could deteriorate ITD perception for temporal quantization that corresponds to the often-used stimulation rate of 900 pulses per second (pps): the just-noticeable difference in ITD with quantization was 177 μs as compared to 129 μs without quantization. For smaller quantization step sizes, no significant deterioration of ITD perception was found. In conclusion, the binaural system can only average out the effect of temporal quantization to some extent, such that pulse timing should be well-considered. As this psychophysical procedure was somewhat unconventional, different procedural parameters were compared by simulating a number of commonly used two-down one-up adaptive procedures in Appendix B.

Binaural Sonar System for Simultaneous Sensing of Distance and Direction of Extended Barriers

Bat navigation via echolocation has inspired numerous acoustic-based obstacle sensing techniques. In this paper, we use a low-cost, small size, single frequency active binaural system for simultaneous detection of distance and direction of extended obstacles using the time of flight cue. For a binaural system and given size of the transducer, the minimum tolerable separation distance between the emitter and receiver as well as best frequency range for effectively sensing a barrier of any orientation will be assessed by simulation. With the given size of transducers of 6.30 mm in radius and for a working distance of 1 m, the minimum separation distance between the emitter and receiver should not be less than 5 cm and the emitter frequency should not exceed higher than 20 kHz. It will be shown in this article that higher frequency limits the lateral distance detection ability of the system. We experimentally verified the technique for detecting a wall of any orientation with respect to the system’s axis. The approach developed in this paper could be useful for mobile robotics, indoor navigation, and personalized navigation, where a compact configuration is of interest.

Theatrical virtual acoustic rendering with head movement interaction

AbstractNowadays, the use of virtual reality/virtual acoustics (VR/VA) in archaeology for rendering lost buildings is an important topic in the cultural heritage field. Moreover, the addition of additional senses apart from the sight increases the feeling of immersion in virtual environments. The aim of this paper is to show the interaction work developed in a VA system, based on Unity and FMOD, the graphical and acoustical reconstruction of an ancient building and the development of a VR goggles with headphones to render 3D audio and video interactively. This system has been implemented to render auralizations in a binaural system and has been applied to the renderization of an old and lost theatre in València (Spain). The first building of theatre was built in the 16th century, and was rebuilt several times until the 18th century. The auralization of several theatrical excerpts of different Spanish authors of that time is also presented. The integrated system has been subjectively evaluated, obtaining very satisfactory results.

A Temporal Filter for Binaural Hearing Is Dynamically Adjusted by Sound Pressure Level.

In natural environments our auditory system is exposed to multiple and diverse signals of fluctuating amplitudes. Therefore, to detect, localize, and single out individual sounds the auditory system has to process and filter spectral and temporal information from both ears. It is known that the overall sound pressure level affects sensory signal transduction and therefore the temporal response pattern of auditory neurons. We hypothesize that the mammalian binaural system utilizes a dynamic mechanism to adjust the temporal filters in neuronal circuits to different overall sound pressure levels. Previous studies proposed an inhibitory mechanism generated by the reciprocally coupled dorsal nuclei of the lateral lemniscus (DNLL) as a temporal neuronal-network filter that suppresses rapid binaural fluctuations. Here we investigated the consequence of different sound levels on this filter during binaural processing. Our in vivo and in vitro electrophysiology in Mongolian gerbils shows that the integration of ascending excitation and contralateral inhibition defines the temporal properties of this inhibitory filter. The time course of this filter depends on the synaptic drive, which is modulated by the overall sound pressure level and N-methyl-D-aspartate receptor (NMDAR) signaling. In psychophysical experiments we tested the temporal perception of humans and show that detection and localization of two subsequent tones changes with the sound pressure level consistent with our physiological results. Together our data support the hypothesis that mammals dynamically adjust their time window for sound detection and localization within the binaural system in a sound level dependent manner.

Robust emotional speech recognition based on binaural model and emotional auditory mask in noisy environments

The performance of automatic speech recognition systems degrades in the presence of emotional states and in adverse environments (e.g., noisy conditions). This greatly limits the deployment of speech recognition application in realistic environments. Previous studies in the emotion-affected speech recognition field focus on improving emotional speech recognition using clean speech data recorded in a quiet environment (i.e., controlled studio settings). The goal of this research is to increase the robustness of speech recognition systems for emotional speech in noisy conditions. The proposed binaural emotional speech recognition system is based on the analysis of binaural input signal and an estimated emotional auditory mask corresponding to the recognized emotion. Whereas the binaural signal analyzer has the task of segregating speech from noise and constructing speech mask in a noisy environment, the estimated emotional mask identifies and removes the most emotionally affected spectro-temporal regions of the segregated target speech. In other words, our proposed system combines the two estimated masks (binary mask and emotion-specific mask) of noise and emotion, as a way to decrease the word error rate for noisy emotional speech. The performance of the proposed binaural system is evaluated in clean neutral train/noisy emotional test scenarios for different noise types, signal-to-noise ratios, and spatial configurations of sources. Speech utterances of the Persian emotional speech database are used for the experimental purposes. Simulation results show that the proposed system achieves higher performance, as compared with automatic speech recognition systems chosen as baseline trained with neutral utterances.