Interaural Phase Research Articles

Clustering of spatial cues by semantic segmentation for anechoic binaural source separation

The recent introduction of neural networks to speech separation has dramatically boosted the separation performance. This paper presents a novel psychoacoustic approach for speech source separation in anechoic conditions, using semantic segmentation of the interaural spectrograms of the audio mixtures. We have trained two separate U-Nets (a specialized neural network for semantic segmentation) on the interaural level difference (ILD) spectrogram, and the interaural phase difference (IPD) spectrogram of a single source. After training, these U-Nets are used to predict the class of each time frequency (TF) unit of the interaural spectrogram of the audio mixture. The ILD and IPD soft masks obtained from these U-Nets are combined by a novel scheme which utilizes the strength of the interaural cues in different frequency bands. The results show improved separation over two state of the art machine learning source separation systems utilizing the same interaural cues. There is average improvement of 7.32 dB in signal to distortion ratio (SDR) and 0.3 points improvement in short term objective intelligibility (STOI) over degenerate un-mixing estimation technique (DUET) algorithm and 2.51 dB improvement in SDR with comparable intelligibility over model-based expectation–maximization source separation and localization (MESSL) algorithm.

Subclinical Auditory Neural Deficits in Patients With Type 1 Diabetes Mellitus

Diabetes mellitus (DM) is associated with a variety of sensory complications. Very little attention has been given to auditory neuropathic complications in DM. The aim of this study was to determine whether type 1 DM (T1DM) affects neural coding of the rapid temporal fluctuations of sounds, and how any deficits may impact on behavioral performance. Participants were 30 young normal-hearing T1DM patients, and 30 age-, sex-, and audiogram-matched healthy controls. Measurements included electrophysiological measures of auditory nerve and brainstem function using the click-evoked auditory brainstem response, and of brainstem neural temporal coding using the sustained frequency-following response (FFR); behavioral tests of temporal coding (interaural phase difference discrimination and the frequency difference limen); tests of speech perception in noise; and self-report measures of auditory disability using the Speech, Spatial and Qualities of Hearing Scale. There were no significant differences between T1DM patients and controls in the auditory brainstem response. However, the T1DM group showed significantly reduced FFRs to both temporal envelope and temporal fine structure. The T1DM group also showed significantly higher interaural phase difference and frequency difference limen thresholds, worse speech-in-noise performance, as well as lower overall Speech, Spatial and Qualities scores than the control group. These findings suggest that T1DM is associated with degraded neural temporal coding in the brainstem in the absence of an elevation in audiometric threshold, and that the FFR may provide an early indicator of neural damage in T1DM, before any abnormalities can be identified using standard clinical tests. However, the relation between the neural deficits and the behavioral deficits is uncertain.

Binaural Codebook-Based Speech Enhancement With Atomic Speech Presence Probability

In this work, we present a universal codebook-based speech enhancement framework that relies on a single codebook to encode both speech and noise components. The atomic speech presence probability (ASPP) is defined as the probability that a given codebook atom encodes speech at a given point in time. We develop ASPP estimators based on binaural cues including the interaural phase and level difference (IPD and ILD), the interaural coherence magnitude (ICM), as well as a combined version leveraging the full interaural transfer function (ITF). We evaluate the performance of the resulting ASPP-based speech enhancement algorithms on binaural mixtures of reverberant speech and real-world noise. The proposed approach improves both objective speech quality and intelligibility over a wide range of input SNR, as measured with PESQ and binaural STOI metrics, outperforming two binaural speech enhancement benchmark methods. We show that the proposed ITF-based ASPP approach achieves a good balance of the trade-off between binaural noise reduction and binaural cue preservation.

Modeling the Comb Filter Effect and Interaural Coherence for Binaural Source Separation

Typical methods for binaural source separation consider only the direct sound as the target signal in a mixture. However, in most scenarios, this assumption limits the source separation performance. It is well known that the early reflections interact with the direct sound, producing acoustic effects at the listening position, e.g. the so-called comb filter effect. In this article, we propose a novel source separation model, that utilizes both the direct sound and the first early reflection information to model the comb filter effect. This is done by observing the interaural phase difference obtained from the time-frequency representation of binaural mixtures. Furthermore, a method is proposed to model the interaural coherence of the signals. Including information related to the sound multipath propagation, the performance of the proposed separation method is improved with respect to the baselines that did not use such information, as illustrated by using binaural recordings made in four rooms, having different sizes and reverberation times.

Effects of interaural phase on the frequency following response

Effects of interaural phase on the frequency following response

Multilevel objective measures of interaural phase difference detection

Sensitivity to interaural phase difference (IPD) cues is critical for localizing sound and hearing in background noise. Recent work from multiple laboratories has explored electrophysiologic methods for assaying IPD sensitivity with the aim of developing an objective measure of binaural hearing. Such a measure would be useful in a variety of clinical and research applications such as understanding the effects of aging on binaural hearing; fine-tuning hearing aids/cochlear implants to maximize binaural benefit; and assessing neural damage after a traumatic brain injury. Here, we present data from an ongoing study evaluating a novel “multi-level” (brainstem, midbrain, and cortex) simultaneous measure of IPD acuity. Amplitude modulated (20, 40, and 80 AM) carrier tones (125, 250, and 500 Hz) with IPDs embedded in them (7 deg–180 deg) were used to generate auditory evoked potentials in young, normal hearing listeners. Neural responses corresponding to the carrier tones (frequency following responses), amplitude modulation envelopes (envelope following responses), and cortical change detection (acoustic change complex) were measured simultaneously to provide a multi-level snapshot of neural IPD processing. Parametric data presented here demonstrate feasibility and limitations of this approach for measuring IPD encoding at multiple levels of the auditory system.

Event-related potentials to single-cycle binaural beats of a pure tone, a click train, and a noise.

There are only few electrophysiological studies on a phenomenon called "binaural beats" (BBs), which is experienced when two tones with frequencies close to each other are dichotically presented to the ears. And, there is no study in which the electrical responses of the brain to BBs of complex sounds are recorded and analyzed. Owing to a recent method based on single-cycle BB stimulation with sub-threshold temporary monaural frequency shifts, we could record the event-related potentials (ERPs) to BBs of a 250-Hz tone as well as those to the BBs of a 250/s click train and to the BBs of a recurrent 4-ms Gaussian noise. Although fundamental components of the click train and noise stimuli were lower in intensity than the tonal stimuli in our experiments, the N1 responses to the BBs of the former two wide-spectrum sounds were recorded with significantly larger amplitudes and shorter latencies than those to the BBs of a tone, suggesting an across-frequency integration of directional information. During a BB cycle of a complex sound, the interaural time differences (ITDs) of the spectral components are all equal to each other at any time; whereas their interaural phase differences (IPDs) are all different. The ITD rather than the IPD should, therefore, be the cue that is relied upon by the binaural mechanism coding the perceived lateral shifts of the sound caused by BBs. This is in line with across-frequency models of human auditory lateralization based on a common ITD, fulfilling a straightness criterion.

Comodulation detection difference and binaural unmasking.

The present study investigated the combined effect of binaural cues and comodulation for a narrowband target noise masked by a narrowband noise. The threshold difference between a diotic condition (same stimuli in both ears) and a dichotic condition (target interaural phase difference of π and diotic masker) decreased with spectral distance between masker and target, irrespective of across-frequency envelope correlation. The threshold difference between a condition with comodulated target and masker and a corresponding uncorrelated condition, i.e., the comodulation detection difference, did not depend on target frequency and interaural correlation, indicating that these two stimulus properties are processed independently.

Individualized Interaural Feature Learning and Personalized Binaural Localization Model

The increasing importance of spatial audio technologies has demonstrated the need and importance of correctly adapting to the individual characteristics of the human auditory system, and illustrates the crucial need for humanoid localization systems for testing these technologies. To this end, this paper introduces a novel feature analysis and selection approach for binaural localization and builds a probabilistic localization mapping model, especially useful for the vertical dimension localization. The approach uses the mutual information as a metric to evaluate the most significant frequencies of the interaural phase difference and interaural level difference. Then, by using the random forest algorithm and embedding the mutual information as a feature selection criteria, the feature selection procedures are encoded with the training of the localization mapping. The trained mapping model is capable of using interaural features more efficiently, and, because of the multiple-tree-based model structure, the localization model shows robust performance to noise and interference. By integrating the direct path relative transfer function estimation, we propose to devise a novel localization approach that has improved performance in the presence of noise and reverberation. The proposed mapping model is compared with the state-of-the-art manifold learning procedure in different acoustical configurations, and a more accurate and robust output can be observed.

Event-related potentials to single-cycle binaural beats and diotic amplitude modulation of a tone.

When two tones with slightly different frequencies are dichotically presented, binaural beats (BBs) are experienced. BBs resulting from the cycling change in interaural phase difference elicit electroencephalographic responses. Because they repeat at short periods, allowing poor recovery of the cortical responses, these steady-state responses have small amplitudes, and their various wave components intermingle and might mask each other. Using single-cycle BBs separated by relatively long inter-onset intervals would be a solution, but introducing a transient interaural frequency shift requires response subtraction which may not be acceptable for non-additive brain responses. The proposed stimulation method employs transient and monaurally subthreshold frequency shifts in opposite directions in the two ears to produce single-cycle BBs of a 250Hz tone. These shifts are perceived as distinct BBs when presented dichotically, but remain subthreshold when presented monotically. Therefore, no frequency-shift response is elicited, and the specific BB response is obtained with no need for waveform subtraction. We recorded from 19 normal hearing participants the event-related potentials (ERPs) to single-cycle BBs and also to temporary diotic amplitude modulation (AM) with matched perceptual salience. The ERPs to single-cycle BBs presented at 2s inter-onset intervals had N1-P2 responses with up to seven times larger amplitudes than the conventional steady-state BB responses in the literature. Significant differences were found between the scalp potential distributions of the N1 responses to BB and AM stimuli, suggesting that the cortical sites, where envelope-based level processing and temporal fine structure-based spatial processing of the stimulus take place, are not totally overlapped.

Perceptually Relevant Preservation of Interaural Time Differences in Binaural Hearing Aids

This paper presents a noise reduction method with perceptually relevant preservation of the interaural time difference (ITD) of the residual noise in binaural hearing aids. The interaural coherence (IC) concept, previously applied to the multichannel Wiener filter (MWF) for preservation of the spatial subjective sensation of diffuse noise fields, is proposed here to both preserve and emphasize the ITD binaural cues of a directional acoustic noise source. It is demonstrated that the previously developed MWF-ITD technique may decrease the original IC magnitude of the processed noise, consequently increasing the variance of the interaural phase difference (IPD) of the output signals. It is shown that the MWF-IC technique simultaneously minimizes a nonlinear function of the difference between input and output IPD, which is strictly related to ITD, and preserves the natural coherence of directional noise captured by the reference microphones. Objective measures and psychoacoustic experiments corroborate the theoretical findings, showing that the MWF-IC technique provides relevant noise reduction, while preserving the original ITD subjective perception and original lateralization for a directional noise source. These results are especially relevant for hearing aid designers, since they indicate MWF-IC as a noise reduction technique that provides residual noise spatial preservation for both diffuse and directional noise sources in frequencies below 1.5 kHz.

Revisiting perceived intracranial lateralization for stimuli with interaural time differences that are larger than the head

The interaural time difference (ITD) is the primary sound-localization cue for humans. While realistic sound sources have energy across a wide frequency range, the ear performs a narrowband frequency decomposition, and within each band are ambiguities in the interaural cross-correlation (an index of the signal ITD). These ambiguities are thought to be resolved by the approximate consistency of ITDs across frequency, or “straightness.” However, straightness has not been evaluated over a wide range of stimuli. Therefore, normal-hearing listeners reported the intracranial lateralization of narrowband noises and tone complexes to better evaluate across-frequency ITD processing. In a new modification of the typical paradigm, listeners were encouraged to give multiple responses if split images were perceived. ITDs larger than those naturally produced by the head (~750 ms) best demonstrate straightness because across-frequency comparison is necessary to resolve the interaural phase ambiguities, which is why ITDs as large as 1500 ms were applied. Straightness effects reported previously for narrowband noises were replicated, and were broadly similar for complex tones. The extent of lateralization of 1500-ms ITDs and the occurrence of split images were difficult to account for using simple lateralization-based models for smaller ITDs.

Effects of Musical Training and Hearing Loss on Fundamental Frequency Discrimination and Temporal Fine Structure Processing: Psychophysics and Modeling

Several studies have shown that musical training leads to improved fundamental frequency (F0) discrimination for young listeners with normal hearing (NH). It is unclear whether a comparable effect of musical training occurs for listeners whose sensory encoding of F0 is degraded. To address this question, the effect of musical training was investigated for three groups of listeners (young NH, older NH, and older listeners with hearing impairment, HI). In a first experiment, F0 discrimination was investigated using complex tones that differed in harmonic content and phase configuration (sine, positive, or negative Schroeder). Musical training was associated with significantly better F0 discrimination of complex tones containing low-numbered harmonics for all groups of listeners. Part of this effect was caused by the fact that musicians were more robust than non-musicians to harmonic roving. Despite the benefit relative to their non-musicians counterparts, the older musicians, with or without HI, performed worse than the young musicians. In a second experiment, binaural sensitivity to temporal fine structure (TFS) cues was assessed for the same listeners by estimating the highest frequency at which an interaural phase difference was perceived. Performance was better for musicians for all groups of listeners and the use of TFS cues was degraded for the two older groups of listeners. These findings suggest that musical training is associated with an enhancement of both TFS cues encoding and F0 discrimination in young and older listeners with or without HI, although the musicians’ benefit decreased with increasing hearing loss. Additionally, models of the auditory periphery and midbrain were used to examine the effect of HI on F0 encoding. The model predictions reflected the worsening in F0 discrimination with increasing HI and accounted for up to 80 % of the variance in the data.

Sensitivity to Interaural Phase in Older Hearing-Impaired Listeners Correlates With Nonauditory Trail Making Scores and With a Spatial Auditory Task of Unrelated Peripheral Origin.

Interaural phase difference (IPD) discrimination upper frequency limits and just-noticeable differences (JNDs), interaural level difference (ILD) JNDs, and diotic intensity JNDs were measured for 20 older hearing-impaired listeners with matched moderate sloping to severe sensorineural hearing losses. The JNDs were measured using tone stimuli at 500 Hz. In addition to these auditory tests, the participants completed a cognitive test (Trail Making Test). Significant performance improvements in IPD discrimination were observed across test sessions. Strong correlations were found between IPD and ILD discrimination performance. Very strong correlations were observed between IPD discrimination and Trail Making performance as well as strong correlations between ILD discrimination and Trail Making performance. These relationships indicate that interindividual variability in IPD discrimination performance did not exclusively reflect deficits specific to any auditory processing, including early auditory processing of temporal information. The observed relationships between spatial audition and cognition may instead be attributable to a modality-general spatial processing deficit and/or individual differences in global processing speed.

Multitask Learning of Time-Frequency CNN for Sound Source Localization

Sound source localization (SSL) is an important technique for many audio processing systems, such as speech enhancement/recognition and human-robot interaction. Although many methods have been proposed for SSL, it still remains a challenging task to achieve accurate localization under adverse acoustic scenarios. In this paper, a novel binaural SSL method based on time-frequency convolutional neural network (TF-CNN) with multitask learning is proposed to simultaneously localize azimuth and elevation under unknown acoustic conditions. First, the interaural phase difference and interaural level difference are extracted from the received binaural signals, which are taken as the input of the proposed SSL neural network. Then, an SSL neural network is designed to map the interaural cues to sound direction, which consists of TF-CNN module and multitask neural network. The TF-CNN module learns and combines the time-frequency information of extracted interaural cues to generate the shared feature for multitask SSL. With the shared feature, a multitask neural network is designed to simultaneously estimate azimuth and elevation through multitask learning, which generates the posterior probability for candidate directions. Finally, the candidate direction with the highest probability is taken as the final direction estimation. The experiments based on public head-related transfer function (HRTF) database demonstrate that the proposed method achieves preferable localization performance compared with other popular methods.

Effects of Age and Noise Exposure on Proxy Measures of Cochlear Synaptopathy.

Although there is strong histological evidence for age-related synaptopathy in humans, evidence for the existence of noise-induced cochlear synaptopathy in humans is inconclusive. Here, we sought to evaluate the relative contributions of age and noise exposure to cochlear synaptopathy using a series of electrophysiological and behavioral measures. We extended an existing cohort by including 33 adults in the age range 37 to 60, resulting in a total of 156 participants, with the additional older participants resulting in a weakening of the correlation between lifetime noise exposure and age. We used six independent regression models (corrected for multiple comparisons), in which age, lifetime noise exposure, and high-frequency audiometric thresholds were used to predict measures of synaptopathy, with a focus on differential measures. The models for auditory brainstem responses, envelope-following responses, interaural phase discrimination, and the co-ordinate response measure of speech perception were not statistically significant. However, both age and noise exposure were significant predictors of performance on the digit triplet test of speech perception in noise, with greater noise exposure (unexpectedly) predicting better performance in the 80 dB sound pressure level (SPL) condition and greater age predicting better performance in the 40 dB SPL condition. Amplitude modulation detection thresholds were also significantly predicted by age, with older listeners performing better than younger listeners at 80 dB SPL. Overall, the results are inconsistent with the predicted effects of synaptopathy.

Measurement and Prediction of Binaural-Temporal Integration of Speech Reflections.

For speech intelligibility in rooms, the temporal integration of speech reflections is typically modeled by separating the room impulse response (RIR) into an early (assumed beneficial for speech intelligibility) and a late part (assumed detrimental). This concept was challenged in this study by employing binaural RIRs with systematically varied interaural phase differences (IPDs) and amplitude of the direct sound and a variable number of reflections delayed by up to 200 ms. Speech recognition thresholds in stationary noise were measured in normal-hearing listeners for 86 conditions. The data showed that direct sound and one or several early speech reflections could be perfectly integrated when they had the same IPD. Early reflections with the same IPD as the noise (but not as the direct sound) could not be perfectly integrated with the direct sound. All conditions in which the dominant speech information was within the early RIR components could be well predicted by a binaural speech intelligibility model using classic early/late separation. In contrast, when amplitude or IPD favored late RIR components, listeners appeared to be capable of focusing on these components rather than on the precedent direct sound. This could not be modeled by an early/late separation window but required a temporal integration window that can be flexibly shifted along the RIR.

Age-related differences in binaural masking level differences: behavioral and electrophysiological evidence.

The effects of aging and stimulus configuration on binaural masking level differences (BMLDs) were measured behaviorally and electrophysiologically, using the frequency-following response (FFR) to target brainstem/midbrain encoding. The tests were performed in 15 younger normal-hearing (<30 yr) and 15 older normal-hearing (>60 yr) participants. The stimuli consisted of a 500-Hz target tone embedded in a narrowband (50-Hz bandwidth) or wideband (1,500-Hz bandwidth) noise masker. The interaural phase conditions included NoSo (tone and noise presented interaurally in-phase), NoSπ (noise presented interaurally in-phase and tone presented out-of-phase), and NπSo (noise presented interaurally out-of-phase and tone presented in-phase) configurations. In the behavioral experiment, aging reduced the magnitude of the BMLD. The magnitude of the BMLD was smaller for the NoSo-NπSo threshold difference compared with the NoSo-NoSπ threshold difference, and it was also smaller in narrowband compared with wideband conditions, consistent with previous measurements. In the electrophysiology experiment, older participants had reduced FFR magnitudes and smaller differences between configurations. There were significant changes in FFR magnitude between the NoSo to NoSπ configurations but not between the NoSo to NπSo configurations. The age-related reduction in FFR magnitudes suggests a temporal processing deficit, but no correlation was found between FFR magnitudes and behavioral BMLDs. Therefore, independent mechanisms may be contributing to the behavioral and neural deficits. Specifically, older participants had higher behavioral thresholds than younger participants for the NoSπ and NπSo configurations but had equivalent thresholds for the NoSo configuration. However, FFR magnitudes were reduced in older participants across all configurations. NEW & NOTEWORTHY Behavioral and electrophysiological testing reveal an aging effect for stimuli presented in wideband and narrowband noise conditions, such that behavioral binaural masking level differences and subcortical spectral magnitudes are reduced in older compared with younger participants. These deficits in binaural processing may limit the older participant's ability to use spatial cues to understand speech in environments containing competing sound sources.

Behavioral and physiological measures of auditory processing in individuals with autism spectrum disorder

Sensory processing abnormalities are a hallmark of autism spectrum disorder (ASD). In the auditory domain, hyper- and hypo-sensitivity to sound, reduced orientation to speech, and difficulty listening in noise are commonly reported. However, the etiology of these auditory processing abnormalities is poorly understood. In this preliminary study, multi-talker speech perception thresholds, otoacoustic emissions, electrophysiological responses, as well as standardized measures of cognition, language, and adaptive function were measured in each individual subject with ASD and their age-matched controls. Speech perception was assessed by estimating target-to-masker ratios at 50% correct for speech targets (0° azimuth) presented with two spatially separated (±45° azimuth) simultaneous speech maskers. The electrophysiological battery used to characterize the transmission and representation of sound included: (1) a click-evoked supra-threshold auditory brainstem response, (2) an envelope following response recorded to a 400-ms-long 4 kHz pure tone carrier amplitude modulated at 100 Hz at two modulations depths (0 and -6 dB) and (3) a binaural potential evoked by an interaural phase difference embedded in an amplitude-modulated carrier tone. Together, these measures provide a behavioral and physiological assay of auditory function and show the potential to further our understanding of the mechanisms underlying auditory abnormalities in individuals with ASD.

Effect of stimulus type and pitch salience on pitch-sequence processing.

Using a same-different discrimination task, it has been shown that discrimination performance for sequences of complex tones varying just detectably in pitch is less dependent on sequence length (1, 2, or 4 elements) when the tones contain resolved harmonics than when they do not [Cousineau, Demany, and Pessnitzer (2009). J. Acoust. Soc. Am. 126, 3179-3187]. This effect had been attributed to the activation of automatic frequency-shift detectors (FSDs) by the shifts in resolved harmonics. The present study provides evidence against this hypothesis by showing that the sequence-processing advantage found for complex tones with resolved harmonics is not found for pure tones or other sounds supposed to activate FSDs (narrow bands of noise and wide-band noises eliciting pitch sensations due to interaural phase shifts). The present results also indicate that for pitch sequences, processing performance is largely unrelated to pitch salience per se: for a fixed level of discriminability between sequence elements, sequences of elements with salient pitches are not necessarily better processed than sequences of elements with less salient pitches. An ideal-observer model for the same-different binary-sequence discrimination task is also developed in the present study. The model allows the computation of d' for this task using numerical methods.