Binaural Sound Source Localization Research Articles

Dissociated Representation of Binaural Cues in Single-Sided Deafness: Implications for Cochlear Implantation.

Congenital single-sided deafness (SSD) leads to an aural preference syndrome that is characterized by overrepresentation of the hearing ear in the auditory system. Cochlear implantation (CI) of the deaf ear is an effective treatment for SSD. However, the newly introduced auditory input in congenital SSD often does not reach expectations in late-implanted CI recipients with respect to binaural hearing and speech perception. In a previous study, a reduction of the interaural time difference (ITD) sensitivity has been shown in unilaterally congenitally deaf cats (uCDCs). In the present study, we focused on the interaural level difference (ILD) processing in the primary auditory cortex. The uCDC group was compared with hearing cats (HCs) and bilaterally congenitally deaf cats (CDCs). The ILD representation was reorganized, replacing the preference for the contralateral ear with a preference for the hearing ear, regardless of the cortical hemisphere. In accordance with the previous study, uCDCs were less sensitive to interaural time differences than HCs, resulting in unmodulated ITD responses, thus lacking directional information. Such incongruent ITDs and ILDs cannot be integrated for binaural sound source localization. In normal hearing, the predominant effect of each ear is excitation of the auditory cortex in the contralateral cortical hemisphere and inhibition in the ipsilateral hemisphere. In SSD, however, auditory pathways reorganized such that the hearing ear produced greater excitation in both cortical hemispheres and the deaf ear produced weaker excitation and preserved inhibition in both cortical hemispheres.

Estimating speaker direction on a humanoid robot with binaural acoustic signals.

To achieve human-like behaviour during speech interactions, it is necessary for a humanoid robot to estimate the location of a human talker. Here, we present a method to optimize the parameters used for the direction of arrival (DOA) estimation, while also considering real-time applications for human-robot interaction scenarios. This method is applied to binaural sound source localization framework on a humanoid robotic head. Real data is collected and annotated for this work. Optimizations are performed via a brute force method and a Bayesian model based method, results are validated and discussed, and effects on latency for real-time use are also explored.

Toward learning robust contrastive embeddings for binaural sound source localization.

Recent deep neural network based methods provide accurate binaural source localization performance. These data-driven models map measured binaural cues directly to source locations hence their performance highly depend on the training data distribution. In this paper, we propose a parametric embedding that maps the binaural cues to a low-dimensional space where localization can be done with a nearest-neighbor regression. We implement the embedding using a neural network, optimized to map points that are close to each other in the latent space (the space of source azimuths or elevations) to nearby points in the embedding space, thus the Euclidean distances between the embeddings reflect their source proximities, and the structure of the embeddings forms a manifold, which provides interpretability to the embeddings. We show that the proposed embedding generalizes well in various acoustic conditions (with reverberation) different from those encountered during training, and provides better performance than unsupervised embeddings previously used for binaural localization. In addition, the proposed method performs better than or equally well as a feed-forward neural network based model that directly estimates the source locations from the binaural cues, and it has better results than the feed-forward model when a small amount of training data is used. Moreover, we also compare the proposed embedding using both supervised and weakly supervised learning, and show that in both conditions, the resulting embeddings perform similarly well, but the weakly supervised embedding allows to estimate source azimuth and elevation simultaneously.

Speech Enhancement Based on Binaural Sound Source Localization and Cosh Measure Wiener Filtering

Speech Enhancement Based on Binaural Sound Source Localization and Cosh Measure Wiener Filtering

Binaural sound source localization based on weighted template matching

In robot binaural sound source localization (SSL), locating the direction of the sound source accurately in the shortest time is important. It refers to the algorithm complexity, but even more to the shortest duration of the required signal. A novel binaural SSL method based on feature and frequency weighting is proposed. More specifically, in the training stage, the direction-related interaural cross-correlation function(CCF) and interaural intensity difference(IID) in each frequency band are calculated under noiseless conditions, which are considered the templates. In the testing stage, first the cosine similarities between the CCF and IID of the test signal and templates are calculated in all features and frequency bands. Then, the direction likelihood can be obtained by weighting the similarities. Finally, the direction with maximum likelihood is specified as the direction of the sound source. Experiments were carried out on CIPIC dataset subject 003 with different noises in the noisex-92 dataset and demonstrated that the method can accurately locate the sound source with a short signal duration.

Head‐related transfer function–reserved time‐frequency masking for robust binaural sound source localization

Various time-frequency (T-F) masks are being applied to sound source localization tasks. Moreover, deep learning has dramatically advanced T-F mask estimation. However, existing masks are usually designed for speech separation tasks and are suitable only for single-channel signals. A novel complex-valued T-F mask is proposed that reserves the head-related transfer function (HRTF), customized for binaural sound source localization. In addition, because the convolutional neural network that is exploited to estimate the proposed mask takes binaural spectral information as the input and output, accurate binaural cues can be preserved. Compared with conventional T-F masks that emphasize single speech source–dominated T-F units, HRTF-reserved masks eliminate the speech component while keeping the direct propagation path. Thus, the estimated HRTF is capable of extracting more reliable localization features for the final direction of arrival estimation. Hence, binaural sound source localization guided by the proposed T-F mask is robust under noisy and reverberant acoustic environments. The experimental results demonstrate that the new T-F mask is superior to conventional T-F masks and lead to the better performance of sound source localization in adverse environments.

Deep Learning for Binaural Sound Source Localization with Low Signal-to-noise Ratio

A novel deep learning (DL) method is proposed for binaural sound source localization with low SNR. Firstly, the binaural sound signals are decomposed into several channels by using Gammatone filter. Secondly, the 4 feature parameters of Head-related Transfer Function, interaural time difference (ITD), interaural coherence (IC), interaural level difference (ILD), and interaural phase difference (IPD) are extracted. Thirdly, ITD and IC go through a Deep Belief Network (DBN) to determine the quadrant of the sound source and reduce the positioning range. Then, ITD, IC, ILD, and IPD go through a Deep Neural Network (DNN) to obtain the azimuthal angle within 90 degrees. Experimental results show that the proposed algorithm can solve the front-back confusion, and obtain a superior performance with lower complexity and higher precision under low SNR conditions.

Deep and CNN fusion method for binaural sound source localisation

In binaural sound source localisation, front–back confusion is often the challenging problem when localising sources in the noisy or reverberant environments. Hence, a novel algorithm fusing deep and convolutional neural network (CNN) is proposed to address this issue. First, joint features, which consist of interaural level differences (ILDs) and cross-correlation function (CCF) within a lag range, are extracted from binaural signals. Second, with the extracted CCF–ILD features, CNN is used for the front–back classification task, while deep neural network is used for azimuth classification task. The front–back features extracted by the CNN can be leveraged as additional information for the sound source localisation task. Also, an angle-loss function is designed to avoid the overfitting problem and to improve the generalisation ability of this method in adverse acoustic conditions. Finally, two branches are concatenated and then followed by an output layer, which generates the posterior probability of azimuth angles, and the azimuth corresponding to the maximum posterior probability is chosen as the direction of sound source. Experimental results demonstrate the effectiveness of the authors’ method for front–back decision and azimuth estimation in noisy and reverberant environments.

Binaural sound localization based on deep neural network and affinity propagation clustering in mismatched HRTF condition

Binaural sound source localization is an important and widely used perceptually based method and it has been applied to machine learning studies by many researchers based on head-related transfer function (HRTF). Because the HRTF is closely related to human physiological structure, the HRTFs vary between individuals. Related machine learning studies to date tend to focus on binaural localization in reverberant or noisy environments, or in conditions with multiple simultaneously active sound sources. In contrast, mismatched HRTF condition, in which the HRTFs used to generate the training and test sets are different, is rarely studied. This mismatch leads to a degradation of localization performance. A basic solution to this problem is to introduce more data to improve generalization performance, which requires a lot. However, simply increasing the data volume will result in data-inefficiency. In this paper, we propose a data-efficient method based on deep neural network (DNN) and clustering to improve binaural localization performance in the mismatched HRTF condition. Firstly, we analyze the relationship between binaural cues and the sound source localization with a classification DNN. Different HRTFs are used to generate training and test sets, respectively. On this basis, we study the localization performance of DNN model trained by each training set on different test sets. The result shows that the localization performance of the same model on different test sets is different, while the localization performance of different models on the same test set may be similar. The result also shows a clustering trend. Secondly, different HRTFs are divided into several clusters. Finally, the corresponding HRTFs of each cluster center are selected to generate a new training set and to train a more generalized DNN model. The experimental results show that the proposed method achieves better generalization performance than the baseline methods in the mismatched HRTF condition and has almost equal performance to the DNN trained with a large number of HRTFs, which means the proposed method is data-efficient.

Wideband sparse Bayesian learning for off-grid binaural sound source localization

Some traditional binaural sound source localization (BSSL) techniques estimate the azimuth using measured head-related transfer function (HRTF) databases, which is constrained by the discretely measured azimuths of HRTF databases. The azimuth localization performance of these HRTF-based BSSL methods may degrade significantly when the true azimuth is not included in the discretely measured azimuths, which is a typical off-grid problem. This paper proposes an off-grid BSSL method based on an off-grid wideband sparse Bayesian learning algorithm. An off-grid binaural sparse signal model is established first, which takes into account both the shadowing effects by the head and the impacts of off-grid problem. Based on the spatial sparsity of sound sources, the off-grid BSSL problem can be reduced to a convex optimization problem. An off-grid wideband sparse Bayesian learning algorithm is further derived to solve the convex optimization problem and thus improve the localization performance. Experimental results demonstrate that the proposed off-grid BSSL method can achieve higher localization accuracy than the state-of-the-art HRTF-based BSSL methods in various acoustic environments, especially in the off-grid situations.

Binaural Sound Source Localization Based on Convolutional Neural Network

Binaural Sound Source Localization Based on Convolutional Neural Network

Enhanced Robot Speech Recognition Using Biomimetic Binaural Sound Source Localization.

Inspired by the behavior of humans talking in noisy environments, we propose an embodied embedded cognition approach to improve automatic speech recognition (ASR) systems for robots in challenging environments, such as with ego noise, using binaural sound source localization (SSL). The approach is verified by measuring the impact of SSL with a humanoid robot head on the performance of an ASR system. More specifically, a robot orients itself toward the angle where the signal-to-noise ratio (SNR) of speech is maximized for one microphone before doing an ASR task. First, a spiking neural network inspired by the midbrain auditory system based on our previous work is applied to calculate the sound signal angle. Then, a feedforward neural network is used to handle high levels of ego noise and reverberation in the signal. Finally, the sound signal is fed into an ASR system. For ASR, we use a system developed by our group and compare its performance with and without the support from SSL. We test our SSL and ASR systems on two humanoid platforms with different structural and material properties. With our approach we halve the sentence error rate with respect to the common downmixing of both channels. Surprisingly, the ASR performance is more than two times better when the angle between the humanoid head and the sound source allows sound waves to be reflected most intensely from the pinna to the ear microphone, rather than when sound waves arrive perpendicularly to the membrane.

Binaural Localization of Multiple Sound Sources by Non-Negative Tensor Factorization

This paper presents non-negative factorization of audio signals for the binaural localization of multiple sound sources within realistic and unknown sound environments. Non-negative tensor factorization (NTF) provides a sparse representation of multichannel audio signals in time, frequency, and space that can be exploited in computational audio scene analysis and robot audition for the separation and localization of sound sources. In the proposed formulation, each sound source is represented by means of spectral dictionaries, temporal activation, and its distribution within each channel (here, left and right ears). This distribution, being dependent on the frequency, can be interpreted as an explicit estimation of the Head-Related Transfer Function (HRTF) of a binaural head which can then be converted into the estimated sound source position. Moreover, the semisupervised formulation of the non-negative factorization allows us to integrate prior knowledge about some sound sources of interest whose dictionaries can be learned in advance, whereas the remaining sources are considered as background sound, which remains unknown and is estimated on the fly. The proposed NTF-based sound source localization is applied here to binaural sound source localization of multiple speakers within realistic sound environments.

Binaural Sound Localization Based on Reverberation Weighting and Generalized Parametric Mapping

Binaural sound source localization is an important technique for speech enhancement, video conferencing, and human-robot interaction, etc. However, in realistic scenarios, the reverberation and environmental noise would degrade the precision of sound direction estimation. Therefore, reliable sound localization is essential to practical applications. To deal with these disturbances, this paper presents a novel binaural sound source localization approach based on reverberation weighting and generalized parametric mapping. First, the reverberation weighting as a preprocessing stage, is used to separately suppress the early and late reverberation, while preserving interaural cues. Then, two binaural cues, i.e., interaural time and intensity differences, are extracted from the frequency-domain representations of dereverberated binaural signals for the online localization. Their corresponding templates are established using the training data. Furthermore, the generalized parametric mapping is proposed to build a generalized parametric model for describing relationships between azimuth and binaural cues analytically. Finally, a two-step sound localization process is introduced to refine azimuth estimation based on the generalized parametric model and template matching. Experiments in both simulated and real scenarios validate that the proposed method can achieve better localization performance compared to state-of-the-art methods.

Position estimation of binaural sound source in reverberant environments

Most binaural sound source systems perform localization in either direction or distance perception. However, in real scenarios both perceptions are important to estimate source position in various environment conditions especially with the rapid technological growth in smart machines and their involvement in human daily life. This paper introduces an approach for azimuth and distance of binaural sound source localization in different reverberating environments using only two microphones. The algorithm is based on statistical features of the binaural cues and the difference of the binaural magnitude spectra of the binaural signal. Gaussian Mixture Models (GMMs) are used to jointly learn both distances and azimuths in different reverberant rooms. The proposed system does not require any prior knowledge of head related transfer function (HRTF), acoustical environment or room parameters. The performance has been evaluated at different aspects and conditions and reported effective and robust results, especially in the case of training set mismatch.

SVM-based Binaural Sound Source Localization

SVM-based Binaural Sound Source Localization

Binaural Sound Source Localization based on Sub-band SNR Estimation

Sound Source Localization (SSL) has a wide application in speech separation, recognition and enhancement. Binaural sound source localization based on human spatial hearing mechanism is an important research field of SSL. The recent binaural SSL research is focused on the system robust against noise and reverberation. In order to improve the localization performance in degraded environment, this paper proposes an algorithm to adaptively select the ‘good’ sub-bands to compute the binaural localization cues. Firstly, sub-band Signal-Noise Ratio (SNR) is estimated based on the autocorrelation matrix of binaural sound signals. Then, Inter-aural Time Difference (ITD) is computed by adaptively selecting the sub-bands which have the high SNR. Since the ITD is calculated through the sub-bands which are less affected by the noise, the sound source azimuth is estimated more accurate. The simulation results show that compared to the conventional binaural SSL algorithm, the localization accuracy of the proposed algorithm has been improved significantly.

Dynamic binaural sound source localization with interaural time difference cues: Artificial listeners

When an array of two acoustic sensors is used to localize sound sources based on time differences alone, possible solutions form a cone of confusion. This study, together with a similar one for human listeners, demonstrates that azimuth/vertical planes localization of a sound source using only time difference information is feasible when self-motion measurement of the listener is available. In particular, the case of a static sound source playing that broadcasts low frequency pure tone signals was investigated. A dummy head is mounted on top of a rotating chair to mimic the head and body motion of human beings, as well as to collect audio signals. A gyroscope was mounted on top of the dummy head to collect self-motion data. A mathematical model was constructed to describe the interaural time difference (ITD) change over time, and an Extended Kalman Filter (EKF) was used to estimate the spatial angles of the sound sources with respect to the listener using the developed mathematical model and measured data. The effectiveness and robustness of the developed algorithm are shown by both the numerical and experimental results, which reveal the quick convergence of the estimated spatial angles toward their real values given noisy measured data. The possibilities of using other spatial hearing cues were also discussed.

Binaural sound source localization using the frequency diversity of the head-related transfer function

The spectral localization cues contained in the head-related transfer function are known to play a contributory role in the sound source localization abilities of humans. However, existing localization techniques are unable to fully exploit this diversity to accurately localize a sound source. The availability of just two measured signals complicates matters further, and results in front to back confusions and poor performance distinguishing between the source locations in a vertical plane. This study evaluates the performance of a source location estimator that retains the frequency domain diversity of the head-related transfer function. First, a method for extracting the directional information in the subbands of a broadband signal is described, and a composite estimator based on signal subspace decomposition is introduced. The localization performance is experimentally evaluated for single and multiple source scenarios in the horizontal and vertical planes. The proposed estimator's ability to successfully localize a sound source and resolve the ambiguities in the vertical plane is demonstrated, and the impact of the source location, knowledge of the source and the effect of reverberation is discussed.

Binaural Sound Source Localization in Noisy Reverberant Environments Based on Equalization-Cancellation Theory

Binaural Sound Source Localization in Noisy Reverberant Environments Based on Equalization-Cancellation Theory