Speech Source Localization Research Articles

Method and practice of microphone array speech source localization based on sound propagation modeling

Abstract This paper realizes the speech source localization for microphone arrays based on the sound propagation model. According to the actual environment and location of the sound source, this paper divides the sound source into far-field source and near-field source and constructs the far-field sound model and near-field sound model applicable to the microphone array. The TDOA time-delayed localization algorithm is employed to locate the voice source of the microphone array by judging the sound far and near the field. In the localization test, this paper selects microphones to form an array according to the actual needs and preprocesses the sound signal data required for practice. The preprocessing data and sound source localization practice prove that the microphone array speech source localization algorithm used in this paper can effectively estimate the actual position of the sound source, and the absolute error between its estimated sound source position and the actual sound source position is only about 0.3m.

Deep Learning-Based Speech Specific Source Localization by Using Binaural and Monaural Microphone Arrays in Hearing Aids

A deep learning-based method is proposed for jointly detecting and localizing speech sources in a complex acoustic scene by using microphones of a hearing aid. Motivated by the human auditory system, peripheral preprocessing is applied on the microphone signals to obtain auditory subband signals that serve as input to the proposed deep neural network for detecting and localizing speech sources. In the proposed neural network, a combination of residual and dense aggregation learning is utilized rather than the conventional residual learning to preserve and reuse the spatial representations at the output layers. This process is performed to improve the gradient flow in deeper layers, in the training stage. The learning curves show that the proposed residual-dense aggregation mapping do improve the speed and accuracy of the convergence. The proposed model shows good performance in joint speech source detection and localization using a binaural microphone array (i.e., three channels at each side) but also using a monaural microphone array (i.e., four channels at the right side) despite of the short distances between the microphones. The proposed methods also outperform neural networks that are directly using STFT components of the binaural or monaural microphone arrays. In addition, the proposed models extended with learnable peripheral processing show slightly improved in detection and localization scores compared to the proposed models using the plain auditory subband signals, in both the binaural and monaural microphone arrays but only so, when the learnable peripheral processing is initialized with parameters stemming from human peripheral processing.

Speech Localization at Low Bitrates in Wireless Acoustics Sensor Networks

The use of speech source localization (SSL) and its applications offer great possibilities for the design of speaker local positioning systems with wireless acoustic sensor networks (WASNs). Recent works have shown that data-driven front-ends can outperform traditional algorithms for SSL when trained to work in specific domains, depending on factors like reverberation and noise levels. However, such localization models consider localization directly from raw sensor observations, without consideration for transmission losses in WASNs. In contrast, when sensors reside in separate real-life devices, we need to quantize, encode and transmit sensor data, decreasing the performance of localization, especially when the transmission bitrate is low. In this work, we investigate the effect of low bitrate transmission on a Direction of Arrival (DoA) estimator. We analyze a deep neural network (DNN) based framework performance as a function of the audio encoding bitrate for compressed signals by employing recent communication codecs including PyAWNeS, Opus, EVS, and Lyra. Experimental results show that training the DNN on input encoded with the PyAWNeS codec at 16.4 kB/s can improve the accuracy significantly, and up to 50% of accuracy degradation at a low bitrate for almost all codecs can be recovered. Our results further show that for the best accuracy of the trained model when one of the two channels can be encoded with a bitrate higher than 32 kB/s, it is optimal to have the raw data for the second channel. However, for a lower bitrate, it is preferable to similarly encode the two channels. More importantly, for practical applications, a more generalized model trained with a randomly selected codec for each channel, shows a large accuracy gain when at least one of the two channels is encoded with PyAWNeS.

An Adaptive Method Based on Multiscale Dilated Convolutional Network for Binaural Speech Source Localization

Most binaural speech source localization models perform poorly in unprecedentedly noisy and reverberant situations. Here, this issue is approached by modelling a multiscale dilated convolutional neural network (CNN). The time-related crosscorrelation function (CCF) and energy-related interaural level differences (ILD) are preprocessed in separate branches of dilated convolutional network. The multiscale dilated CNN can encode discriminative representations for CCF and ILD, respectively. After encoding, the individual interaural representations are fused to map source direction. Furthermore, in order to improve the parameter adaptation, a novel semiadaptive entropy is proposed to train the network under directional constraints. Experimental results show the proposed method can adaptively locate speech sources in simulated noisy and reverberant environments.

Spectral Flux-Based Convolutional Neural Network Architecture for Speech Source Localization and Its Real-Time Implementation.

In this article, we present a real-time convolutional neural network (CNN)-based Speech source localization (SSL) algorithm that is robust to realistic background acoustic conditions (noise and reverberation). We have implemented and tested the proposed method on a prototype (Raspberry Pi) for real-time operation. We have used the combination of the imaginary-real coefficients of the short-time Fourier transform (STFT) and Spectral Flux (SF) with delay-and-sum (DAS) beamforming as the input feature. We have trained the CNN model using noisy speech recordings collected from different rooms and inference on an unseen room. We provide quantitative comparison with five other previously published SSL algorithms under several realistic noisy conditions, and show significant improvements by incorporating the Spectral Flux (SF) with beamforming as an additional feature to learn temporal variation in speech spectra. We perform real-time inferencing of our CNN model on the prototyped platform with low latency (21 milliseconds (ms) per frame with a frame length of 30 ms) and high accuracy (i.e. 89.68% under Babble noise condition at 5dB SNR). Lastly, we provide a detailed explanation of real-time implementation and on-device performance (including peak power consumption metrics) that sets this work apart from previously published works. This work has several notable implications for improving the audio-processing algorithms for portable battery-operated Smart loudspeakers and hearing improvement (HI) devices.

Multiple Speech Sources Localization in Room Reverberant Environment Using Spherical Harmonic Sparse Bayesian Learning

In recent years, sparse representation techniques have been proposed for source localization using spherical microphone arrays (SMAs). However, the performance of these sparse representation techniques for SMAs degrades for speech source localization in the room environment due to sound reverberation. This article proposes a robust sparse presentation method for localization of multiple speech sources in the room environment using an SMA, which employs the spherical harmonic temporal extension of multiple response model sparse Bayesian learning. Real-world experimental results demonstrate that the proposed method outperforms its existing counterparts for speech source localization in the real room environment.

Real-Time Convolutional Neural Network-Based Speech Source Localization on Smartphone

In this paper, we present a real-time convolutional neural network (CNN) based approach for speech source localization (SSL) using Android-based smartphone and its two built-in microphones under noisy conditions. We propose a new input feature set - using real and imaginary parts of the short-time Fourier transform (STFT) for CNN-based SSL. We use simulated noisy data from popular datasets that was augmented with few hours of real recordings collected on smartphones to train our CNN model. We compare the proposed method to recent CNN-based SSL methods that are trained on our dataset and show that our CNN-based SSL method offers higher accuracy on identical test datasets. Another unique aspect of this work is that we perform real-time inferencing of our CNN model on an Android smartphone with low latency (14 milliseconds(ms) for single frame-based estimation, 180 ms for multi frame-based estimation and frame length is 20 ms for both cases) and high accuracy (i.e. 88.83% at 0dB SNR). We show that our CNN model is rather robust to smartphone hardware mismatch, hence we may not need to retrain the entire model again for use with different smartphones. The proposed application provides a 'visual' indication of the direction of a talker on the screen of Android smartphones for improving the hearing of people with hearing disorders.

Correction to: Non-Uniform Microphone Arrays for Robust Speech Source Localization for Smartphone-Assisted Hearing Aid Devices

In the original publication of the article, Fig. 1 was incorrect. The correct figure is shown below.

Non-Uniform Microphone Arrays for Robust Speech Source Localization for Smartphone-Assisted Hearing Aid Devices.

Robust speech source localization (SSL) is an important component of the speech processing pipeline for hearing aid devices (HADs). SSL via time direction of arrival (TDOA) estimation has been known to improve performance of HADs in noisy environments, thereby providing better listening experience for hearing aid users. Smartphones now possess the capability to connect to the HADs through wired or wireless channel. In this paper, we present our findings about the non-uniform non-linear microphone array (NUNLA) geometry for improving SSL for HADs using an L-shaped three-element microphone array available on modern smartphones. The proposed method is implemented on a frame-based TDOA estimation algorithm using a modified Dictionary-based singular value decomposition method (SVD) method for localizing single speech sources under very low signal to noise ratios (SNR). Unlike most methods developed for uniform microphone arrays, the proposed method has low spatial aliasing as well as low spatial ambiguity while providing a robust low-error with 360° DOA scanning capability. We present the comparison among different types of microphone arrays, as well as compare their performance using the proposed method.

Window-Dominant Signal Subspace Methods for Multiple Short-Term Speech Source Localization

Signal subspace has been widely exploited to localize multiple speech sources. However, most signal subspace methods cannot count the number of sources, and do not make use of speech sparsity in the frequency domain. This paper presents a grid search window-dominant signal subspace GS-WDSS method and a closed-form WDSS CF-WDSS method to localize short-term speech sources. Such methods are based upon the generalized sparsity assumption that each window containing some time-adjacent bins is dominated by one source, as opposed to the conventional assumption that each individual bin is dominated by one source. The generalized assumption enables the principal eigenvector of the spatial correlation matrix on each window to span the signal subspace of the window-dominant source. The direction-of-arrival DOA of the dominant source is estimated from the principal eigenvector. The DOAs and the number of sources are eventually summarized from the DOA histogram of all dominant sources. The conventional assumption is a special case of the generalized assumption. By using the generalized assumption, the performance in estimating DOAs of the window-dominant sources is significantly improved at the cost of acceptable masking effect. The superiority of the proposed methods is verified by simulated and real experiments.

Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion.

Speaker diarization consists of assigning speech signals to people engaged in a dialogue. An audio-visual spatiotemporal diarization model is proposed. The model is well suited for challenging scenarios that consist of several participants engaged in multi-party interaction while they move around and turn their heads towards the other participants rather than facing the cameras and the microphones. Multiple-person visual tracking is combined with multiple speech-source localization in order to tackle the speech-to-person association problem. The latter is solved within a novel audio-visual fusion method on the following grounds: binaural spectral features are first extracted from a microphone pair, then a supervised audio-visual alignment technique maps these features onto an image, and finally a semi-supervised clustering method assigns binaural spectral features to visible persons. The main advantage of this method over previous work is that it processes in a principled way speech signals uttered simultaneously by multiple persons. The diarization itself is cast into a latent-variable temporal graphical model that infers speaker identities and speech turns, based on the output of an audio-visual association process, executed at each time slice, and on the dynamics of the diarization variable itself. The proposed formulation yields an efficient exact inference procedure. A novel dataset, that contains audio-visual training data as well as a number of scenarios involving several participants engaged in formal and informal dialogue, is introduced. The proposed method is thoroughly tested and benchmarked with respect to several state-of-the art diarization algorithms.

Binary Sparse Coding of Convolutive Mixtures for Sound Localization and Separation via Spatialization

We propose a sparse coding approach to address the problem of source-sensor localization and speech reconstruction. This approach relies on designing a dictionary of spatialized signals by projecting the microphone array recordings into the array manifolds characterized for different locations in a reverberant enclosure using the image model. Sparse representation over this dictionary enables identifying the subspace of the actual recordings and its correspondence to the source and sensor locations. The speech signal is reconstructed by inverse filtering the acoustic channels associated to the array manifolds. We provide rigorous analysis on the optimality of speech reconstruction by elucidating the links between inverse filtering and source separation followed by deconvolution. This procedure is evaluated for localization, reconstruction and recognition of simultaneous speech sources using real data recordings. The results demonstrate the effectiveness of the proposed approach and compare favorably against beamforming and independent component analysis techniques.

Computational methods for underdetermined convolutive speech localization and separation via model-based sparse component analysis

In this paper, the problem of speech source localization and separation from recordings of convolutive underdetermined mixtures is addressed. This problem is cast as recovering the spatio-spectral speech information embedded in a microphone array compressed measurements of the acoustic field. A model-based sparse component analysis framework is formulated for sparse reconstruction of the speech spectra in a reverberant acoustic resulting in joint localization and separation of the individual sources. We compare and contrast the algorithmic approaches to model-based sparse recovery exploiting spatial sparsity as well as spectral structures underlying spectrographic representation of speech signals. In this context, we explore identification of the sparsity structures at the auditory and acoustic representation spaces. The audiory structures are formulated upon the principles of structural grouping based on proximity, autoregressive correlation and harmonicity of the spectral coefficients and they are incoporated for sparse reconstruction. The acoustic structures are formulated upon the image model of multipath propagation and they are exploited to characterize the compressive measurement matrix associated with microphone array recordings.Three approaches to sparse recovery relying on combinatorial optimization, convex relaxation and sparse Bayesian learning are studied and evaluated on thorough experiments. The sparse Bayesian learning method is shown to yield better perception quality while the interference suppression is also achieved using the combinatorial approach with the advantage of offering the most efficient computational cost. Furthermore, it is demonstrated that an average autoregressive model can be learned for speech localization while exploiting the proximity structure in the form of block sparse coefficients enables accurate localization and high quality speech separation. Throughout the extensive empirical evaluation, we confirm that a large and random placement of the microphones enables significant improvement in source localization and separation performance.

Robust and Fast Localization of Single Speech Source Using a Planar Array

Heavy computational load and acoustic interferences are two major problems to speech source localization in real applications. Conventional methods can mitigate one problem, but deteriorate the other. This letter proposes an algorithm of direction-of-arrival (DOA) estimation, which is both computationally efficient and robust in the presence of acoustic interferences. The robustness is considered in two aspects. One is the eigenanalysis-based enhancement to reduce acoustic interferences such as noise and reverberation. The other is the coefficients that weight the pairwise time delays to mitigate the effect of delay outliers on DOA. The high computational efficiency is achieved by making use of a concave cost function, from which, the optimal estimate of DOA is given by a closed-form solution. The grid-search method often adopted in conventional algorithms is no longer used in this algorithm. We conduct some experiments in both simulated and real environments with a 9-element circular array. The proposed algorithm runs about ten times faster than Steered Response Power PHAse Transform (SRP-PHAT), and outperforms SRP-PHAT in terms of robustness.

A Speech Recognizing Circuit for Home Service Robot

Due to the increased customer demand for various applications in various fields, service robots are becoming very popular. Human Robot Interaction (HRI) is very essential for the good performance of the home service robot. A robot can be controlled by many methods and here the robot is controlled by voice commands which are given orally. The robot must be able to perform various tasks such as speech source localization, source separation and classification etc. The robot must be capable to identify the source accurately. The speech recognizer and a speaker classifier are used to improve the performance of the robot. Reverberation deteriorates the quality and intelligibility of speech. This leads to the poor performance of classification systems. The noise effects can also degrade the performance of the classification systems. The channel may also have ISI. So channel must be equalized to nullify these effects. Here a new blind deconvolutive neural network is used for speech classification in the robot. Using this new method the robot can identify the commands given to it more accurately than by the existing methods.

A two-microphone method for localization of multiple speech sources using complex exponential transform of phase differences

This paper proposes a two-microphone method for localization of multiple speech sources by utilizing speech's sparse attribute in time-frequency domain. The proposed method estimates time-delay of each source by applying complex exponential transform to the inter-channel phase differences (IPDs). In order to improve the performance in noisy environment, the proposed method selects time-frequency points with high SNR. The method obtains the initial time-delay estimate of each speech source by utilizing the IPDs at low frequencies, and then iteratively updates the time-delay by utilizing the whole selected points. With the complex exponential transform on IPDs, the proposed method takes full advantage of the high-frequency phase information, not requiring phase compensation for IPDs at high frequencies. Experiments have been conducted to study the effect of the exponential factor on the performance of the proposed method and to compare the performance of the proposed method with the generalized hard cluster...

Evaluation of generalized cross-correlation methods for direction of arrival estimation using two microphones in real environments

The localization of sound sources, and particularly speech, has a numerous number of applications to the industry. This has motivated a continuous effort in developing robust direction-of-arrival detection algorithms, in order to overcome the limitations imposed by real scenarios, such as multiple reflections and undesirable noise sources. Time difference of arrival-based methods, and particularly, generalized cross-correlation approaches have been widely investigated in acoustic signal processing, but there is considerable lack in the technical literature about their evaluation in real environments when only two microphones are used. In this work, four generalized cross-correlation methods for localization of speech sources with two microphones have been analyzed in different real scenarios with a stationary noise source. Furthermore, these scenarios have been acoustically characterized, in order to relate the behavior of these cross-correlation methods with the acoustic properties of noisy scenarios. The scope of this study is not only to assess the accuracy and reliability of a set of well-known localization algorithms, but also to determine how the different acoustic properties of the room under analysis have a determinant influence in the final results, by incorporating in the analysis additional factors to the reverberation time and signal-to-noise ratio. Results of this study have outlined the influence of the acoustic properties analysed in the performance of these methods.

A Novel Multiple Sparse Source Localization Using Triangular Pyramid Microphone Array

Making use of the time-frequency spectra sparsity of the speech sources and the spatial and inter-relation information provided from a triangular pyramid microphone array (TPMA), the ratio of the inter-sensor phase difference (RIPD) is defined and a direct relationship between RIPD information and the direction of arrival (DOA) of each source is obtained. A novel multiple speech source localization algorithm (named as TPMA-RIPD) using the histogram clustering technique is proposed, which has been evaluated by several simulation experiments. Experimental results show that the TPMA-RIPD algorithm is able to provide high source localization accuracy in noisy environment for all angles. It is also able to estimate multiple speech sources when the number of sources is larger than that of the microphones used.

Source localization for multiple speech sources using low complexity non-parametric source separation and clustering

This article presents a new method for localization of multiple concurrent speech sources that relies on simultaneous blind signal separation and direction of arrival (DOA) estimation, as well as a method to solve the intersection point selection problem that arises when locating multiple speech sources using multiple sensor arrays. The proposed method is based on a low complexity non-parametric blind signal separation method, making is suitable for real-time applications on embedded platforms. On top of reduced complexity in comparison to a previously presented method, the DOA estimation accuracy is also improved. Evaluation of the performance is done with both real recording and simulations, and a real-time prototype of the proposed method has been implemented on a DSP platform to evaluate the computational and the memory complexities in a real application.

Performance Improvement of TDOA-Based Speaker Localization in Joint Noisy and Reverberant Conditions

TDOA- (time difference of arrival-) based algorithms are common methods for speech source localization. The generalized cross correlation (GCC) method is the most important approach for estimating TDOA between microphone pairs. The performance of this method significantly degrades in the presence of noise and reverberation. This paper addresses the problem of 3D localization in joint noisy and reverberant conditions and a single-speaker scenario. We first propose a modification to make the GCC-PHAse transform (GCC-PHAT) method robust against environment noise. Then, we use an iterative technique that employs location estimation to improve TDOAs accuracy. Extensive experiments on both simulated and real (practical) data (in a single-source scenario) show the capability of the proposed methods to significantly improve TDOA accuracy and, consequently, source location estimates.