Multiple Sound Source Localization Research Articles

Counting and Localization of Multiple Unknown Harmonic Sound Sources in Entire 3D Space Using Just Five Microphones

Counting and localization of multiple sound sources in an unknown environment has many applications, namely fault diagnosis of complex machineries, hearing aids' development, and localization of enemy vehicles in defense. However, multiple sound source localization is extremely challenging because it requires numerous microphones, which increases the cost, computational time, and installation difficulties. Here, we present a new method to count the number of stationary harmonic sound sources and localize them in a free-field environment using a novel oblique square pyramid shaped array of only 5 microphones. We find FFT peaks of microphone signals and compare the ratio of the magnitude of the most dominant peak with remaining peaks' one-by-one. If these ratios are below 25 dB difference, they are considered as sound source and counted. A new time delay and intensity ratio computation is applied to the segregated source signals to calculate exact coordinates of each source. Simulation studies of four concurrent sources show that our method predicts all sources' coordinates at every octant location with mean error of 3.4%, and less than 5% error beyond 20 dB Signal-to-Noise ratio. Mean computational time is 0.92 second. Therefore, our method is low-cost, quicker, more accurate and more reliable than existing techniques.

LaSNet: An end-to-end network based on steering vector filter for sound source localization and separation

In this paper, we propose a novel time-domain end-to-end network (LaSNet) for solving the problem of multiple sound sources localization (SSL) and separation based on microphone array. The traditional time-frequency (T-F) signal representation is subject to various prior conditions and fails to separate the different sound signal components. Even the data-driven neural network does not develop an effectively integrated approach where localization and separation interplay to serve both challenge. To address the aforementioned issue, we propose a novel approach that involves the implementation of a Separation Driving Localization Network (SDLNet). This framework operates by extracting latent features from a separation network and subsequently employing them in the context of a localization network. Then we propose a simple multi-task network for both SSL and separation. Through the analysis of steering vector filter, we find that the localization and separation problems can be linked by the operation of pseudo-inverse (pinv). To facilitate a synergistic relationship between SSL and sound separation, while also enabling end-to-end network training, we develop a Pinv Module (PM). Fianlly, the Localization and Separation Network (LaSNet) structure of this paper is proposed. Inspired by the overlay mechanism of network, LasNet is extended to a multi-task and multi-layer network, in which separation task is divided into multiple subtasks. A fuzzy separation loss function is introduced for training multi-layer network. Numerical experiments demonstrate that the proposed method has a clearly better advantageous improvement than several well known models. LaSNet has greatly performance improvement in both separation and localization, and achieves at least 32% relative reduction in model size, compared with the baseline models.

Three-dimensional sound source diagnostic using a spherical microphone array from multiple capture positions

Most sound source diagnosis systems based on acoustic images are spatially stationary and unable to combine information from multiple capture positions, resulting in unreliable and partial diagnostics. In this paper it is proposed, developed and validated a measurement method based on a mobile spherical microphone array that generates acoustic images using beamforming from multiple capture positions, called Acoustic Imaging Structure From Motion (AISFM). The tridimensional localization, magnitude and directivity of multiple sound sources may be recovered with or without knowledge of the capture positions. An automatic and unsupervised acoustic image processing algorithm is deployed to facilitate the measurement workflow. An acoustic image correspondence problem is solved in order to eliminate false-positives and false-negatives from the diagnostic and render the AISFM feasible. Sound sources localization uncertainties estimates related to the sources’ magnitude, directivity and extension projected on the acoustic images are used in order to improve the diagnosis. The method and its effectiveness are experimentally validated with a measurement setup comprising five simultaneous sound sources on an anechoic environment.

A fast deconvolution method for multiple sound source localization based on Hilbert curve

To improve the resolution and efficiency of the multiple sound source localization, a fast deconvolution method based on the Hilbert curve, called CLEAN-VH, is developed in this paper. Firstly, the initial cross-spectral matrix of the measured array signal is decomposed by eigenvalue decomposition, and a new cross-spectral matrix function is constructed using the eigenvalues and eigenvectors. Then, the function is used to update the cross-spectral matrix in the process of deconvolution iteration (CLEAN), and the Hilbert curve is introduced to improve the computational efficiency of the output power in the monitoring area. Finally, the distribution of multiple sound sources can be obtained by superimposing the clean beams and residual power spectrum. The superiority of CLEAN-VH is proved by analyzing the key parameters of the method. The search area optimization based on Hilbert curve greatly alleviates the grid mismatch problem and improves the performance of the proposed method. Through the simulation and experimental comparisons with the other three algorithms, such as CLEAN, FB and CLEAN-SC, the proposed CLEAN-VH achieves high-resolution localization of multiple sound sources, better dynamic range and higher computation efficiency. The proposed method provides another choice for the localization of multiple sound sources.

A survey of sound source localization with deep learning methods.

This article is a survey of deep learning methods for single and multiple sound source localization, with a focus on sound source localization in indoor environments, where reverberation and diffuse noise are present. We provide an extensive topography of the neural network-based sound source localization literature in this context, organized according to the neural network architecture, the type of input features, the output strategy (classification or regression), the types of data used for model training and evaluation, and the model training strategy. Tables summarizing the literature survey are provided at the end of the paper, allowing a quick search of methods with a given set of target characteristics.

Prediction model formulations for detection, enumeration, and localization of multiple sound sources using spherical harmonics

A spherical microphone array is used to detect and localize sound sources in terms of model-based machine learning (ML). In this application, it is crucial to establish parametric models to distinguish background sound environment from presence of sound sources. In the presence of sound sources, the parameter models are also used to localize an unknown number of potentially multiple sound sources. In this work, a model-based Bayesian learning framework is presented for localizing an unknown number of sound sources. Among them, a no-source scenario needs to be accounted for. The model-based machine learning applies the model comparison between the no-source model and the one-source model for sound source detection. After detecting sound sources, the machine learning needs to involve sound source enumeration and localization in order to correctly localize potential multiple sound sources. Specifically, sound environment is analyzed using Bayesian model comparison of two different models accounting for absence and presence of the sound sources for source detection. Upon a positive detection, potentially multiple source models are involved to analyze direction of arrivals (DoAs) for far-field and to localize sound sources for near-field including source distances, amplitudes, and DoAs using Bayesian model selection and parameter estimation.

Multiple Sound Source Localization, Separation, and Reconstruction by Microphone Array: A DNN-Based Approach

Synchronistical localization, separation, and reconstruction for multiple sound sources are usually necessary in various situations, such as in conference rooms, living rooms, and supermarkets. To improve the intelligibility of speech signals, the application of deep neural networks (DNNs) has achieved considerable success in the area of time-domain signal separation and reconstruction. In this paper, we propose a hybrid microphone array signal processing approach for the nearfield scenario that combines the beamforming technique and DNN. Using this method, the challenge of identifying both the sound source location and content can be overcome. Moreover, the use of a sequenced virtual sound field reconstruction process enables the proposed approach to be quite suitable for a sound field which contains a dominant, stronger sound source and masked, weaker sound sources. Using this strategy, all traceable, mainly sound, sources can be discovered by loops in a given sound field. The operational duration and accuracy of localization are further improved by substituting the broadband weighted multiple signal classification (BW-MUSIC) method for the conventional delay-and-sum (DAS) beamforming algorithm. The effectiveness of the proposed method for localizing and reconstructing speech signals was validated by simulations and experiments with promising results. The localization results were accurate, while the similarity and correlation between the reconstructed and original signals was high.

Efficient energy-based orthogonal matching pursuit algorithm for multiple sound source localization with unknown source count

In this paper, we propose a compressed sensing (CS) sound source localization algorithm based on signal energy to solve the problem of stopping the iteration condition of the orthogonal matching pursuit (OMP) reconstruction algorithm in CS. The orthogonal matching tracking algorithm needs to stop iteration according to the number of sound sources or the change of residual. Generally, the number of sound sources cannot be known in advance, and the residual often leads to unnecessary calculation. Because the sound source is sparsely distributed in space, and its energy is concentrated and higher than that of the environmental noise, the comparison of the signal energy at different positions in each iteration reconstruction signal is used to determine whether the new sound source is added in this iteration. At the same time, the block sparsity is introduced by using multiple frequency points to avoid the problem of different iteration times for different frequency points in the same frame caused by the uneven energy distribution in the signal frequency domain. Simulation and experimental results show that the proposed algorithm retains the advantages of the orthogonal matching tracking sound source localization algorithm, and can complete the iteration well. Under the premise of not knowing the number of sound sources, the maximum error between the number of iterations and the set number of sound sources is 0.31. The experimental results show that the proposed algorithm has good positioning accuracy and has certain anti-reverberation capability. Compared with other OMP algorithms, the proposed algorithm has better iterative ability and stability. This work is helpful in promoting the development of multiple sound source localization.

A Multiple Sound Source Localization and Counting Method Based on the Kernel Density Estimator

An unambiguous signal processing algorithm when using a wide intermicrophone distance is proposed in this paper for simultaneously locating and counting multiple active sound sources. Based on the kernel density estimator, a multistage structure in the time-frequency domain is used to suppress the influence of spatial aliasing, then the pooled angular spectrum is combined with a peak search method having an updated cut-off threshold and a source merging module. Complete source localization and counting is realized through the combination of these two steps. Simulation results show that the proposed method has a more robust performance than the classic counterpart, especially in adverse environments with spatial aliasing, reverberation, and interference between different sound sources.

3D Multiple Sound Source Localization by Proposed T-Shaped Circular Distributed Microphone Arrays in Combination with GEVD and Adaptive GCC-PHAT/ML Algorithms.

Multiple simultaneous sound source localization (SSL) is one of the most important applications in the speech signal processing. The one-step algorithms with the advantage of low computational complexity (and low accuracy), and the two-step methods with high accuracy (and high computational complexity) are proposed for multiple SSL. In this article, a combination of one-step-based method based on the generalized eigenvalue decomposition (GEVD), and a two-step-based method based on the adaptive generalized cross-correlation (GCC) by using the phase transform/maximum likelihood (PHAT/ML) filters along with a novel T-shaped circular distributed microphone array (TCDMA) is proposed for 3D multiple simultaneous SSL. In addition, the low computational complexity advantage of the GCC algorithm is considered in combination with the high accuracy of the GEVD method by using the distributed microphone array to eliminate spatial aliasing and thus obtain more appropriate information. The proposed T-shaped circular distributed microphone array-based adaptive GEVD and GCC-PHAT/ML algorithms (TCDMA-AGGPM) is compared with hierarchical grid refinement (HiGRID), temporal extension of multiple response model of sparse Bayesian learning with spherical harmonic (SH) extension (SH-TMSBL), sound field morphological component analysis (SF-MCA), and time-frequency mixture weight Bayesian nonparametric acoustical holography beamforming (TF-MW-BNP-AHB) methods based on the mean absolute estimation error (MAEE) criteria in noisy and reverberant environments on simulated and real data. The superiority of the proposed method is presented by showing the high accuracy and low computational complexity for 3D multiple simultaneous SSL.

Multiple Sound Source Localization in Three Dimensions Using Convolutional Neural Networks and Clustering Based Post-Processing

Sound source localization methods are successfully applied for various estimation tasks, such as tracking and detecting objects, aiming cameras, and navigating robots. However, large and usually complex distributed microphone arrays are used for three-dimensional acoustic source localization. This study proposes a convolutional neural network architecture for three-dimensional sound source localization using a single tetrahedral microphone array. A spectrum phase component of a microphone array signal was designed as the input of the model, while the output represents a three-dimensional space. The paper provides extensive experimental results of the given method on a semi-synthetic audio data set and a real-world microphone array. Furthermore, cluster-based post-processing has been shown to increase the accuracy of three-dimensional localization by more than 30%. The experimental results on a synthesized data set using the image source method showed 1.08 m localization uncertainties. The estimate of the investigated sound sources had a mean absolute error of 18.97° and elevation error of 48.49°. An additional advantage of the proposed method is the ability to predict the location of the sound source from a single signal analysis frame. This gives instant localization and is in line with many alternative applications. The proposed solution does not require intensive preprocessing of the audio signals and can be used as a video camera pointing system based on a microphone array. In the future, it would be relevant to investigate the localization performance of more than two sound sources, and the variable acoustic conditions could also be assessed.

Deep Learning-Enabled High-Resolution and Fast Sound Source Localization in Spherical Microphone Array System

While sound source localization (SSL) using a spherical microphone array system can be applied to obtain visual beam patterns of source distribution maps in a range of omnidirectional acoustic applications, the present challenges of the spherical measurement system on the valid frequency ranges and the spatial distortion as well as the grid-related limitations of data-driven SSL approaches raise the need to develop an appropriate method. Imbued by these challenges, this study proposes a deep learning approach to achieve the high-resolution performance of localizing multiple sound sources tailored for omnidirectional acoustic applications. First, we present a spherical target map representation that can panoramically pinpoint the position and strength information of multiple sound sources without any grid-related constraints. Then, a dual-branched spherical convolutional autoencoder is proposed to obtain high-resolution localization results from the conventional spherical beamforming maps while incorporating frequency-variant and distortion-invariant strategies to address the inherent challenges. We quantitatively and qualitatively assess our proposed method’s localization capability for multiple sound sources and validate that the proposed method can achieve far more precise and computationally-efficient results than the existing approaches. By extension, we newly present the experimental setup that can create omnidirectional acoustic scenarios for the multiple sound source localization. By evaluating our proposed method in this experimental setup, we demonstrate the effectiveness and applicability of the proposed method with the experimental data. Our study delivers the proposed approach’s potential of being utilized in various sound source localization applications.

Multi-source localization by using offset residual weight

Multiple sound source localization is a hot issue of concern in recent years. The Single Source Zone (SSZ) based localization methods achieve good performance due to the detection and utilization of the Time-Frequency (T-F) zone where only one source is dominant. However, some T-F points consisting of components from multiple sources are also included in the detected SSZ sometimes. Once a T-F point in SSZ is contributed by multiple components, this point is defined as an outlier. The existence of outliers within the detected SSZ is usually an unavoidable problem for SSZ-based methods. To solve this problem, a multi-source localization by using offset residual weight is proposed in this paper. In this method, an assumption is developed: the direction estimated by all the T-F points within the detected SSZ has a difference along with the actual direction of sources. But this difference is much smaller than the difference between the directions estimated by the outliers along with the actual source localization. After verifying this assumption experimentally, Point Offset Residual Weight (PORW) and Source Offset Residual Weight (SORW) are proposed to reduce the influence of outliers on the localization results. Then, a composite weight is formed by combining PORW and SORW, which can effectively distinguish the outliers and desired points. After that, the outliers are removed by composite weight. Finally, a statistical histogram of DOA estimation with outliers removed is used for multi-source localization. The objective evaluation of the proposed method is conducted in various simulated environments. The results show that the proposed method achieves a better performance compared with the reference methods in sources localization.

Deep learning-based method for multiple sound source localization with high resolution and accuracy

Deep learning-based methods are attracting interest in sound source localization, showing promising results compared to conventional model-based approaches. While these deep learning-based methods have been mainly developed into two approaches, i.e., grid-based and grid-free methods, they inherently involve several limitations that the sound sources should be assumed on the grid points or the number of sound sources should be pre-defined when constructing a deep neural network’s architecture. Breaking away from the existing methods’ limitations, we propose a deep learning approach to fulfill multiple sound source localization with high resolution and accuracy, for whether the sound sources are located on the grid points or not. We first suggest a target function to obtain spatial source distribution maps, that can represent multiple sources’ positional and strength information, even when the sources are placed off the grid points. While the multiple sound source localization is expanded by the proposed source map into image-to-image pixel-level prediction task, we then propose a fully convolutional neural network (FCN) with an encoder-decoder structure to estimate the multiple sources’ positions and strength precisely. Based on the dataset acquired by one to three monopole sources on a square plane of 2.68 × 2.68 m, with a spiral array of 60 microphones at 1, 2, and 10 kHz, we assess both quantitative and qualitative results of the proposed model and demonstrate that our proposed model can achieve highly precise localization results regardless of frequency and the number of sound sources. Besides, we validate that high-resolution source distribution maps can be obtained by the proposed model, from which the positions and the strengths of sound sources are accurately predicted. Lastly, we compare the proposed model with several deconvolution methods, and the results show that the proposed deep learning model significantly outperforms the model-based methods.

Millimeter-Wave Bat for Mapping and Quantifying Micromotions in Full Field of View.

Echolocating bats possess remarkable capability of multitarget spatial localization and micromotion sensing in a full field of view (FFOV) even in cluttered environments. Artificial technologies with such capability are highly desirable for various fields. However, current techniques such as visual sensing and laser scanning suffer from numerous fundamental problems. Here, we develop a bioinspired concept of millimeter-wave (mmWave) full-field micromotion sensing, creating a unique mmWave Bat (“mmWBat”), which can map and quantify tiny motions spanning macroscopic to μm length scales of full-field targets simultaneously and accurately. In mmWBat, we show that the micromotions can be measured via the interferometric phase evolution tracking from range-angle joint dimension, integrating with full-field localization and tricky clutter elimination. With our approach, we demonstrate the capacity to solve challenges in three disparate applications: multiperson vital sign monitoring, full-field mechanical vibration measurement, and multiple sound source localization and reconstruction (radiofrequency microphone). Our work could potentially revolutionize full-field micromotion monitoring in a wide spectrum of applications, while may inspiring novel biomimetic wireless sensing systems.

Multiple Sound Source Localization and Counting Using One Pair of Microphones in Noisy and Reverberant Environments

A multiple sound source localization and counting method based on an angular spectrum is proposed in this paper. Local signal-to-noise ratio tracking, onset detection, and a coherence test are introduced to filter the generalized cross-correlation angular spectrum in the time-frequency domain for multiple sound source localization and counting in noisy and reverberant environments. Then, dual-width matching pursuit is introduced to replace peak search as the method of localization and counting. A comprehensive comparison of two statistical indicators, mean precision and mean absolute estimated error, indicates that the proposed localization and counting algorithm using both the filtered angular spectrum and dual-width matching pursuit method is more robust and accurate than the classic counterpart, especially in environments with low signal-to-noise ratio, strong reverberation, and abundant sound sources.

3D Multiple Sound Source Localization by Proposed Cuboids Nested Microphone Array in Combination with Adaptive Wavelet-Based Subband GEVD

Sound source localization is one of the applicable areas in speech signal processing. The main challenge appears when the aim is a simultaneous multiple sound source localization from overlapped speech signals with an unknown number of speakers. Therefore, a method able to estimate the number of speakers, along with the speaker’s location, and with high accuracy is required in real-time conditions. The spatial aliasing is an undesirable effect of the use of microphone arrays, which decreases the accuracy of localization algorithms in noisy and reverberant conditions. In this article, a cuboids nested microphone array (CuNMA) is first proposed for eliminating the spatial aliasing. The CuNMA is designed to receive the speech signal of all speakers in different directions. In addition, the inter-microphone distance is adjusted for considering enough microphone pairs for each subarray, which prepares appropriate information for 3D sound source localization. Subsequently, a speech spectral estimation method is considered for evaluating the speech spectrum components. The suitable spectrum components are selected and the undesirable components are denied in the localization process. The speech information is different in frequency bands. Therefore, the adaptive wavelet transform is used for subband processing in the proposed algorithm. The generalized eigenvalue decomposition (GEVD) method is implemented in sub-bands on all nested microphone pairs, and the probability density function (PDF) is calculated for estimating the direction of arrival (DOA) in different sub-bands and continuing frames. The proper PDFs are selected by thresholding on the standard deviation (SD) of the estimated DOAs and the rest are eliminated. This process is repeated on time frames to extract the best DOAs. Finally, K-means clustering and silhouette criteria are considered for DOAs classification in order to estimate the number of clusters (speakers) and the related DOAs. All DOAs in each cluster are intersected for estimating the position of the 3D speakers. The closest point to all DOA planes is selected as a speaker position. The proposed method is compared with a hierarchical grid (HiGRID), perpendicular cross-spectra fusion (PCSF), time-frequency wise spatial spectrum clustering (TF-wise SSC), and spectral source model-deep neural network (SSM-DNN) algorithms based on the accuracy and computational complexity of real and simulated data in noisy and reverberant conditions. The results show the superiority of the proposed method in comparison with other previous works.

Adaptive Speech Separation Based on Beamforming and Frequency Domain-Independent Component Analysis

Voice signals acquired by a microphone array often include considerable noise and mutual interference, seriously degrading the accuracy and speed of speech separation. Traditional beamforming is simple to implement, but its source interference suppression is not adequate. In contrast, independent component analysis (ICA) can improve separation, but imposes an iterative and time-consuming process to calculate the separation matrix. As a supporting method, principle component analysis (PCA) contributes to reduce the dimension, retrieve fast results, and disregard false sound sources. Considering the sparsity of frequency components in a mixed signal, we propose an adaptive fast speech separation algorithm based on multiple sound source localization as preprocessing to select between beamforming and frequency domain ICA according to different mixing conditions per frequency bin. First, a fast positioning algorithm allows calculating the maximum number of components per frequency bin of a mixed speech signal to prevent the occurrence of false sound sources. Then, PCA reduces the dimension to adaptively adjust the weight of beamforming and ICA for speech separation. Subsequently, the ICA separation matrix is initialized based on the sound source localization to notably reduce the iteration time and mitigate permutation ambiguity. Simulation and experimental results verify the effectiveness and speedup of the proposed algorithm.

Indoor Multiple Sound Source Localization via Multi-Dimensional Assignment Data Association

In this paper, we address the multiple sound source localization problem by associating and fusing the direction of arrival DOA estimates from multiple microphone arrays. For multi-source scenarios especially in indoor environments, a critical issue is to tell the correspondence among DOA estimates across different arrays, which is known as the data association problem. We propose a multi-dimensional assignment-based data association approach to find the optimal associations of DOA estimates from the same source. First, in the sense of maximum likelihood, the data association problem is formulated by finding the most likely partition of the measurement set into the source-originated and false alarm-originated subsets. Next, by defining the association costs appropriately, the problem of finding the most likely measurement partition is transformed into a generalized multi-dimensional assignment problem which can be solved efficiently by a Lagrangian relaxation algorithm. After the optimal associations of DOA estimates across different arrays are obtained, the locations of sources can be estimated by fusing the same source-originated DOA estimates. In the presence of missed detections, false alarms and the unknown number of sources, the proposed method achieves high accuracy in data association and localization, and outperforms the competing method in reverberant and noisy environments. In addition, since our method does not require additional features and uses DOA estimates only, it is more computationally efficient than the competing method.

Reflection Assisted Sound Source Localization Through a Harmonic Domain MUSIC Framework

This work presents a method that persuades acoustic reflections to be a favorable property for sound source localization. Whilst most real world spatial audio applications utilize prior knowledge of sound source position, estimating such positions in reverberant environments is still considered to be a difficult problem due to acoustic reflections. This article presents a novel MUSIC framework for multiple sound source localization (range, elevation, azimuth) in reverberant rooms by incorporating a recently proposed region-to-region room transfer model. The method is built upon the received signals of a higher order microphone and a spherical harmonic representation of the room transfer function. We demonstrate the method's general applicability and multiple source localization performance through a simulation study across an assortment of reverberant conditions. Additionally, we investigate robustness against various system modeling errors to gauge implementation viability. Finally, we prove the method in a practical experiment inside a real-world room with measured region-to-region transfer function parameters.