Number Of Sound Sources Research Articles

Deep learning-based method for multiple sound source localization with high resolution and accuracy

Deep learning-based methods are attracting interest in sound source localization, showing promising results compared to conventional model-based approaches. While these deep learning-based methods have been mainly developed into two approaches, i.e., grid-based and grid-free methods, they inherently involve several limitations that the sound sources should be assumed on the grid points or the number of sound sources should be pre-defined when constructing a deep neural network’s architecture. Breaking away from the existing methods’ limitations, we propose a deep learning approach to fulfill multiple sound source localization with high resolution and accuracy, for whether the sound sources are located on the grid points or not. We first suggest a target function to obtain spatial source distribution maps, that can represent multiple sources’ positional and strength information, even when the sources are placed off the grid points. While the multiple sound source localization is expanded by the proposed source map into image-to-image pixel-level prediction task, we then propose a fully convolutional neural network (FCN) with an encoder-decoder structure to estimate the multiple sources’ positions and strength precisely. Based on the dataset acquired by one to three monopole sources on a square plane of 2.68 × 2.68 m, with a spiral array of 60 microphones at 1, 2, and 10 kHz, we assess both quantitative and qualitative results of the proposed model and demonstrate that our proposed model can achieve highly precise localization results regardless of frequency and the number of sound sources. Besides, we validate that high-resolution source distribution maps can be obtained by the proposed model, from which the positions and the strengths of sound sources are accurately predicted. Lastly, we compare the proposed model with several deconvolution methods, and the results show that the proposed deep learning model significantly outperforms the model-based methods.

Multichannel environmental sound segmentation

This paper proposes a multichannel environmental sound segmentation method. Environmental sound segmentation is an integrated method to achieve sound source localization, sound source separation and classification, simultaneously. When multiple microphones are available, spatial features can be used to improve the localization and separation accuracy of sounds from different directions; however, conventional methods have three drawbacks: (a) Sound source localization and sound source separation methods using spatial features and classification using spectral features trained in the same neural network, may overfit to the relationship between the direction of arrival and the class of a sound, thereby reducing their reliability to deal with novel events. (b) Although permutation invariant training used in autonomous speech recognition could be extended, it is impractical for environmental sounds that include an unlimited number of sound sources. (c) Various features, such as complex values of short time Fourier transform and interchannel phase differences have been used as spatial features, but no study has compared them. This paper proposes a multichannel environmental sound segmentation method comprising two discrete blocks, a sound source localization and separation block and a sound source separation and classification block. By separating the blocks, overfitting to the relationship between the direction of arrival and the class is avoided. Simulation experiments using created datasets including 75-class environmental sounds showed the root mean squared error of the proposed method was lower than that of conventional methods.

Compressive Sensing-Based Sound Source Localization for Microphone Arrays

Sound source localization with less data is a challenging task. To address this problem, a novel sound source localization method based on compressive sensing theory is proposed in this paper. Specifically, a sparsity basis is first constructed for each microphone by shifting the audio signal recorded from one reference microphone. In this manner, the microphones except the reference one are allowed to capture audio signals under the sampling rate far below the Nyquist criterion. Next, the source positions are estimated by solving an $$l_1$$ minimization based on each frame of audio signals. Finally, a fine localization scheme is presented by fusing the estimated source positions from multiple frames. The proposed method can directly determine the number of sound sources in one step and successfully estimate the source positions in noisy and reverberant environments. Experimental results demonstrate the validity of the proposed method.

Robust Auditory Functions Based on Probabilistic Integration of MUSIC and CGMM

Sound source localization and separation are essential functions for robot audition to comprehend acoustic environments. The widely-used multiple signal classification (MUSIC) can precisely estimate the directions of arrival (DoAs) of multiple sound sources if its hyperparameters are selected appropriately depending on the surrounding environment. A popular separation method based on a complex Gaussian mixture model (CGMM), on the other hand, can extract multiple sources even in noisy environments if its latent variables are properly initialized to avoid bad local optima. To overcome the drawbacks of both the MUSIC and CGMM, we propose a robot audition framework that complementarily combines the MUSIC and CGMM in a probabilistic manner. Our method is based on a variant of the CGMM conditioned by the localization results of MUSIC. The hyperparameters of MUSIC are estimated by the type II maximum likelihood estimation of the CGMM, and the CGMM itself is efficiently initialized and regularized by using the localization results of MUSIC. Experimental results show that our method outperformed conventional localization and separation methods even when the number of sound sources is unknown. we also demonstrate that our method can work even with moving sound sources in real time.

Neural Network Adaptation and Data Augmentation for Multi-Speaker Direction-of-Arrival Estimation

Deep neural networks have been successfully applied to sound direction-of-arrival estimation under challenging conditions. However, such a learning-based approach requires a large amount of labeled training data, which is difficult to acquire. To address this problem, we propose a novel approach for multi-speaker direction-of-arrival estimation with data augmentation and weakly-supervised domain adaptation. We generate source domain data with simulation, and collect real data annotated with the number of sound sources as the weak labels. The real data are further augmented by mixing single-source segments. Then, weakly-supervised domain adaptation is applied to models pre-trained on the simulated data. We define a loss function for the adaptation process which exploits the weak labels and the mixture component information in the augmented data. Experiments with real robot audio data show that our proposed approach achieves similar performance as if the fully-labeled real data are used. This paper suggests an effective development procedure for DOA estimation models applied to new types of microphone arrays with minimal data collection efforts.

SPEAKER DISCRIMINATION IN MULTISOURCE ENVIRONMENTS AURALIZED IN REAL ROOMS

With the recent development of audio in modern VR/AR systems and the increasing capability of synthesizing natural sound fields over headphones with head tracking, the question of the ability of our hearing system to discriminate multiple concurrent sound sources has become important again. We must understand how psychoacoustical and psychophysical limitations of the hearing system cope with novel technologies of virtual acoustics that can simulate an almost unlimited number of sound sources. Previous research has shown that the capacity of human hearing to discriminate a reference sound source is limited when there is background noise, a reverberant surrounding, or when other, disturbing sound sources simultaneously mask the reference source. A set of listening tests based on the cocktail-party effect was designed to determine the intelligibility of speech emitted by a reference sound source, with one to six disturbing sound sources simultaneously emitting speech from different directions around the listener. The tests were repeated in three test rooms with different acoustical properties, and two test signals were used: logatomes and regular spoken sentences with specific keywords. The results have revealed the changes in speech intelligibility scores in relation to the number of disturbing sources, their positions, and acoustical properties of test rooms.

Forming clusters of multiple sounds improves talker identification in an auditory scene

In several recent papers we have shown that listeners perceive differences in the size of a small auditory scene (< 4 sources) for short-duration sounds (e.g., consonant-vowel, CV, pairs) presented at about the same time, but such differences are not perceived for larger scenes. Spatial separation of a small number of sound sources affects performance much more than spatially separating a larger number of sound sources. In this presentation, the number of sound sources is small, but not the number of sounds (CVs). For example, listeners are poor at determining if a target talker is the same as a cue talker, when the target is presented at one sound-source location and several distractor talkers are presented each at different sound-source locations. If all distractor talkers are “clustered (mixed)” at a single source spatially separated from the target sound source, then listeners are better at determining if the target talker was the same as the cue. Several additional scenarios were tested, all indicating that “clustering” sounds into a small number of sound sources improves target-talker identification compared to when sounds are presented from a larger number of sources.

Sound source localizations using various microphone arrays

Estimating directions of arrival (DoAs) of multiple sound events still posts challenges in acoustic source localization. This paper reviews recent development on model-based direction-of-arrival analysis using three different microphone array configurations, in situations involving potentially multiple concurrent sound sources. The array configurations include a sparse arrangement consisting of two microphones, coprime linear microphone arrays, and spherical microphone arrays. Formulation and evaluation of prediction models are of central importance in coping with these challenges. These three array configurations are all used to solve a two-level inferential problem of sound source enumeration and direction of arrival estimation. The prediction models based on specific array configurations are evaluated upon experimental data derived from pertinent array processing in order to select the simplest such model that can adequately describe the experimental data, thereby enumerating first the concise number of sound sources, then their DoA information. This paper reviews the prediction models recently established for the three microphone arrays, the two levels of inference, and analysis results. This paper will highlight the importance of the prediction models for the model-based analysis of complex sound environments with potentially multiple concurrent sources.

A novel Directional Framework for Source Counting and Source Separation in Instantaneous Underdetermined Audio Mixtures

The audio source separation problem is a well-known problem that was addressed using a variety of techniques. A common setback in these techniques is that the total number of sound sources in the audio mixture must be known beforehand. However, this knowledge is not always available and thus needs to be estimated. Many approaches have attempted to estimate the number of sources in an audio mixture. There are several clustering techniques that can count the sources in an audio mixture, nonetheless, there are cases, where the directionality of the audio data in the mixture may lead these techniques to failure. In this article, we propose a generalised Directional Fuzzy C-Means (DFCM) framework that offers a complete multi-dimensional, directional solution to this problem. Our proposal shows remarkably high performance in estimating the correct number of sources in the majority of the cases and in addition, it can be used as an effective mechanism to separate the sources. The complete source counting-separation framework can act as a robust low-complexity simultaneous solution to both problems.

Parallel Stacked Hourglass Network for Music Source Separation

Music source separation is one of the old and challenging problems in music information retrieval society. Improvements in deep learning lead to big progress in decomposing music into its constitutive components with a variety of music. This research uses three types of datasets for source separation namely; Korean traditional music Pansori dataset, MIR-1K dataset, and DSD100 dataset. DSD100 dataset includes multiple sound sources and other two datasets has relatively smaller number of sound sources. We synthetically constructed a novel dataset for Pansori music and trained a novel parallel stacked hourglass network (PSHN) with multiple band spectrograms. In comparison with past study, proposed architecture performs the best results in real-world test samples of Pansori music of any length. The network performance was also tested for the public DSD100 and MIR-1K dataset for strength comparison in multiple source data and found comparable quantitative and qualitative outcomes. System performance is evaluated using median value of signal-to-distortion ratio (SDR), source-to-interference ratio (SIR), and source-to-artifacts ratio (SAR) measured in decibels (dB) and visual comparison of prediction results with ground truth. We report better performance in the Pansori dataset and MIR-1K dataset and perform detailed ablation studies based on architecture variation. The proposed system is better applicable for separating the music source with voices and single or fewer musical instruments.

Model-based Bayesian direction of arrival analysis for sound sources using a spherical microphone array.

In many room acoustics and noise control applications, it is often challenging to determine the directions of arrival (DoAs) of incoming sound sources. This work seeks to solve this problem reliably by beamforming, or spatially filtering, incoming sound data with a spherical microphone array via a probabilistic method. When estimating the DoA, the signal under consideration may contain one or multiple concurrent sound sources originating from different directions. This leads to a two-tiered challenge of first identifying the correct number of sources, followed by determining the directional information of each source. To this end, a probabilistic method of model-based Bayesian analysis is leveraged. This entails generating analytic models of the experimental data, individually defined by a specific number of sound sources and their locations in physical space, and evaluating each model to fit the measured data. Through this process, the number of sources is first estimated, and then the DoA information of those sources is extracted from the model that is the most concise to fit the experimental data. This paper will present the analytic models, the Bayesian formulation, and preliminary results to demonstrate the potential usefulness of this model-based Bayesian analysis for complex noise environments with potentially multiple concurrent sources.

Towards Size of Scene in Auditory Scene Analysis: A Systematic Review.

Auditory scene analysis is defined as a listener’s ability to segregate a meaningful message from meaningless background noise in a listening environment. To gain better understanding of auditory perception in terms of message integration and segregation ability among concurrent signals, we aimed to systematically review the size of auditory scenes among individuals. A total of seven electronic databases were searched from 2000 to the present with related key terms. Using our inclusion criteria, 4,507 articles were classified according to four sequential steps–identification, screening, eligibility, included. Following study selection, the quality of four included articles was evaluated using the CAMARADES checklist. In general, studies concluded that the size of auditory scene increased as the number of sound sources increased; however, when the number of sources was five or higher, the listener’s auditory scene analysis reached its maximum capability. Unfortunately, the score of study quality was not determined to be very high, and the number of articles used to calculate mean effect size and statistical significance was insufficient to draw significant conclusions. We suggest that study design and materials that consider realistic listening environments should be used in further studies to deep understand the nature of auditory scene analysis within various groups.

Bayesian Inference for Acoustic Direction of Arrival Analysis Using Spherical Harmonics.

This work applies two levels of inference within a Bayesian framework to accomplish estimation of the directions of arrivals (DoAs) of sound sources. The sensing modality is a spherical microphone array based on spherical harmonics beamforming. When estimating the DoA, the acoustic signals may potentially contain one or multiple simultaneous sources. Using two levels of Bayesian inference, this work begins by estimating the correct number of sources via the higher level of inference, Bayesian model selection. It is followed by estimating the directional information of each source via the lower level of inference, Bayesian parameter estimation. This work formulates signal models using spherical harmonic beamforming that encodes the prior information on the sensor arrays in the form of analytical models with an unknown number of sound sources, and their locations. Available information on differences between the model and the sound signals as well as prior information on directions of arrivals are incorporated based on the principle of the maximum entropy. Two and three simultaneous sound sources have been experimentally tested without prior information on the number of sources. Bayesian inference provides unambiguous estimation on correct numbers of sources followed by the DoA estimations for each individual sound sources. This paper presents the Bayesian formulation, and analysis results to demonstrate the potential usefulness of the model-based Bayesian inference for complex acoustic environments with potentially multiple simultaneous sources.

An impact study of acoustic environment on users in large interior spaces

As many large buildings have been built worldwide recently, it is necessary to study how the acoustic environment in those buildings affects people in order to improve the acoustical comfort in them. The aim of this study is to explore the influence of acoustic environment on people in eight large-scale spaces, which are divided into three categories according to function, through grounded theory, and questionnaire. The results showed that “loud background noise,” “large number of sound sources,” “emotional change,” “mixed sounds,” and “sensible sound with certain spectrum component” were people’s main evaluation to the acoustic environment in large-scale spaces. Based on respondents’ perception, the influence of the acoustic environment in large-scale spaces could be classified into the following three aspects: emotional effect, influence on attention, and influence on thinking ability and behavior. Although the evaluation of the acoustic environment varied widely with the difference in spatial functions, same perception dimensions could be summarized.

Discrimination of changes in spatial configuration for multiple, simultaneously presented sounds.

Listeners discriminated changes in the spatial configuration of two-to-eight consonant-vowel (CV) stimuli spoken by different talkers, all simultaneously presented from different loudspeakers in various azimuthal spatial configurations. The number of CVs, spatial configuration of the sound sources, and similarity of the talkers speaking the CVs were varied. Experiment I used a same-different procedure to determine the discriminability of different spatial configurations of multiple sound sources. In experiment II, listeners determined the direction (clockwise or counterclockwise) of sound source rotation over eight rotational steps. In both experiments, performance declined as the number of sound sources increased beyond two.

Time reversal acoustics applied to rooms of various reverberation times.

Time Reversal (TR) is a technique that may be used to focus an acoustic signal at a particular point in space. While many variables contribute to the quality of TR focusing of sound in a particular room, the most important have been shown to be the number of sound sources, signal bandwidth, and absorption properties of the medium as noted by Ribay, de Rosny, and Fink [J. Acoust. Soc. Am. 117(5), 2866-2872 (2005)]. However, the effect of room size on TR focusing has not been explored. Using the image source method algorithm proposed by Allen and Berkley [J. Acoust. Soc. Am. 65(4), 943-950 (1979)], TR focusing was simulated in a variety of rooms with different absorption and volume properties. Experiments are also conducted in a couple rooms to verify the simulations. The peak focal amplitude, the temporal focus quality, and the spatial focus clarity are defined and calculated for each simulation. The results are used to determine the effects of absorption and room volume on TR. Less absorption increases the amplitude of the focusing and spatial clarity while decreasing temporal quality. Dissimilarly, larger volumes decrease focal amplitude and spatial clarity while increasing temporal quality.

Model-based direction of arrival estimations for sound sources using a spherical microphone array

In many room acoustics and noise control applications, it is often challenging to identify the directions of arrivals (DoAs) of incoming sound sources. This work seeks to solve this problem reliably by beamforming, or spatially filtering, incoming sound data with a spherical microphone array via a probabilistic method. When estimating the DoA, the signal under consideration may contain one or multiple concurrent sound sources originating from different directions. This leads to a two-tiered challenge of first identifying the correct number of sources, followed by determining the directional information of each source. To this end, a probabilistic method of model-based Bayesian analysis is leveraged. This entails generating analytic models of the experimental data, individually defined by a specific number of sound sources and their locations in physical space, and evaluating each model to fit the measured data. Through this process, the number of sources is first estimated, and then the DoA information of those sources is extracted from the model being the most concise to fit the experimental data. This paper will present the analytic models, the Bayesian formulation, and preliminary results to demonstrate the potential usefulness of this model-based Bayesian analysis for complex noise environments with potentially multiple concurrent sources.

Experimental acoustic modal analysis of an automotive cabin: challenges and solutions

In this paper, a full Acoustic Modal Analysis (AMA) procedure to improve the CAE predictions of the car interior noise level is proposed. Some of the challenges that can be experienced during such an analysis are described and new solutions to face them are proposed. Particular AMA challenges range from the arrangement of the experimental setup to the post-processing analysis. Since a large number of microphones are needed, a smart localization procedure, which automatically determines the microphone three dimensional (3-D) positions and dramatically reduces the setup time, is presented herein. Furthermore, the need for a large number of sound sources spread across the cavity to assure a homogeneous sound field makes modal parameter estimation a nontrivial task. Traditional modal parameter estimators have indeed proven not to be effective in cases where many input excitation locations have to be used. Hence, a more suitable estimator, the Maximum Likelihood Modal Model-based (ML-MM) method, will be employed for such an analysis.

Parametric directional coding for precomputed sound propagation

Convincing audio for games and virtual reality requires modeling directional propagation effects. The initial sound's arrival direction is particularly salient and derives from multiply-diffracted paths in complex scenes. When source and listener straddle occluders, the initial sound and multiply-scattered reverberation stream through gaps and portals, helping the listener navigate. Geometry near the source and/or listener reveals its presence through anisotropic reflections. We propose the first precomputed wave technique to capture such directional effects in general scenes comprising millions of polygons. These effects are formally represented with the 9D directional response function of 3D source and listener location, time, and direction at the listener, making memory use the major concern. We propose a novel parametric encoder that compresses this function within a budget of ~100MB for large scenes, while capturing many salient acoustic effects indoors and outdoors. The encoder is complemented with a lightweight signal processing algorithm whose filtering cost is largely insensitive to the number of sound sources, resulting in an immediately practical system.

Understanding the effect of room parameters on acoustic time reversal

Time Reversal (TR) is a technique used to focus an acoustic signal at a particular point in space. Of the many variables that contribute to the quality of TR focusing, the most important are the number of sound sources, signal bandwidth. and properties of the medium. Although much research has been done to quantify the effects that the number of sound sources and signal bandwidth have on the TR process, little has been done in regard to the effect that room parameters have on TR. This is largely due to the difficulty involved with changing room acoustic parameters to measure their effects. We use the image source method (using the algorithm proposed by Allen and Berkley) to simulate the TR process in a variety of rooms with different acoustic and geometric room parameters. We define and calculate the maximum focal amplitude, the temporal focus quality and the spatial focus quality for each simulation. We compare the results and determine the effects of absorption and room volume on TR. We find that less absorption improves max response and spatial quality while it decreases temporal quality and that larger volumes have decreased max response and spatial quality while having increased temporal quality.Time Reversal (TR) is a technique used to focus an acoustic signal at a particular point in space. Of the many variables that contribute to the quality of TR focusing, the most important are the number of sound sources, signal bandwidth. and properties of the medium. Although much research has been done to quantify the effects that the number of sound sources and signal bandwidth have on the TR process, little has been done in regard to the effect that room parameters have on TR. This is largely due to the difficulty involved with changing room acoustic parameters to measure their effects. We use the image source method (using the algorithm proposed by Allen and Berkley) to simulate the TR process in a variety of rooms with different acoustic and geometric room parameters. We define and calculate the maximum focal amplitude, the temporal focus quality and the spatial focus quality for each simulation. We compare the results and determine the effects of absorption and room volume on TR. We find that less a...