Sound Source Separation Research Articles

Sound source separation using image signal processing based on sparsity of sound field

Microphone arrays have been used to separate sound sources to improve speech recognition in a noisy environment. We propose a method using image signal processing to achieve highly accurate sound source separation. The microphone array is 1-m long and consists of eight microphones. Temporal sequences of the sound pressures obtained from the eight microphones are converted into sequences of luminance. These are arranged in parallel in an image, which is referred to as aspatio-temporal sound pressure distribution image. Sparse modeling using L1 regularization (Lasso) is applied to the image for restoring a high-resolution image. The spatial spectrum of the restored image is obtained using a two-dimensional fast Fourier transform (FFT) algorithm. In this spectrum, the angle of a line through the center denotes the arrival direction of sound, and the distance from the center indicates its frequency. By extracting a line from the spectrum, the sound source can be separated. A computational experiment revealed that high-resolution sound field obtained by a 512-microphone array could be restored using the proposed method. Moreover, SNR was improved by 32.1 dB when two sounds arrived 45° apart, indicating sufficient performance to extract a desired sound. [Work supported by JSPS KAKENHI Grant (16K06384).]

Fast multiple moving sound sources localization utilizing sparseness of speech signals

We propose multiple moving sound sources localization with reduced operation time utilizing the sparseness of speech signals, and we demonstrate the efficacy of proposed method in actual environment experiments. Sound source localization is an essential function for tasks such as robot audition, and particularly for sound sources that often move in the actual environments. Sound source localization requires high resolution and real-time processing because they are utilized for post-processes such as sound source separation. The proposed method is based on MUSIC (MUltiple SIgnal Classification) which is known as a sound source localization method with high resolution. However, MUSIC has a large computational complexity. When the signal has large energy over a wideband in the frequency domain, the operation time increases because MUSIC estimates the direction of the sound source at every frequency. Therefore, we attempted to reduce the operation time by using the sparseness of speech signals in which the sound energies exist sparsely in the time-frequency domain. We chose frequencies with large power from frequency characteristics of the observed signals. Sound source localization was performed in the chosen frequencies. The operation time of the proposed method was about 8.1 times faster than the case of utilizing all frequencies.

Separation and integration of sound sources in auditory processing

Some perceptual experiments were conducted to elucidate sound source separation processing in speech perception. The phenomenon of phonemic restoration was tested if it occurs in dichotic listening. An original speech is Japanese /VCV/, the stimulus is made by manipulating the /VCV/ as was replaced the consonant part with noise. The duration of replacement was varied at six levels: 10, 20, 40, 60, 80, and 100 ms. There were two hearing modes: in one case, the monaural stimulus was presented to both the ear, and in the other case, the speech part of the stimulus was presented to an ear and the noise part to the opposite ear simultaneously. Thirty subjects in the age of twenties participate the listening test. The results show that phonemic restoration effect is clearly observed in both hearing modes. The rate of restoration is 92%, and 95% even in dichotic listening. There was no feature of speech in the noise has no relation with speech sound. Moreover, the direction of the noise sound was opposite to that of speech sound source, different in 180 degree. Even though the two sound signal are somehow integrated to hear restored speech.

Acoustic lens with exponential horns for selective distant recording

Many signal processing techniques for sound source separation using microphone array have been studied as a robust audio recording method to ambient noise. However, the implementation in real-time is still hard to be realized in terms of its large computational cost. As an another approach, a directional microphone has a capability to record sound from a specific direction; however, it is impossible to pick up sound at a specific distance. As a part of research on acoustic transducer, on the other hand, a method to broaden the directivity by using the slant-plate acoustical lenses at the front of loudspeaker has been utilized. In addition, there is a report that a slant-plate acoustical lens combined with a conical horn shows a sharper directivity than that by a parabolic microphone. The authors have studied selective distant recording using acoustic lens that synchronously adds the wave fronts after removing the time difference of arrival via the speaking tubes having different lengths. As a result of that the shape of the speaking tube in the acoustic lens was replaced with exponential horn, an improvement in terms of sound quality with a reduction of resonance was obtained.

Blind separation of incoherent and spatially disjoint sound sources

Blind separation of sound sources aims at reconstructing the individual sources which contribute to the overall radiation of an acoustical field. The challenge is to reach this goal using distant measurements when all sources are operating concurrently. The working assumption is usually that the sources of interest are incoherent – i.e. statistically orthogonal – so that their separation can be approached by decorrelating a set of simultaneous measurements, which amounts to diagonalizing the cross-spectral matrix. Principal Component Analysis (PCA) is traditionally used to this end. This paper reports two new findings in this context. First, a sufficient condition is established under which “virtual” sources returned by PCA coincide with true sources; it stipulates that the sources of interest should be not only incoherent but also spatially orthogonal. A particular case of this instance is met by spatially disjoint sources – i.e. with non-overlapping support sets. Second, based on this finding, a criterion that enforces both statistical and spatial orthogonality is proposed to blindly separate incoherent sound sources which radiate from disjoint domains. This criterion can be easily incorporated into acoustic imaging algorithms such as beamforming or acoustical holography to identify sound sources of different origins. The proposed methodology is validated on laboratory experiments. In particular, the separation of aeroacoustic sources is demonstrated in a wind tunnel.

Role of Binaural Temporal Fine Structure and Envelope Cues in Cocktail-Party Listening.

While conversing in a crowded social setting, a listener is often required to follow a target speech signal amid multiple competing speech signals (the so-called "cocktail party" problem). In such situations, separation of the target speech signal in azimuth from the interfering masker signals can lead to an improvement in target intelligibility, an effect known as spatial release from masking (SRM). This study assessed the contributions of two stimulus properties that vary with separation of sound sources, binaural envelope (ENV) and temporal fine structure (TFS), to SRM in normal-hearing (NH) human listeners. Target speech was presented from the front and speech maskers were either colocated with or symmetrically separated from the target in azimuth. The target and maskers were presented either as natural speech or as "noise-vocoded" speech in which the intelligibility was conveyed only by the speech ENVs from several frequency bands; the speech TFS within each band was replaced with noise carriers. The experiments were designed to preserve the spatial cues in the speech ENVs while retaining/eliminating them from the TFS. This was achieved by using the same/different noise carriers in the two ears. A phenomenological auditory-nerve model was used to verify that the interaural correlations in TFS differed across conditions, whereas the ENVs retained a high degree of correlation, as intended. Overall, the results from this study revealed that binaural TFS cues, especially for frequency regions below 1500 Hz, are critical for achieving SRM in NH listeners. Potential implications for studying SRM in hearing-impaired listeners are discussed. Acoustic signals received by the auditory system pass first through an array of physiologically based band-pass filters. Conceptually, at the output of each filter, there are two principal forms of temporal information: slowly varying fluctuations in the envelope (ENV) and rapidly varying fluctuations in the temporal fine structure (TFS). The importance of these two types of information in everyday listening (e.g., conversing in a noisy social situation; the "cocktail-party" problem) has not been established. This study assessed the contributions of binaural ENV and TFS cues for understanding speech in multiple-talker situations. Results suggest that, whereas the ENV cues are important for speech intelligibility, binaural TFS cues are critical for perceptually segregating the different talkers and thus for solving the cocktail party problem.

Detection of binaural interaction in free-field evoked auditory brainstem responses by time-scale representations.

The so called β-wave of the binaural interaction component (BIC) in auditory brainstem responses (ABR) has been shown to be an objective measure for binaural interaction (BI). This component is the arithmetical difference between the sum of the monaurally evoked ABRs and the binaurally evoked ABR. Unfortunately, these neural responses are known to be very fragile and as a result the calculated BIC. An additional issue is, that the findings of this measurement are predominantly needed in people with hearing loss who may use hearing devices like hearing aids (HA) or cochlear implants (CI), thus they are not able to use headphones (like in conventional ABR measurements) during the detection of possible BI. This is a crucial problem, because it is known that factors like the interaural time delay (ITD) between the receiving ears are responsible for solving tasks like sound source localization or sound source separation, but specially designed measurements to coordinate the fitting of HAs or CIs with respect to BI are still missing. In this paper, we introduce a new measurement setup that is able to detect BI depending on different ITDs in free-field evoked responses by using the more reliable instantaneous phase in the time-scale representation. With this pilot study we are able to demonstrate a decreasing BI with an increasing ITD using the wavelet phase synchronization stability analysis in ten normal hearing subjects.

The importance of phase, attention, intelligibility, and recall in sound system design

Measures for the accuracy of speech transmission over an audio channel, such as Speech Transmission Index, rely on random word recognition as the standard for quality. In this paper, we propose that the ability to recall information at a later time is a more useful standard for sound quality than simply recognizing words. Recall, particularly in a complex or noisy environment, depends critically on the focused attention of a listener. There is evidence that “proximity,” the auditory perception that a source is close to a listener, enhances both focused attention and the ability to localize and separate sound sources from competing signals. We find that proximity and the benefits it brings depend on the phase alignment of frequencies above 1000 Hz. This alignment is lost when there are excess reflections, reverberation, noise, or multiple loudspeakers. We will present measurement techniques that can quantify proximity from an impulse response, and sound designs that maximize it in practice.

Spatial Stream Segregation by Cats.

Listeners can perceive interleaved sequences of sounds from two or more sources as segregated streams. In humans, physical separation of sound sources is a major factor enabling such stream segregation. Here, we examine spatial stream segregation with a psychophysical measure in domestic cats. Cats depressed a pedal to initiate a target sequence of brief sound bursts in a particular rhythm and then released the pedal when the rhythm changed. The target bursts were interleaved with a competing sequence of bursts that could differ in source location but otherwise were identical to the target bursts. This task was possible only when the sources were heard as segregated streams. When the sound bursts had broad spectra, cats could detect the rhythm change when target and competing sources were separated by as little as 9.4°. Essentially equal levels of performance were observed when frequencies were restricted to a high, 4-to-25-kHz, band in which the principal spatial cues presumably were related to sound levels. When the stimulus band was restricted from 0.4 to 1.6 kHz, leaving interaural time differences as the principal spatial cue, performance was severely degraded. The frequency sensitivity of cats in this task contrasts with that of humans, who show better spatial stream segregation with low- than with high-frequency sounds. Possible explanations for the species difference includes the smaller interaural delays available to cats due to smaller sizes of their heads and the potentially greater sound-level cues available due to the cat’s frontally directed pinnae and higher audible frequency range.

Design and implementation of a space domain spherical microphone array with application to source localization and separation.

In this paper, four delay-and-sum (DAS) beamformers formulated in the modal domain and the space domain for open and solid spherical apertures are examined through numerical simulations. The resulting beampatterns reveal that the mainlobe of the solid spherical DAS array is only slightly narrower than that of the open array, whereas the sidelobes of the modal domain array are more significant than those of the space domain array due to the discrete approximation of continuous spherical Fourier transformation. To verify the theory experimentally, a three-dimensionally printed spherical array on which 32 micro-electro-mechanical system microphones are mounted is utilized for localization and separation of sound sources. To overcome the basis mismatch problem in signal separation, source localization is first carried out using minimum variance distortionless response beamformer. Next, Tikhonov regularization (TIKR) and compressive sensing (CS) are employed to extract the source signal amplitudes. Simulations and experiments are conducted to validate the proposed spherical array system. Objective perceptual evaluation of speech quality test and a subjective listening test are undertaken in performance evaluation. The experimental results demonstrate better separation quality achieved by the CS approach than by the TIKR approach at the cost of computational complexity.

距離画像センサを利用した音源分離の研究

In order to operate the home electric appliances with speech recognition, we need to extract the objective voice from noise. One means of extracting the objective voice, there is a sound source separation with microphone array. Sound source separation needs a direction of the objective voice. There are many studies being performed in order to estimate a direction of the voice. But there are still problems to estimate the direction. So it’s difficult to go on extracting the objective voice. We focused to on depth image sensor to measure the position of objective human. We try to separate the objective voice from noise with depth image sensor.

Encoding and communicating navigable speech soundfields

This paper describes a system for encoding and communicating navigable speech soundfields for applications such as immersive audio/visual conferencing, audio surveillance of large spaces and free viewpoint television. The system relies on recording speech soundfields using compact co-incident microphone arrays that are then processed to identify sources and their spatial location using the well-known assumption that speech signals are sparse in the time-frequency domain. A low-delay Direction of Arrival (DOA)-based frequency domain sound source separation approach is proposed that requires only 250 ms of speech signal. Joint compression is achieved through a previously proposed perceptual analysis-by-synthesis spatial audio coding scheme that encodes sources into a mixture signal that can be compressed by a standard speech codec at 32 kbps. By also transmitting side information representing the original spatial location of each source, the received mixtures can be decoded and then flexibly reproduced using loudspeakers at a chosen listening point within a synthesised speech scene. The system was implemented based on this framework for an example application encoding a three-talker navigable speech scene at a total bit rate of 48 kbps. Subjective listening tests were conducted to evaluate the quality of the reproduced speech scenes at a new listening point as compared to a true recording at that point. Results demonstrate the approach successfully encodes multiple spatial speech scenes at low bit rates whilst maintaining perceptual quality in both anechoic and reverberant environments.

Synchronization based on mixture alignment for sound source separation in wireless acoustic sensor networks

Desynchronization degrades the performance of many signal processing algorithms in Wireless Acoustic Sensor Networks. It is mainly caused by the different distances between the source and each node and by the clock phase offset and frequency skew. Classical solutions use clock synchronization protocols and algorithms in the communication layer, but these alternatives do not tackle the lack of synchronization caused by the distances between sources and nodes.In this paper, we present a novel study of the synchronization problem in acoustic sensor networks from a signal processing point of view. First, we propose a theoretical framework that allows us to study the effects of misalignment over any short-time based algorithm, focusing on the requirements of the effective length of the analysis time frame. From this framework, a theoretical synchronization delay is established aimed at reducing the required length of the time frame. Second, two novel alignment methods are developed and are tuned up to reduce the amount of synchronization information required for transmission. The results obtained demonstrate that our proposed methods represent a good solution in terms of performance over the quality of a standard Blind Source Separation algorithm, allowing us to reduce the transmission bandwidth required for synchronization data.

Distant speech separation using predicted time–frequency masks from spatial features

Speech separation algorithms are faced with a difficult task of producing high degree of separation without containing unwanted artifacts. The time–frequency (T–F) masking technique applies a real-valued (or binary) mask on top of the signal’s spectrum to filter out unwanted components. The practical difficulty lies in the mask estimation. Often, using efficient masks engineered for separation performance leads to presence of unwanted musical noise artifacts in the separated signal. This lowers the perceptual quality and intelligibility of the output.Microphone arrays have been long studied for processing of distant speech. This work uses a feed-forward neural network for mapping microphone array’s spatial features into a T–F mask. Wiener filter is used as a desired mask for training the neural network using speech examples in simulated setting. The T–F masks predicted by the neural network are combined to obtain an enhanced separation mask that exploits the information regarding interference between all sources. The final mask is applied to the delay-and-sum beamformer (DSB) output.The algorithm’s objective separation capability in conjunction with the separated speech intelligibility is tested with recorded speech from distant talkers in two rooms from two distances. The results show improvement in instrumental measure for intelligibility and frequency-weighted SNR over complex-valued non-negative matrix factorization (CNMF) source separation approach, spatial sound source separation, and conventional beamforming methods such as the DSB and minimum variance distortionless response (MVDR).

Acoustic space learning for sound-source separation and localization on binaural manifolds.

In this paper, we address the problems of modeling the acoustic space generated by a full-spectrum sound source and using the learned model for the localization and separation of multiple sources that simultaneously emit sparse-spectrum sounds. We lay theoretical and methodological grounds in order to introduce the binaural manifold paradigm. We perform an in-depth study of the latent low-dimensional structure of the high-dimensional interaural spectral data, based on a corpus recorded with a human-like audiomotor robot head. A nonlinear dimensionality reduction technique is used to show that these data lie on a two-dimensional (2D) smooth manifold parameterized by the motor states of the listener, or equivalently, the sound-source directions. We propose a probabilistic piecewise affine mapping model (PPAM) specifically designed to deal with high-dimensional data exhibiting an intrinsic piecewise linear structure. We derive a closed-form expectation-maximization (EM) procedure for estimating the model parameters, followed by Bayes inversion for obtaining the full posterior density function of a sound-source direction. We extend this solution to deal with missing data and redundancy in real-world spectrograms, and hence for 2D localization of natural sound sources such as speech. We further generalize the model to the challenging case of multiple sound sources and we propose a variational EM framework. The associated algorithm, referred to as variational EM for source separation and localization (VESSL) yields a Bayesian estimation of the 2D locations and time-frequency masks of all the sources. Comparisons of the proposed approach with several existing methods reveal that the combination of acoustic-space learning with Bayesian inference enables our method to outperform state-of-the-art methods.

Outdoor Acoustic Event Identification using Sound Source Separation and Deep Learning with a Quadrotor-Embedded Microphone Array

Outdoor Acoustic Event Identification using Sound Source Separation and Deep Learning with a Quadrotor-Embedded Microphone Array

2P1-V09 少素子マイクロフォンアレーを有する音源追尾ロボットの複数音源環境下における音源分離特性

2P1-V09 少素子マイクロフォンアレーを有する音源追尾ロボットの複数音源環境下における音源分離特性

Multichannel Sound Source Dereverberation and Separation for Arbitrary Number of Sources Based on Bayesian Nonparametrics

Multichannel signal processing using a microphone array provides fundamental functions for coping with multi-source situations, such as sound source localization and separation, that are needed to extract the auditory information for each source. Auditory uncertainties about the degree of reverberation and the number of sources are known to degrade performance or limit the practical application of microphone array processing. Such uncertainties must therefore be overcome to realize general and robust microphone array processing. These uncertainty issues have been partly addressed-existing methods focus on either source number uncertainty or the reverberation issue, where joint separation and dereverberation has been achieved only for the overdetermined conditions. This paper presents an all-round method that achieves source separation and dereverberation for an arbitrary number of sources including underdetermined conditions. Our method uses Bayesian nonparametrics that realize an infinitely extensible modeling flexibility so as to bypass the model selection in the separation and dereverberation problem, which is caused by the source number uncertainty. Evaluation using a dereverberation and separation task with various numbers of sources including underdetermined conditions demonstrates that (1) our method is applicable to the separation and dereverberation of underdetermined mixtures, and that (2) the source extraction performance is comparable to that of a state-of-the-art method suitable only for overdetermined conditions.

Probing auditory scene analysis.

Probing auditory scene analysis.

Experimental modeling of Wiener filters estimated on an operating diesel engine

Sound source separation in diesel engines can be implemented using a Wiener filter, or spectrofilter, that can extract the combustion contribution in the overall noise. In this study this filter characterizes the transfer function between a cylinder pressure and a measurement point. An engine is characterized by several filters (one for each cylinder) which are estimated for many operating conditions (engine speed and load). The purpose of this work is to obtain an averaged spectrofilter allowing the synthesis of combustion noise in all operating conditions. This synthesis should be accurate enough to be used in perceptive studies. In order to refine the spectrofilter estimation in the medium frequency band, this paper consists in taking advantage of the multitude of information given by the estimations from different operating conditions. To do this, an experimental model is adopted so modal parameters are extracted from a great number of measured filters. Different procedures such as the ESPRIT method or the LSCE method (modal analysis) are used to decompose the impulse responses on a complex exponential basis. The spectrofilters estimated from different operating conditions are analyzed and compared in this reduced basis, in order to identify the underlying structural parameters. These parameters are compared to the results of an experimental characterization of the stopped engine. The accuracy of the synthesis (number of components of the filter) is an important issue because these filters will be used in perceptive applications, extracting combustion noises.This paper is an extended version of the work initially presented at the conference Surveillance 6 in November 2011 in Compiègne, France [1] (J. Drouet, Quentin Leclere, Etienne Parizet. Experimental modeling of Wiener filters estimated on an operating diesel engine, in: Proceedings of the Surveillance, vol. 6, Compi‘egne, France, 2011.).