Multi-talker Speech Separation Research Articles

ONLINE BINAURAL SPEECH SEPARATION OF MOVING SPEAKERS WITH A WAVESPLIT NETWORK.

Binaural speech separation in real-world scenarios often involves moving speakers. Most current speech separation methods use utterance-level permutation invariant training (u-PIT) for training. In inference time, however, the order of outputs can be inconsistent over time particularly in long-form speech separation. This situation which is referred to as the speaker swap problem is even more problematic when speakers constantly move in space and therefore poses a challenge for consistent placement of speakers in output channels. Here, we describe a real-time binaural speech separation model based on a Wavesplit network to mitigate the speaker swap problem for moving speaker separation. Our model computes a speaker embedding for each speaker at each time frame from the mixed audio, aggregates embeddings using online clustering, and uses cluster centroids as speaker profiles to track each speaker throughout the long duration. Experimental results on reverberant, long-form moving multitalker speech separation show that the proposed method is less prone to speaker swap and achieves comparable performance with u-PIT based models with ground truth tracking in both separation accuracy and preserving the interaural cues.

Implementation of Real-Time Speech Separation Model Using Time-Domain Audio Separation Network (TasNet) and Dual-Path Recurrent Neural Network (DPRNN)

Abstract The purpose of this research is to develop a model that is able to perform real-time speaker independent multi-talker speech separation task in time-domain using Time-Domain Audio Separation Network (TasNet) and Dual-Path Recurrent Neural Network (DPRNN). This research will conduct experiments on some RNN architectures, number of batch size, and optimizers as hyper-parameters in order to implement TasNet and DPRNN. This research also try to analyze the impact of these hyperparameters setup on model performance. The expected result of this research is a more accurate model and lower latency to complete speaker independent multi-talker speech separation task in real-time than previous research model.

A Two-Stage Phase-Aware Approach for Monaural Multi-Talker Speech Separation

A Two-Stage Phase-Aware Approach for Monaural Multi-Talker Speech Separation

Deep neural networks based binary classification for single channel speaker independent multi-talker speech separation

Speech separation is an important task of separating a target speech from the mixture signals. Speaker-independent multi-talker speech separation is a challenging task due to unpredictability of the target and interfering speech in the target-interference mixtures. Conventionally, speech separation is used as a signal processing problem, but recently it is formulated as a deep learning problem and discriminative patterns of the speech are learned from the training data. In this paper, we consider the ideal binary mask (IBM) as a supervised binary classification training-target by using fully connected deep neural networks (DNN) for single-channel speaker-independent multi-talker speech separation. The train DNNs is used to estimate IBM training-target. The mean square error (MSE) is used as an objective cost function. Standard backpropagation and Monte-Carlo dropout regularization approaches are used for better generalization and overfitting during training. The estimated training-target is applied to the mixtures to obtain the separated target speech. We have addressed the over-smoothing problem and performed equalization of spectral variances to match the estimated and clean speech features. Our experimental results in various evaluating conditions report that the proposed method outperformed the competing methods in terms of the Perceptual Evaluation of Speech Quality (PESQ), Segmental SNR (SNRSeg), Short-time objective intelligibility (STOI), normalized Frequency weighted SNRSeg (nFwSNRSeg) and HIT-FA rates.

Sparsity-based phase spectrum compensation for single-channel speech source separation

This paper proposes a sparsity-based phase spectrum compensation (SPSC) function to improve the quality of reconstructed signals by using magnitude-based single-channel speech source separation. While conventional approaches to the reconstruction of separated sources use the input phase spectrum, the proposed SPSC function modifies each source's phase spectrum using the estimated magnitude spectra of multiple sources and their spectro-temporal sparsity. In particular, the spectro-temporal sparsity is estimated from the signal-to-interference ratio between the magnitude spectrum of a source to be separated and those of the other sources, including the background noise. The proposed SPSC function is first embedded into the reconstruction stage of magnitude-based speech source separation methods that employ deep recurrent neural network (DRNN) and sparse nonnegative matrix factorization (SNMF), respectively. Then, speech denoising is performed under four different noise conditions with signal-to-noise ratios (SNRs) in the range of 0–15 dB. Both objective and subjective tests show that both DRNN and SNMF-based speech separation methods employing the proposed SPSC function substantially outperform those employing the conventional PSC function for the speech denoising task. Moreover, the DRNN combined with the proposed SPSC function offers higher average perceptual evaluation of speech quality (PESQ) scores for speech denoising than SEGAN, which is based on a very large end-to-end neural networks model. Next, the proposed SPSC method is also applied on the reconstruction stage of a deep clustering-based speech separation method to examine its contribution on a multi-talker speech separation task, where spoken utterances of males and females are mixed at −3, 0, and 3 dB SNR. It is shown from the experiment that the proposed SPSC function further improves SDR and PESQ scores of the magnitude spectra processed by the deep clustering-based single-channel speech separation method.

Multitalker Speech Separation With Utterance-Level Permutation Invariant Training of Deep Recurrent Neural Networks

In this paper, we propose the utterance-level permutation invariant training (uPIT) technique. uPIT is a practically applicable, end-to-end, deep-learning-based solution for speaker independent multitalker speech separation. Specifically, uPIT extends the recently proposed permutation invariant training (PIT) technique with an utterance-level cost function, hence eliminating the need for solving an additional permutation problem during inference, which is otherwise required by frame-level PIT. We achieve this using recurrent neural networks (RNNs) that, during training, minimize the utterance-level separation error, hence forcing separated frames belonging to the same speaker to be aligned to the same output stream. In practice, this allows RNNs, trained with uPIT, to separate multitalker mixed speech without any prior knowledge of signal duration, number of speakers, speaker identity, or gender. We evaluated uPIT on the WSJ0 and Danish two- and three-talker mixed-speech separation tasks and found that uPIT outperforms techniques based on nonnegative matrix factorization and computational auditory scene analysis, and compares favorably with deep clustering, and the deep attractor network. Furthermore, we found that models trained with uPIT generalize well to unseen speakers and languages. Finally, we found that a single model, trained with uPIT, can handle both two-speaker, and three-speaker speech mixtures.

A microphone array beamforming-based system for multi-talker speech separation

This paper presents a Multichannel Speech Separation System (MCSS) based on new beamforming frequency domain method. The beamformer exploits the spatial properties of the source signals using a microphone array. Therefore, it is based on a prior knowledge of the position of the speakers relative to the array. The proposed beamformer is defined with two processing steps: the first one is to keep a unit gain of the desired signal and the other blocks the wanted signal and minimises the output power of the interferences within only one step. In order to separate multiple speakers, multiple beamformers are used simultaneously, where a beamformer is computed for each source considering the remaining sources as interferers. We test and evaluate the proposed MCSS on real recording mixtures extracted from 'Multichannel In-Car Speech Database'. The experimental results proved the effectiveness of the proposed system in terms of speech separation. The quality of speech will be improved compared to the state-of-the-art.

Supervised Single-Microphone Multi-Talker Speech Separation with Conditional Random Fields

We apply conditional random field (CRF) for single-microphone speech separation in a supervised learning scenario. We train the parameters with mixture data in which the sources are competing with the same average signal power. Compared with factorial hidden Markov model (HMM) baselines, the CRF settings require fewer training mixture data to improve objective speech quality measures and speech recognition accuracy of the reconstructed sources, when mixing ratios of training and testing mixture data are matched. The CRF settings also handle minor mixing ratio mismatch after adjusting the gain factors of the sources with non-linear mappings inspired from the mixture-maximization model. When the mixing ratio mismatch further increases such that the speech mixture is dominated by only one source, factorial HMM finally catches up with and performs better than the CRF settings due to improved model accuracy. We also develop a convex statistical inference simplification based on linear-chain CRFs. The simplification achieves the same performance level as the original CRF settings after integrating additional observations.

Single-Channel Multitalker Speech Recognition

We have described some of the problems with modeling mixed acoustic signals in the log spectral domain using graphical models, as well as some current approaches to handling these problems for multitalker speech separation and recognition. We have also reviewed methods for inference on FHMMs (factorial hidden Markov model) and methods for handling the nonlinear interaction function in the log spectral domain. These methods are capable of separating and recognizing speech better than human listeners on the SSC task.