Speech Source Separation Research Articles

Room impulse response reshaping-based expectation–maximization in an underdetermined reverberant environment

Source separation in an underdetermined reverberation environment is a very challenging issue. The classical method is based on the expectation–maximization algorithm. However, it is limited to high reverberation environments, resulting in bad or even invalid separation performance. To eliminate this restriction, a room impulse response reshaping-based expectation–maximization method is designed to solve the problem of source separation in an underdetermined reverberant environment. Firstly, a room impulse response reshaping technology is designed to eliminate the influence of audible echo on the reverberant environment, improving the quality of the received signals. Then, a new mathematical model of time-frequency mixing signals is established to reduce the approximation error of model transformation caused by high reverberation. Furthermore, an improved expectation–maximization method is proposed for real-time update learning rules of model parameters, and then the sources are separated using the estimators provided by the improved expectation–maximization method. Experimental results based on source separation of speech and music mixtures demonstrate that the proposed algorithm achieves better separation performance while maintaining much better robustness than popular expectation–maximization methods.

An optimized approach to speech transcription using blind source separation and speech-to-text machine learning models

The use of speech-to-text transcription has a multitude of applications in various industries, including accessibility support, language processing, and automatic subtitling. In recent years, there has been greater interest in incorporating automatic speech source separation features to improve the accuracy and efficiency of transcription mechanisms. This paper aims to design a transcription mechanism that utilizes DUET algorithm to separate speech sources in a stereo setup. The separated sources are then transcribed into text using a machine learning model. The study evaluates the effectiveness of this approach using a dataset of speech recordings. The results of the study indicate high accuracy in speech separation and transcription, highlighting the potential of this approach for practical applications. However, the study also revealed potential issues with the mechanism, indicating the need for further exploration and refinement. These findings indicate the potential of the proposed approach for practical applications, and propose insight for further development and researches in this area.

Separation of Multiple Speech Sources in Reverberant Environments Based on Sparse Component Enhancement

Separation of Multiple Speech Sources in Reverberant Environments Based on Sparse Component Enhancement

Speech source separation avoiding initial value dependency by cepstral-basis-decomposed nonnegative matrix factorization

Speech source separation avoiding initial value dependency by cepstral-basis-decomposed nonnegative matrix factorization

Determined Reverberant Blind Source Separation of Audio Mixing Signals

Audio signal separation is an open and challenging issue in the classical “Cocktail Party Problem”. Especially in a reverberation environment, the separation of mixed signals is more difficult separated due to the influence of reverberation and echo. To solve the problem, we propose a determined reverberant blind source separation algorithm. The main innovation of the algorithm focuses on the estimation of the mixing matrix. A new cost function is built to obtain the accurate demixing matrix, which shows the gap between the prediction and the actual data. Then, the update rule of the demixing matrix is derived using Newton gradient descent method. The identity matrix is employed as the initial demixing matrix for avoiding local optima problem. Through the real-time iterative update of the demixing matrix, frequency-domain sources are obtained. Then, time-domain sources can be obtained using an inverse short-time Fourier transform. Experimental results based on a series of source separation of speech and music mixing signals demonstrate that the proposed algorithm achieves better separation performance than the state-of-the-art methods. In particular, it has much better superiority in the highly reverberant environment.

State-of-the-Art Analysis of Deep Learning-Based Monaural Speech Source Separation Techniques

The monaural speech source separation problem is an important application in the signal processing field. But recent interaction of deep learning algorithms with signal processing achieves remarkable performance improvement for speech source separation problems. This paper explores the numerous state-of-the-art deep learning-based monaural speech source separation algorithms in the time-frequency (T-F), time, and hybrid domains. The motivation, algorithm, and framework of different deep learning models for monaural speech source separation are analyzed. The benchmarked algorithms in the T-F domain can be categorized as deep neural networks (DNN), clustering, permutation, multi-task learning, computational auditory sense analysis (CASA), and phase reconstruction-based techniques, whereas the state-of-the-art time-domain approaches can be categorized as CNN, RNN, multi-scale fusion (MSF), and transformer-based techniques. The end-to-end post filter (E2EPF) is a hybrid algorithm combining T-F and time-domain works to achieve enhanced results. Time-domain models have shown improvement in separation performance compared to the T-F and hybrid domain models with small model sizes. Methods in T-F, time, and hybrid domains are compared using <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$SDR$ </tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$SI-SDR$ </tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$SI-SNR$ </tex-math></inline-formula> , PESQ, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$STOI$ </tex-math></inline-formula> as quality assessment metrics on some benchmark datasets.

Violin Teaching Improvement Strategy in the Context of Intelligent Internet of Things

With the progress of science and technology, the construction of intelligent teaching is also developing and making progress. Intelligent teaching is a kind of teaching mode with the help of emerging information technology such as the Internet of things, cloud computing, and big data to build a diversified classroom. The construction of intelligent teaching is one of the important parts in the field of education informatization, and it is also an active field. The research in this area is beneficial to China’s growth of information education. Online education has progressed significantly in the context of the intelligent Internet of Things. The focus of this paper is on violin online education. In the course of the research, it was discovered that violin teachers have little control over their students’ after-school practice results, and that parents of violin students have little time to supervise their children’s practice at any given moment. To solve this problem, the mixed speech composed of speech and violin accompaniment is selected as the research object. Audio signal analysis and neural network algorithm are used for analysis and research. The time domain and frequency domain characteristics of various audio signals are analyzed, including sparsity of speech signals and repeatability of music signals. The logarithmic power spectrum of audio signal is selected as the characteristic parameter, the sample is preprocessed and the feature is extracted, and the sound source separation model based on deep neural network is realized. The PESQ index was used to evaluate the separation results after using the audio source separation model to separate the mixed speech consisted of speech and violin accompaniment. By comparing the PESQ evaluation results of 40, 50 and 60 iterations, the model converges when the number of iterations is 50. The comparison between the model and L-MMSE algorithm shows that the performance of the model is better in speech source separation. The system will serve for the automatic evaluation of violin teachers’ homework and the remote practice supervision of teachers and parents.

Underdetermined blind speech source separation based on deep nearest neighbor clustering algorithm

Underdetermined blind speech source separation based on deep nearest neighbor clustering algorithm

A Unified Approach for Simultaneous Graph Learning and Blind Separation of Graph Signal Sources

Blind separation of the sources supported by graphs, i.e., graph signals, is a nascent and challenging Blind Source Separation (BSS) problem. In these cases, along with the statistical independence of the sources, additional dependency information can be interpreted from their graph structure, which provides exploiting different Graph Signal Processing (GSP) tools to improve the separation performance. To the best of our knowledge, in these cases, only GraDe and GraphJADE methods have been proposed to exploit the graph dependencies and/or GSP techniques to improve the separation quality rather than the conventional BSS methods. Despite the significant advantages of these graph-based methods, assuming that the underlying graphs are known is a serious drawback, especially in many real-world applications. To address this issue, in this paper, we propose a Unified objective function for GraphJADE with Graph Learning, namely U-GraphJADE-GL, and use the Block Coordinate Descent (BCD) to optimize it. In our proposed method, along with the separation task, the underlying graphs are learned simultaneously in an efficient manner. We compare the performance of the U-GraphJADE-GL with that of the GraDe with Graph Learning (U-GraDe-GL) and the conventional BSS methods in the BSS task and also analyze the GL performance. Besides, as well as the theoretical and experimental convergence analysis, we derive/state the Cram&#x00E9;r-Rao bound (CRB) on the estimation of the mixing and unmixing matrices and also on the attainable Interference-to-Source Ratio (ISR) and compare the asymptotic performance of the proposed method with that of the optimal CRB estimators. To investigate the applicability in real applications, the performance of the proposed method is also compared with that of the abovementioned algorithms for denoising the epileptic Electroencephalogram (EEG) signals and also for the audio speech source separation task. The results show the superiority of the proposed algorithm compared to the other methods in the case of BSS of graph signals with unknown graphs.

An Overview of Variational Autoencoders for Source Separation, Finance, and Bio-Signal Applications.

Autoencoders are a self-supervised learning system where, during training, the output is an approximation of the input. Typically, autoencoders have three parts: Encoder (which produces a compressed latent space representation of the input data), the Latent Space (which retains the knowledge in the input data with reduced dimensionality but preserves maximum information) and the Decoder (which reconstructs the input data from the compressed latent space). Autoencoders have found wide applications in dimensionality reduction, object detection, image classification, and image denoising applications. Variational Autoencoders (VAEs) can be regarded as enhanced Autoencoders where a Bayesian approach is used to learn the probability distribution of the input data. VAEs have found wide applications in generating data for speech, images, and text. In this paper, we present a general comprehensive overview of variational autoencoders. We discuss problems with the VAEs and present several variants of the VAEs that attempt to provide solutions to the problems. We present applications of variational autoencoders for finance (a new and emerging field of application), speech/audio source separation, and biosignal applications. Experimental results are presented for an example of speech source separation to illustrate the powerful application of variants of VAE: VAE, -VAE, and ITL-AE. We conclude the paper with a summary, and we identify possible areas of research in improving performance of VAEs in particular and deep generative models in general, of which VAEs and generative adversarial networks (GANs) are examples.

Online blind source separation and dereverberation of speech based on a joint diagonalizability constraint

Online blind source separation and dereverberation of speech based on a joint diagonalizability constraint

End-to-End Monaural Speech Separation with a Deep Complex U-Shaped Network

Conventional time–frequency (TF) domain source separation methods mainly focus on predicting TF-masks or speech spectrums, where complex ideal ratio mask (cIRM) is an effective target for speech enhancement and separation. However, some recent studies employ a real-valued network, such as a general convolutional neural network (CNN) and a recurrent neural network (RNN), to predict a complex-valued mask or a spectrogram target, leading to the unbalanced training results of real and imaginary parts. In this paper, to estimate the complex-valued target more accurately, a novel U-shaped complex network for the complex signal approximation (uCSA) method is proposed. The uCSA is an adaptive front-end time-domain separation method, which tackles the monaural source separation problem in three ways. First, we design and implement a complex U-shaped network architecture comprising well-defined complex-valued encoder and decoder blocks, as well as complex-valued bidirectional Long Short-Term Memory (BLSTM) layers, to process complex-valued operations. Second, the cIRM is the training target of our uCSA method, optimized by signal approximation (SA), which takes advantage of both real and imaginary components of the complex-valued spectrum. Third, we re-formulate STFT and inverse STFT into derivable formats, and the model is trained with the scale-invariant source-to-noise ratio (SI-SNR) loss, achieving end-to-end training of the speech source separation task. Moreover, the proposed uCSA models are evaluated on the WSJ0-2mix datasets, which is a valid corpus commonly used by many supervised speech separation methods. Extensive experimental results indicate that our proposed method obtains state-of-the-art performance on the basis of the perceptual evaluation of speech quality (PESQ) and the short-time objective intelligibility (STOI) metrics.

Interactive speech source separation based on independent low-rank matrix analysis

Interactive speech source separation based on independent low-rank matrix analysis

Blind and Semi-blind Anechoic Mixing System Identification Using Multichannel Matching Pursuit

Sparse component analysis techniques have been successfully applied to the separation of speech sources. This paper presents an efficient algorithm based on the matching pursuit approach to deal with multichannel records. The proposed algorithm explicitly employs spatial constraints among different channels to express mixed signals as linear combinations of delayed components selected from an overcomplete dictionary. We present a new procedure for estimating the mixing system parameters (attenuations and delays), which can be applied to more than two mixtures and is not restricted to non-negative attenuation coefficients. The proposed mixing system estimation method can accommodate delays of greater magnitude than traditional approaches. In addition, learned dictionaries that improve the identification step can be used when excerpts from sources (exogenous to mixtures) are available. The simulation results show that semi-blind dictionaries perform better than those used in blind configurations.

Clustering of spatial cues by semantic segmentation for anechoic binaural source separation

The recent introduction of neural networks to speech separation has dramatically boosted the separation performance. This paper presents a novel psychoacoustic approach for speech source separation in anechoic conditions, using semantic segmentation of the interaural spectrograms of the audio mixtures. We have trained two separate U-Nets (a specialized neural network for semantic segmentation) on the interaural level difference (ILD) spectrogram, and the interaural phase difference (IPD) spectrogram of a single source. After training, these U-Nets are used to predict the class of each time frequency (TF) unit of the interaural spectrogram of the audio mixture. The ILD and IPD soft masks obtained from these U-Nets are combined by a novel scheme which utilizes the strength of the interaural cues in different frequency bands. The results show improved separation over two state of the art machine learning source separation systems utilizing the same interaural cues. There is average improvement of 7.32 dB in signal to distortion ratio (SDR) and 0.3 points improvement in short term objective intelligibility (STOI) over degenerate un-mixing estimation technique (DUET) algorithm and 2.51 dB improvement in SDR with comparable intelligibility over model-based expectation–maximization source separation and localization (MESSL) algorithm.

Speech Source Separation Using Variational Autoencoder and Bandpass Filter

Speech source separation is essential for speech-related applications because this process enhances the input speech signal for the main processing model. Most of the current approaches for this task focus on separating the speech of commonly high-frequency noises or a particular background sound. They cannot clear the signals which intersect with the human speech in its frequency range. To deal with this problem, we propose a hybrid approach combining a variational autoencoder (VAE) and a bandpass filter (BPF). This method can extract and enhance the speech signal in the mixture of many elements such as speech signal, the high-frequency noises, and many kinds of different background sounds which interfere with the speech sound. Experimental results showed that our model can extract effectively the speech signal with 15.02 dB in Signal to Interference Ratio (SIR) and 12.99 dB in Signal to Distortion Ratio (SDR). On the other hand, we can adjust the passband to identify the range of frequency at the output signal to apply for a particular application like gender classification.

Sparsity-based phase spectrum compensation for single-channel speech source separation

This paper proposes a sparsity-based phase spectrum compensation (SPSC) function to improve the quality of reconstructed signals by using magnitude-based single-channel speech source separation. While conventional approaches to the reconstruction of separated sources use the input phase spectrum, the proposed SPSC function modifies each source's phase spectrum using the estimated magnitude spectra of multiple sources and their spectro-temporal sparsity. In particular, the spectro-temporal sparsity is estimated from the signal-to-interference ratio between the magnitude spectrum of a source to be separated and those of the other sources, including the background noise. The proposed SPSC function is first embedded into the reconstruction stage of magnitude-based speech source separation methods that employ deep recurrent neural network (DRNN) and sparse nonnegative matrix factorization (SNMF), respectively. Then, speech denoising is performed under four different noise conditions with signal-to-noise ratios (SNRs) in the range of 0–15 dB. Both objective and subjective tests show that both DRNN and SNMF-based speech separation methods employing the proposed SPSC function substantially outperform those employing the conventional PSC function for the speech denoising task. Moreover, the DRNN combined with the proposed SPSC function offers higher average perceptual evaluation of speech quality (PESQ) scores for speech denoising than SEGAN, which is based on a very large end-to-end neural networks model. Next, the proposed SPSC method is also applied on the reconstruction stage of a deep clustering-based speech separation method to examine its contribution on a multi-talker speech separation task, where spoken utterances of males and females are mixed at −3, 0, and 3 dB SNR. It is shown from the experiment that the proposed SPSC function further improves SDR and PESQ scores of the magnitude spectra processed by the deep clustering-based single-channel speech separation method.

A Speech Synthesis Approach for High Quality Speech Separation and Generation

We propose a new method for source separation by synthesizing the source from a speech mixture corrupted by various environmental noise. Unlike traditional source separation methods which estimate the source from the mixture as a replica of the original source (e.g. by solving an inverse problem), our proposed method is a synthesis-based approach which aims to generate a new signal (i.e. “fake” source) that sounds similar to the original source. The proposed system has an encoder-decoder topology, where the encoder predicts intermediate-level features from the mixture, i.e. Mel-spectrum of the target source, using a hybrid recurrent and hourglass network, while the decoder is a state-of-the-art WaveNet speech synthesis network conditioned on the Mel-spectrum, which directly generates time-domain samples of the sources. Both objective and subjective evaluations were performed on the synthesized sources, and show great advantages of our proposed method for high-quality speech source separation and generation.

Expectation‐maximisation for speech source separation using convolutive transfer function

This paper addresses the problem of under-determinded speech source separation from multichannel microphone singals, i.e. the convolutive mixtures of multiple sources. The time-domain signals are first transformed to the short-time Fourier transform (STFT) domain. To represent the room filters in the STFT domain, instead of the widely-used narrowband assumption, we propose to use a more accurate model, i.e. the convolutive transfer function (CTF). At each frequency band, the CTF coefficients of the mixing filters and the STFT coefficients of the sources are jointly estimated by maximizing the likelihood of the microphone signals, which is resolved by an Expectation-Maximization (EM) algorithm. Experiments show that the proposed method provides very satisfactory performance under highly reverberant environments.

An Expectation–Maximization-Based IVA Algorithm for Speech Source Separation Using Student’s t Mixture Model Based Source Priors

The performance of the independent vector analysis (IVA) algorithm depends on the choice of the source prior to better model the speech signals as it employs a multivariate source prior to retain the dependency between frequency bins of each source. Identical source priors are frequently used for the IVA methods; however, different speech sources will generally have different statistical properties. In this work, instead of identical source priors, a novel Student’s t mixture model based source prior is introduced for the IVA algorithm that can adapt to the statistical properties of different speech sources and thereby enhance the separation performance of the IVA algorithm. The unknown parameters of the source prior and unmixing matrices are estimated together by deriving an efficient expectation maximization (EM) algorithm. Useful improvement in the separation performance in different realistic scenarios is confirmed by experimental studies on real datasets.