Single Channel Audio Source Separation Research Articles

Speech Enhancement using Convolutional Autoencoder Network

Abstract—We present an end-to-end deep learning approach to denoising speech signals by processing the raw waveform directly. Given input audio containing speech corrupted by an additive background signal, the system aims to produce a processed signal that contains only the speech content. Recent approaches have shown promising results using various deep network architec- tures. In this paper, we propose to train a fully-convolutional context aggregation network using a deep feature loss. That loss is based on comparing the internal feature activations in a different network, trained for acoustic environment detection and domestic audio tagging. Our approach outperforms the stateof- the-art in objective speech quality metrics and in large-scale perceptual experiments with human listeners. It also outperforms an identical network trained using traditional regression losses. The advantage of the new approach is particularly pronounced for the hardest data with the most intrusive background noise, for which denoising is most needed and most challenging. Index Terms—speech enhancement,Fully convolutional denois- ing autoencoders, single channel audio source separation, stacked convolutional autoencoders, deep convolutional neural networks, deep learning.

SFSRNet: Super-resolution for Single-Channel Audio Source Separation

The problem of single-channel audio source separation is to recover (separate) multiple audio sources that are mixed in a single-channel audio signal (e.g. people talking over each other). Some of the best performing single-channel source separation methods utilize downsampling to either make the separation process faster or make the neural networks bigger and increase accuracy. The problem concerning downsampling is that it usually results in information loss. In this paper, we tackle this problem by introducing SFSRNet which contains a super-resolution (SR) network. The SR network is trained to reconstruct the missing information in the upper frequencies of the audio signal by operating on the spectrograms of the output audio source estimations and the input audio mixture. Any separation method where the length of the sequence is a bottleneck in speed and memory can be made faster or more accurate by using the SR network. Based on the WSJ0-2mix benchmark where estimations of the audio signal of two speakers need to be extracted from the mixture, in our experiments our proposed SFSRNet reaches a scale-invariant signal-to-noise-ratio improvement (SI-SNRi) of 24.0 dB outperforming the state-of-the-art solution SepFormer which reaches an SI-SNRi of 22.3 dB.

Majorization-Minimization Algorithm for Discriminative Non-Negative Matrix Factorization

This paper proposes a basis training algorithm for discriminative non-negative matrix factorization (NMF) with applications to single-channel audio source separation. With an NMF-based approach to supervised audio source separation, NMF is first applied to train the basis spectra of each source using training examples and then applied to the spectrogram of a mixture signal using the pretrained basis spectra at test time. The source signals can then be separated out using a Wiener filter. Here, a typical way to train the basis spectra is to minimize the dissimilarity measure between the observed spectrogram and the NMF model. However, obtaining the basis spectra in this way does not ensure that the separated signal will be optimal at test time due to the inconsistency between the objective functions for training and separation (Wiener filtering). To address this mismatch, a framework called discriminative NMF (DNMF) has recently been proposed. While this framework is noteworthy in that it uses a common objective function for training and separation, the objective function becomes more analytically complex than that of regular NMF. In the original DNMF work, a multiplicative update algorithm was proposed for the basis training; however, the convergence of the algorithm is not guaranteed and can be very slow. To overcome this weakness, this paper proposes a convergence-guaranteed algorithm for DNMF based on a majorization-minimization principle. Experimental results show that the proposed algorithm outperform the conventional DNMF algorithm as well as the regular NMF algorithm in terms of both the signal-to-distortion and signal-to-interference ratios.

Two-Stage Single-Channel Audio Source Separation Using Deep Neural Networks

Most single channel audio source separation approaches produce separated sources accompanied by interference from other sources and other distortions. To tackle this problem, we propose to separate the sources in two stages. In the first stage, the sources are separated from the mixed signal. In the second stage, the interference between the separated sources and the distortions are reduced using deep neural networks (DNNs). We propose two methods that use DNNs to improve the quality of the separated sources in the second stage. In the first method, each separated source is improved individually using its own trained DNN, while in the second method all the separated sources are improved together using a single DNN. To further improve the quality of the separated sources, the DNNs in the second stage are trained discriminatively to further decrease the interference and the distortions of the separated sources. Our experimental results show that using two stages of separation improves the quality of the separated signals by decreasing the interference between the separated sources and distortions compared to separating the sources using a single stage of separation.

On-the-Fly Audio Source Separation—A Novel User-Friendly Framework

This paper addresses the challenging problem of single-channel audio source separation. We introduce a novel user-guided framework where source models that govern the separation process are learned on-the-fly from audio examples retrieved online. The user only provides the search keywords that describe the sources in the mixture. In this framework, the generic spectral characteristics of each source are modeled by a universal sound class model learned from the retrieved examples via nonnegative matrix factorization. We propose several group sparsity-inducing constraints in order to efficiently exploit a relevant subset of the universal model adapted to the mixture to be separated. We then derive the corresponding multiplicative update rules for parameter estimation. Separation results obtained from automated and user tests on mixtures containing various types of sounds confirm the effectiveness of the proposed framework.

A multiresolution non-negative tensor factorization approach for single channel sound source separation

We propose a single channel audio source separation method to alleviate the smearing effects caused by fixed time-frequency (TF) resolution Short-Time Fourier Transform (STFT). We introduce a multiresolution representation based on Non-negative Tensor Factorization (NTF) where each layer of the tensor represents the mixture signal at a different time-frequency resolution. In order to fuse the information at different layers, the source separation is modeled as a joint optimization problem where the optimal solution is derived based on the Kullback–Leibler (KL) divergence. The resynthesis is made through an additional adaptive weighted fusion procedure which combines the sources separated at different scales by maximizing energy concentration. Numerical results over a large sound database indicate that the proposed joint optimization scheme enhances the quality of the separated sources both in terms of the conventional and the perceptual distortion measures.

Single-channel speech separation based on long–short frame associated harmonic model

Harmonic model is widely used in single-channel audio source separation. It has been proven effective in music source separation problem, where the harmonic peaks among the sources differ greatly from each other. However, in analyzing a speech signal, the short time window always introduces the harmonic overlapping in the frequency domain. In order to overcome the shortcoming, we propose a long-short frame associated harmonic (LSAH) model to separate two speech sources from a single-channel recording. The long frame can achieve high harmonic resolution, while the short frame can ensure the short time stationary feature of the speech signal. They are jointly used to improve the accuracy of the multi-pitch estimation. Autocorrelation method is adopted to estimate the prominent pitch with simplicity and accuracy. LSAH model and the prominent pitch are proposed to judge the state of the mixture and estimate the other pitch candidate. Our method can guarantee both the high harmonic resolution and the short time stationarity of the speech signal. Furthermore, it can separate some unvoiced segments from the mixture which cannot be handled by many of the existed methods. Experiments on 30 groups of mixtures show that the proposed algorithm outperforms the standard short time harmonic model in terms of both signal-to-noise ratio (SNR) and subjective listening quality.