Audio Separation Research Articles

Speech Isolation and Recognition in Crowded Noise Using a Dual-Path Recurrent Neural Network

Speech separation is crucial for effective speech processing in multi-talker conditions, especially in real-time, low-latency applications. In this study, the Time-Domain Audio Separation Network (TasNet) and Dual-Path Recurrent Neural Network were used to perform a time-domain multiple-speaker speech separation challenge. One-dimensional conventional recurrent neural networks (RNNs) are not capable of accurately simulating long sequences. When their receptive length exceeds the sequence field, 1-D CNNs cannot recreate utterance-level sequences. Dual-Path Recurrent Neural Network (DPRNN) breaks up the lengthy sequential input that progressively performs intra- and inter-chunk operations with input lengths proportional to the square root of the beginning sequence length. Model outputs are more efficient than earlier systems, improving performance on the Libri Mix dataset. Investigations show that the DPRNN, sample-level modeling, and time-domain audio separation network can replace present methods. EEND-SS and other separation algorithms perform worse than DPRNN. The suggested model was able to achieve (12.376) SI-SDR, (0.969) STOI (short-time objective intelligibility), (12.363) SDR, (9.363) DER, and (97.193) SCA.

The whole is greater than the sum of its parts: improving music source separation by bridging networks

This paper presents the crossing scheme (X-scheme) for improving the performance of deep neural network (DNN)-based music source separation (MSS) with almost no increasing calculation cost. It consists of three components: (i) multi-domain loss (MDL), (ii) bridging operation, which couples the individual instrument networks, and (iii) combination loss (CL). MDL enables the taking advantage of the frequency- and time-domain representations of audio signals. We modify the target network, i.e., the network architecture of the original DNN-based MSS, by adding bridging paths for each output instrument to share their information. MDL is then applied to the combinations of the output sources as well as each independent source; hence, we called it CL. MDL and CL can easily be applied to many DNN-based separation methods as they are merely loss functions that are only used during training and do not affect the inference step. Bridging operation does not increase the number of learnable parameters in the network. Experimental results showed that the validity of Open-Unmix (UMX), densely connected dilated DenseNet (D3Net) and convolutional time-domain audio separation network (Conv-TasNet) extended with our X-scheme, respectively called X-UMX, X-D3Net and X-Conv-TasNet, by comparing them with their original versions. We also verified the effectiveness of X-scheme in a large-scale data regime, showing its generality with respect to data size. X-UMX Large (X-UMXL), which was trained on large-scale internal data and used in our experiments, is newly available at https://github.com/asteroid-team/asteroid/tree/master/egs/musdb18/X-UMX.

Intelligent Substation Noise Monitoring System: Design, Implementation and Evaluation

In recent years, the State Grid of China has placed significant emphasis on the monitoring of noise in substations, driven by growing environmental concerns. This paper presents a substation noise monitoring system designed based on an end-network-cloud architecture, aiming to acquire and analyze substation noise, and report anomalous noise levels that exceed national standards for substation operation and maintenance. To collect real-time noise data at substations, a self-developed noise acquisition device is developed, enabling precise analysis of acoustic characteristics. Moreover, to subtract the interfering environmental background noise (bird/insect chirping, human voice, etc.) and determine if noise exceedances are originating from substation equipment, an intelligent noise separation algorithm is proposed by leveraging the convolutional time-domain audio separation network (Conv-TasNet), dual-path recurrent neural network (DPRNN), and dual-path transformer network (DPTNet), respectively, and evaluated under various scenarios. Experimental results show that (1) deep-learning-based separation algorithms outperform the traditional spectral subtraction method, where the signal-to-distortion ratio improvement (SDRi) and the scale-invariant signal-to-noise ratio improvement (SI-SNRi) of Conv-TasNet, DPRNN, DPTNet and the traditional spectral subtraction are 12.6 and 11.8, 13.6 and 12.4, 14.2 and 12.9, and 4.6 and 4.1, respectively; (2) DPTNet and DPRNN exhibit superior performance in environment noise separation and substation equipment noise separation, respectively; and (3) 91% of post-separation data maintains sound pressure level deviations within 1 dB, showcasing the effectiveness of the proposed algorithm in separating interfering noises while preserving the accuracy of substation noise sound pressure levels.

Anime Audio Retrieval Based on Audio Separation and Feature Recognition

This paper proposes an anime audio retrieval method based on audio separation and feature recognition techniques, aiming to help users conveniently locate their desired audio segments and enhance the overall user experience. Additionally, by establishing an audio fingerprint database and a corresponding copyright information management system, it becomes possible to track and manage the audio content within anime, effectively preventing piracy and unauthorized use, thereby improving the management and protection of audio resources. Traditional methods for anime audio feature recognition suffer from issues like low efficiency and subjective factors. In contrast, the proposed approach overcomes these limitations by automatically separating and extracting audio fingerprints from different audio sources within anime and creating an anime audio fingerprint database for fast retrieval. The paper utilizes an improved audio separation model based on the efficient channel attention mechanism to separate the anime audio. Subsequently, feature recognition is performed on the separated anime audio, employing a contrastive learning-based audio fingerprint retrieval method for anime audio fingerprinting. Experimental results demonstrate that the proposed algorithm effectively alleviates the issue of poor audio separation performance in anime audio, while also improving retrieval efficiency and accuracy, meeting the demands for anime audio content retrieval.

A Facial Feature and Lip Movement Enhanced Audio-Visual Speech Separation Model.

The cocktail party problem can be more effectively addressed by leveraging the speaker's visual and audio information. This paper proposes a method to improve the audio's separation using two visual cues: facial features and lip movement. Firstly, residual connections are introduced in the audio separation module to extract detailed features. Secondly, considering the video stream contains information other than the face, which has a minimal correlation with the audio, an attention mechanism is employed in the face module to focus on crucial information. Then, the loss function considers the audio-visual similarity to take advantage of the relationship between audio and visual completely. Experimental results on the public VoxCeleb2 dataset show that the proposed model significantly enhanced SDR, PSEQ, and STOI, especially 4 dB improvements in SDR.

Overlapped spectral demodulation of fiber Bragg grating using convolutional time-domain audio separation network

Overlapped spectral demodulation of fiber Bragg grating using convolutional time-domain audio separation network

Ensemble System of Deep Neural Networks for Single-Channel Audio Separation

Speech separation is a well-known problem, especially when there is only one sound mixture available. Estimating the Ideal Binary Mask (IBM) is one solution to this problem. Recent research has focused on the supervised classification approach. The challenge of extracting features from the sources is critical for this method. Speech separation has been accomplished by using a variety of feature extraction models. The majority of them, however, are concentrated on a single feature. The complementary nature of various features have not been thoroughly investigated. In this paper, we propose a deep neural network (DNN) ensemble architecture to completely explore the complimentary nature of the diverse features obtained from raw acoustic features. We examined the penultimate discriminative representations instead of employing the features acquired from the output layer. The learned representations were also fused to produce a new features vector, which was then classified by using the Extreme Learning Machine (ELM). In addition, a genetic algorithm (GA) was created to optimize the parameters globally. The results of the experiments showed that our proposed system completely considered various features and produced a high-quality IBM under different conditions.

Audio feature extraction: Foreground and Background audio separation using KNN algorithm

Data Science is a fairly novel field, and it predominantly deals with analysis and assortment of data. Machine Learning is a field that goes hand in hand in this regard. Various Algorithms, which are trained on a dataset predict results based on their training, and thus the accuracy of a model is determined by the testing dataset. Foreground feature extraction is another interesting application. Using data visualization and processing, we can plot the graphs for the audio frequency and intensity. This proves useful in feature extraction using MFCC (Mel-frequency cepstral coefficients).

Cross-Domain Conv-TasNet Speech Enhancement Model with Two-Level Bi-Projection Fusion of Discrete Wavelet Transform

Nowadays, time-domain features see wide use in speech enhancement (SE) networks such as frequency-domain features to achieve excellent performance in eliminating noise from input utterances. This study primarily investigates how to extract information from time-domain utterances to create more effective features in SE. We extend our recent work by employing sub-signals which dwell in multiple acoustic frequency bands in the time domain and integrating them into a unified time-domain feature set. The discrete wavelet transform (DWT) is applied to decompose each input frame signal to obtain sub-band signals, and a projection fusion process is performed on these signals to create the ultimate features. The corresponding fusion strategy is either bi-projection fusion (BPF) or multiple projection fusion (MPF). In short, MPF exploits the softmax function to replace the sigmoid function in order to create ratio masks for multiple feature sources. The concatenation of fused DWT features and time features serves as the encoder output of two celebrated SE frameworks, the fully convolutional time-domain audio separation network (Conv-TasNet) and the dual-path transformer network (DPTNet), to estimate the mask and then produce the enhanced time-domain utterances. The evaluation experiments are conducted on the VoiceBank-DEMAND and VoiceBank-QUT tasks, and the results reveal that the proposed method achieves higher speech quality and intelligibility than the original Conv-TasNet that uses time features only, indicating that the fusion of DWT features created from the input utterances can benefit time features to learn a superior Conv-TasNet/DPTNet network in SE.

A speech separation algorithm based on the comb-filter effect

In the field of speech separation, the traditional single-channel and multi-channel speech separation methods have made great progress. However, the accuracy of separation and automatic speech recognition(ASR) rate are not yet satisfactory. With the development of neural networks, some scholars began to use deep learning to achieve speech separation. Although this kind of method improves the accuracy of speech separation, it also leads to the need for pre-training the model, higher computational complexity and reduced separation performance when the model does not match the mixed signal. This paper has conducted an in-depth study on the scene of multi-speaker separation, and proposed a new dual-channel speech separation algorithm based on the Comb-Filter Effect (CFE). The CFE is an effect that occurs when a signal passes through a first-order differential microphone(FDM) array. And this effect is discovered and exploited for the first time. By using this effect, this paper designed a new signal spectrum estimation method that can realize accurate estimation of speech signal, and combined this method with traditional spectral subtraction to achieve the purpose of speech separation. Finally, this paper compared the proposed algorithm with the traditional FastICA-based algorithm and the fully-convolutional time-domain audio separation network(Conv-TasNet)-based algorithm. The results of simulation and comparison experiments show that the algorithm can effectively separate two-way speech signals while greatly reducing the computational complexity and has excellent robustness. In various situations, the proposed algorithm can obtain the Scale-Invariant Source-to-Noise Ratio improvement (SI-SNRi) of 9.19 dB on average. In addition, the Short-Time Objective Intelligibility (STOI) and Perceptual Evaluation of Speech Quality (PESQ) of the speech signal can be improved by an average of 33% and 70% or more respectively.

Single-channel blind separation of co-frequency signals based on convolutional network

Single-channel blind separation (SCBS) of co-frequency signals has been a challenging task in the blind signal processing field. Most effective approaches were still formulated in signal space, triggering complicated signal processing and high computational cost. To reduce the complexity and improve the separation accuracy of SCBS, in this paper, we propose a new waveform separation-demodulation scheme. As the first quintessential step, we design a convolutional time-domain network with squeeze-and-excitation blocks (CNSE) to carry out waveform separation in latent feature space, then the separated results are demodulated by the low-complexity per-survivor processing (LC-PSP) method. Our network adopts the backbone structure of encoder, separator, and decoder. And, it is worth noting that we deepen the encoder/decoder of the network to emphasize the low-level features and further weigh these features by means of the channel attention module. The encoder-decoder structure helps to eliminate the influence of noise during the stage of signal feature extraction and signal reconstruction. In our experiment, CNSE surpasses long short-term memory network and convolutional time-domain audio separation network as evaluated by signal-to-distortion ratio, signal-to-interferences ratio, and symbol error rate. Additionally, to reduce the influence of the truncation effect in separation, we also propose a data pre-processing method called redundancy protection (RP), and our trials reveal that RP can compensate for the performance of CNSE in the whole signal-to-noise ratio range. It is also verified that, compared with the traditional PSP algorithm, our method has better noise robustness and higher separation efficiency, especially when processing more complex mixtures.

Zero-Shot Audio Source Separation through Query-Based Learning from Weakly-Labeled Data

Deep learning techniques for separating audio into different sound sources face several challenges. Standard architectures require training separate models for different types of audio sources. Although some universal separators employ a single model to target multiple sources, they have difficulty generalizing to unseen sources. In this paper, we propose a three-component pipeline to train a universal audio source separator from a large, but weakly-labeled dataset: AudioSet. First, we propose a transformer-based sound event detection system for processing weakly-labeled training data. Second, we devise a query-based audio separation model that leverages this data for model training. Third, we design a latent embedding processor to encode queries that specify audio targets for separation, allowing for zero-shot generalization. Our approach uses a single model for source separation of multiple sound types, and relies solely on weakly-labeled data for training. In addition, the proposed audio separator can be used in a zero-shot setting, learning to separate types of audio sources that were never seen in training. To evaluate the separation performance, we test our model on MUSDB18, while training on the disjoint AudioSet. We further verify the zero-shot performance by conducting another experiment on audio source types that are held-out from training. The model achieves comparable Source-to-Distortion Ratio (SDR) performance to current supervised models in both cases.

Fast accuracy estimation of deep learning based multi-class musical source separation

Music source separation represents the task of extracting all the instruments from a given song. Recent breakthroughs on this challenge have gravitated around a single dataset, MUSDB, only limited to four instrument classes. Larger datasets and more instruments are costly and time-consuming in collecting data and training deep neural networks (DNNs). In this work, we propose a fast method to evaluate the separability of instruments in any dataset without training and tuning a DNN.This separability measure helps to select appropriate samples for the efficient training of neural networks. Based on the oracle principle with an ideal ratio mask, our approach is an excellent proxy to estimate the separation performances of state-of-the-art deep learning approaches such as TasNet or Open-Unmix.Our results contribute to revealing two essential points for audio source separation: 1) the ideal ratio mask, although light and straightforward, provides an accurate measure of the audio separability performance of recent neural nets, and 2) new end-to-end learning methods such as Tasnet, that operate directly on waveforms, are, in fact, internally building a Time-Frequency (TF) representation, so that they encounter the same limitations as the TF based-methods when separating audio pattern overlapping in the TF plane.

Gestalt Principles Emerge When Learning Universal Sound Source Separation

Sound source separation is an essential aspect in auditory scene analysis, which is still an urgent challenge for machine hearing. In this paper, a fully convolutional time-domain audio separation network (ConvTasNet) is trained for universal two-source separation, consisting of speech, environmental sounds, and music. Besides the separation performance of the network, the underlying separation mechanisms are our main concern. Through a series of classic auditory segregation experiments, we systematically explore the principles learned by the network for simultaneous and sequential organization. The results show that without prior knowledge of auditory scene analysis imparted on the network, it spontaneously learns the separation mechanisms from raw waveforms that are similar to those which have developed over many years in humans. The Gestalt principles for separation in the human auditory system are shown to be effective in our network: harmonicity, onset synchrony and common fate (coherent modulation in amplitude and frequency), proximity, continuity, similarity. The universal sound source separation network following Gestalt principles is not limited to specific sources and can be applied to various acoustic situations like human hearing, providing new directions for solving the problem of auditory scene analysis.

Dual-Path Hybrid Attention Network for Monaural Speech Separation

Recent advances in the time-domain speech separation methods, particularly those specialized in using attention mechanisms to model sequences, have significantly improved speech separation performance. In this paper, we address monaural (one microphone) speaker separation, mainly in the case of two concurrent speakers. We propose a dual-path hybrid attention network (DPHA-Net) for monaural speech separation based on time-domain. The critical component of DPHA-Net, the DPHA module, comprises multiple attentions and is designed to capture the short and long-term context information dependencies. DPHA module consists of the multi-head self-attention (MHSA), element-wise attention (EA), and adaptive feature fusion (AFF) units. We proposed an improved multi-stage aggregation training strategy during the training. That strategy has proven very effective for audio separation in this paper. The results of experiments on the benchmark WSJ0-2mix, WHAM! and Libri2Mix datasets show that our proposed DPHA-Net can achieves the competitive performance. For the task of two speaker separation on the WSJ0-2mix dataset, our proposed DPHA-Net is superior to the state of the art with a margin of 0.3 dB absolute improvement on the SI-SNRi and a margin of 0.4 dB absolute improvement on the SDRi in the same condition.

Auditory-like simultaneous separation mechanisms spontaneously learned by a deep source separation network

Selective listening to different sound sources in complex acoustic scenes is still an urgent challenge for machine hearing. In this study, the separation of two sound sources from monaural mixtures is investigated. An end-to-end fully convolutional time-domain audio separation network (ConvTasNet) is trained by a universal dataset, which includes speech, environmental sounds, and music. After the best performing network, mixtures in the test dataset can obtain an average scale-invariant signal-to-distortion ratio improvement (SI-SDRi) of 11.70 dB, which is comparable with the human performance to separate natural sources. Except for the promising performance, the main contribution of our study is to reveal the underlying separation mechanisms of the network through a series of classical human auditory segregation experiments. Results show that without any biological modeling of the auditory system, the proposed network spontaneously mimics aspects of the auditory system to separate sources. Not only the frequency proximity and harmonicity principles for auditory scene analysis are spontaneously learned by such a pure statistical deep network, but also the frequency selectivity in high and low frequencies and the resolvability of harmonics are precisely simulated. The emergence of deep networks with similar behavior characteristics to human beings provides the possibility to develop a universal network that can be adapted to all scenes and achieve selective listening like the human ear. On the other hand, it also provides a new perspective to the modeling of the auditory system for other problems such as recognition and localization.

Attention-Based Joint Training of Noise Suppression and Sound Event Detection for Noise-Robust Classification.

Sound event detection (SED) recognizes the corresponding sound event of an incoming signal and estimates its temporal boundary. Although SED has been recently developed and used in various fields, achieving noise-robust SED in a real environment is typically challenging owing to the performance degradation due to ambient noise. In this paper, we propose combining a pretrained time-domain speech-separation-based noise suppression network (NS) and a pretrained classification network to improve the SED performance in real noisy environments. We use group communication with a context codec method (GC3)-equipped temporal convolutional network (TCN) for the noise suppression model and a convolutional recurrent neural network for the SED model. The former significantly reduce the model complexity while maintaining the same TCN module and performance as a fully convolutional time-domain audio separation network (Conv-TasNet). We also do not update the weights of some layers (i.e., freeze) in the joint fine-tuning process and add an attention module in the SED model to further improve the performance and prevent overfitting. We evaluate our proposed method using both simulation and real recorded datasets. The experimental results show that our method improves the classification performance in a noisy environment under various signal-to-noise-ratio conditions.

Research on Blind Source Separation of Transformer Vibration Signal Based on Full Convolution Time Domain Audio Separation Network

The vibration signal on the surface of the transformer tank is formed by mixing the winding and the iron core. Blind source separation of vibration signals can diagnose the operating status of windings and iron cores separately, and improve the accuracy of using vibration signals to diagnose transformers. However, due to the high correlation between the vibration signals of the winding and the iron core, traditional blind source separation algorithms such as FastICA based on the assumption of signal independence have certain limitations. In this paper, a fully convolutional time-domain audio separation network (ConvTasNet) based on a deep learning model is used to blind source separation of transformer vibration signals. The training set and verification set are used to iterate the network. The training is completed when the network loss value is iterated to the specified requirements. This will realize the separation of the vibration signal of the winding and the iron core. This paper has carried out simulation experiments to verify the algorithm. Compared with the traditional FastICA method, the separation effect has been improved. It can more accurately separate the vibration signals of different sources, and the fault diagnosis of transformer windings and iron cores based on vibration signals Promotion and application are of great significance.

ConvTasNet-based anomalous noise separation for intelligent noise monitoring

Noise pollution has become a growing concern in public health. The availability of low-cost wireless acoustic sensor networks permits continuous monitoring of noise. However, real acoustic scenes are composed of irrelevant sources (anomalous noise) that overlap with monitored noise, causing biased evaluation and controversy. One classical scene is selected in our study. For road traffic noise assessment, other possible non-traffic noise (e.g., speech, thunder) should be excluded to obtain a reliable evaluation. Because anomalous noise is diverse, occasional, and unpredictable in real-life scenes, removing it from the mixture is a challenge. We explore a fully convolutional time-domain audio separation network (ConvTasNet) for arbitrary sound separation. ConvTasNet is trained by a large dataset, including environmental sounds, speech, and music over 150 hours. After training, the scale-invariant signal-to-distortion ratio (SI-SDR) is improved by 11.40 dB on average for an independent test dataset. ConvTasNet is next applied to anomalous noise separation of traffic noise scenes. We mix traffic noise and anomalous noise at random SNR between -10 dB to 0 dB. Separation is especially effective for salient and long-term anomalous noise, which smooth the overall sound pressure level curve over time. Results emphasize the importance of anomalous noise separation for reliable evaluation.

Target exaggeration for deep learning-based speech enhancement

Deep learning has been actively utilized for speech enhancement. However, deep learning-based speech enhancement usually produces over-smoothed speech, resulting in speech distortion and degraded intelligibility. In this paper, we propose the exaggeration of the training target so that the dynamic range of the enhanced speech becomes more similar to that of the clean speech. Target exaggeration can be implemented in two ways. The first approach is to exaggerate the target feature in the cost function of a deep learning-based speech enhancement system. This method can be implemented without additional parameters or computation, but can only be applied to schemes working in the time-frequency domain with the mean-square error cost function. The second approach is to introduce an additional deep neural network (DNN) that estimates the residual error in the output of a deep learning-based speech enhancement. This requires more computation, but can be applied even to time-domain approaches. To evaluate the performance of the proposed target exaggeration, it is applied to a feed-forward DNN- and long short-term memory (LSTM)-based speech enhancement scheme in the time-frequency domain, and the convolutional time-domain audio separation network (Conv-TasNet)-based speech enhancement scheme in the time domain. Experimental results showed that the proposed method improved the quality of speech produced by the deep learning-based speech enhancement system in terms of the perceptual evaluation of speech quality (PESQ) scores and outperformed other approaches, including global variance equalization and a perceptually optimized speech denoising autoencoder, to alleviate the over-smoothing problem.