Monaural Speech Enhancement Research Articles

DPHT-ANet: Dual-path high-order transformer-style fully attentional network for monaural speech enhancement

Dual-path Transformer-style models have demonstrated significant effectiveness in speech enhancement. However, extensive parameterization and computational complexity present challenges for practical applications. This study presents an encoder-decoder-based dual-path high-order transformer-style fully-attentional network (DPHT-ANet) to address the speech enhancement problem with a smaller parameter size and reduced computational complexity. The DPHT-ANet incorporates a high-order information interaction module and replaces the multi-head attention module with a recursive gated convolution (GnConv). This enables the DPHT-ANet to effectively capture deep-level information across time and frequency dimensions, improving its ability to capture complex temporal and spectral patterns. Furthermore, DPHT-ANet uses a unified activation and attention mechanism in the convolutional encoder-decoder layers, resulting in a fully attentional network that prioritizes relevant high-level features at earlier stages. The DPHT-ANet uses interactive feature learning and fusion of varying lengths and dimensions with pre-trained features from a large-scale dataset to further enhance its robustness. Experimental results on the VCTK+DEMAND and WSJ0-SI84 datasets demonstrate the effectiveness of the proposed approach. On the WSJ0-SI84 dataset, the DPHT-ANet significantly improves ESTOI (38.28%), PESQ (1.21), and SDR (10.83 dB) over the noisy mixture. Similarly, on the VCTK+DEMAND, the DPHT-ANet improves STOI (3.50%), PESQ (1.22), and SegSNR (9.93 dB) over the noisy mixture, showcasing superior performance in speech enhancement.

Supervised Attention Multi-Scale Temporal Convolutional Network for monaural speech enhancement

Speech signals are often distorted by reverberation and noise, with a widely distributed signal-to-noise ratio (SNR). To address this, our study develops robust, deep neural network (DNN)-based speech enhancement methods. We reproduce several DNN-based monaural speech enhancement methods and outline a strategy for constructing datasets. This strategy, validated through experimental reproductions, has effectively enhanced the denoising efficiency and robustness of the models. Then, we propose a causal speech enhancement system named Supervised Attention Multi-Scale Temporal Convolutional Network (SA-MSTCN). SA-MSTCN extracts the complex compressed spectrum (CCS) for input encoding and employs complex ratio masking (CRM) for output decoding. The supervised attention module, a lightweight addition to SA-MSTCN, guides feature extraction. Experiment results show that the supervised attention module effectively improves noise reduction performance with a minor increase in computational cost. The multi-scale temporal convolutional network refines the perceptual field and better reconstructs the speech signal. Overall, SA-MSTCN not only achieves state-of-the-art speech quality and intelligibility compared to other methods but also maintains stable denoising performance across various environments.

Stacked Multiscale Densely Connected Temporal Convolutional Attention Network for Multi-Objective Speech Enhancement in an Airborne Environment

Airborne speech enhancement is always a major challenge for the security of airborne systems. Recently, multi-objective learning technology has become one of the mainstream methods of monaural speech enhancement. In this paper, we propose a novel multi-objective method for airborne speech enhancement, called the stacked multiscale densely connected temporal convolutional attention network (SMDTANet). More specifically, the core of SMDTANet includes three parts, namely a stacked multiscale feature extractor, a triple-attention-based temporal convolutional neural network (TA-TCNN), and a densely connected prediction module. The stacked multiscale feature extractor is leveraged to capture comprehensive feature information from noisy log-power spectra (LPS) inputs. Then, the TA-TCNN adopts a combination of these multiscale features and noisy amplitude modulation spectrogram (AMS) features as inputs to improve its powerful temporal modeling capability. In TA-TCNN, we integrate the advantages of channel attention, spatial attention, and T-F attention to design a novel triple-attention module, which can guide the network to suppress irrelevant information and emphasize informative features of different views. The densely connected prediction module is used to reliably control the flow of the information to provide an accurate estimation of clean LPS and the ideal ratio mask (IRM). Moreover, a new joint-weighted (JW) loss function is constructed to further improve the performance without adding to the model complexity. Extensive experiments on real-world airborne conditions show that our SMDTANet can obtain an on-par or better performance compared to other reference methods in terms of all the objective metrics of speech quality and intelligibility.

Decoupling-style monaural speech enhancement with a triple-branch cross-domain fusion network

Monaural speech enhancement aims to remove background noise from noisy speech signals captured by a single microphone. In recent years, several cross-domain monaural speech enhancement methods are developed to leverage both waveform and harmonic information. However, these methods fall short in fully capturing the dependencies between the time domain and time-frequency (T-F) domain, as well as in harnessing the benefits of the target decoupling strategy. This paper proposes a causal encoder-decoder-based Triple-branch Cross-domain Fusion Network (TCF-Net), which effectively processes speeches by leveraging both time domain and T-F domain features. The proposed approach enables the parallel recovery of magnitude and phase information to alleviate the compensation problem between them. TCF-Net forms a triple-branch network by collaboratively reconstructing the enhanced spectrum with a complex spectrum branch and a magnitude spectrum branch, while incorporating time-domain information with a waveform compensation branch. To fully leverage the information from three domains, Triple-domain Fusion Modules (TFMs) are inserted in each intermediate layer of the model to extract and merge the information from two T-F domain branches and one time domain branch. The TFMs generate masks to progressively compensate for the magnitude of the two T-F domain branches and promote information interaction, further restoring the magnitude of the clean speech. Experimental results demonstrate that TCF-Net outperforms state-of-the-art (SOTA) cross-domain methods and target decoupling methods under causal configuration in all evaluation metrics, which validates the importance of the proposed cross-domain information fusion strategy and target decoupling strategy.

Improving Monaural Speech Enhancement by Mapping to Fixed Simulation Space With Knowledge Distillation

Improving Monaural Speech Enhancement by Mapping to Fixed Simulation Space With Knowledge Distillation

Spectral Masking With Explicit Time-Context Windowing for Neural Network-Based Monaural Speech Enhancement

Spectral Masking With Explicit Time-Context Windowing for Neural Network-Based Monaural Speech Enhancement

Neural-Free Attention for Monaural Speech Enhancement Toward Voice User Interface for Consumer Electronics

The traditional graphic user interface in healthcare-oriented consumer electronics faced challenges such as high operational complexity, time-consuming operations, and a high risk of infection. The adoption of voice user interface (VUI) could promote network automation with enhanced efficiency, reduced simplicity and operating expense in various applications. Given noisy operational environments, speech enhancement acts as an indispensable component for VUIs towards consumer devices. Recently, attention mechanism is studied for speech enhancement and exhibits promising potential. In this paper, we propose a novel and effective attention module for speech enhancement, called neural-free attention (NFA), which is a lightweight and plug-and-play module that enables the backbone network to capture the energy distribution information of speech signals along frequency-wise channels. Particularly, NFA adopts a learnable Gaussian function to perform the excitation operation and produce the attention weights for each frequency channel. The NFA is comprehensively evaluated as part of the residual temporal convolution network (ResTCN) backbone network on two commonly used training targets. Experimental results show NFA substantially improves the ResTCN backbone in speech quality and intelligibility, with extremely low parameter overhead. Also, the ResTCN+NFA shows superiority over several recent baseline models, indicating the strong potential for VUIs toward consumer devices.

Deep neural network techniques for monaural speech enhancement and separation: state of the art analysis

AbstractDeep neural networks (DNN) techniques have become pervasive in domains such as natural language processing and computer vision. They have achieved great success in tasks such as machine translation and image generation. Due to their success, these data driven techniques have been applied in audio domain. More specifically, DNN models have been applied in speech enhancement and separation to perform speech denoising, dereverberation, speaker extraction and speaker separation. In this paper, we review the current DNN techniques being employed to achieve speech enhancement and separation. The review looks at the whole pipeline of speech enhancement and separation techniques from feature extraction, how DNN-based tools models both global and local features of speech, model training (supervised and unsupervised) to how they address label ambiguity problem. The review also covers the use of domain adaptation techniques and pre-trained models to boost speech enhancement process. By this, we hope to provide an all inclusive reference of all the state of art DNN based techniques being applied in the domain of speech separation and enhancement. We further discuss future research directions. This survey can be used by both academic researchers and industry practitioners working in speech separation and enhancement domain.

CompNet: Complementary network for single-channel speech enhancement

Recent multi-domain processing methods have demonstrated promising performance for monaural speech enhancement tasks. However, few of them explain why they behave better over single-domain approaches. As an attempt to fill this gap, this paper presents a complementary single-channel speech enhancement network (CompNet) that demonstrates promising denoising capabilities and provides a unique perspective to understand the improvements introduced by multi-domain processing. Specifically, the noisy speech is initially enhanced through a time-domain network. However, despite the waveform can be feasibly recovered, the distribution of the time–frequency bins may still be partly different from the target spectrum when we reconsider the problem in the frequency domain. To solve this problem, we design a dedicated dual-path network as a post-processing module to independently filter the magnitude and refine the phase. This further drives the estimated spectrum to closely approximate the target spectrum in the time–frequency domain. We conduct extensive experiments with the WSJ0-SI84 and VoiceBank + Demand datasets. Objective test results show that the performance of the proposed system is highly competitive with existing systems.

Dual branch deep interactive UNet for monaural noisy-reverberant speech enhancement

Noise and reverberation can severely degrade speech quality and intelligibility, so many deep neural network-based noisy-reverberant speech enhancement methods have been proposed, among which classic methods include spectral masking and spectral mapping. Spectrum masking and spectrum mapping have their advantages and disadvantages in different noise environments, and they are complementary. This paper proposes a dual branch deep interactive UNet (DBDIUNet) for monaural speech enhancement to combine the advantages of spectral mapping and spectral masking. The DBDIUNet uses a classical encoder-decoder architecture, including a shared encoder and two decoders. One decoder outputs the complex ideal ratio mask (cIRM), and the other outputs the enhanced complex spectrum. The two signals are coupled by coherent averaging to get the enhanced speech signal. A novel deep interaction structure is proposed for the interaction of information between the two decoders, which achieves a very significant performance improvement at the minimal cost of computational consumption and hyperparameters. Compared with the noisy speech on the Interspeech 2020 deep noise suppression challenge blind test set, DBDIUNet improves the WB-PESQ, NB-PESQ, STOI, SI-SDR indicators by 1.575, 0.955, 7.9%, 8.67 respectively. In the noisy-reverberant speech enhancement test, DBDIUNet improves the WB-PESQ, STOI, SI-SDR, DNSMOS, and SRMR by 0.98, 10.24%, 5.43, 1.51, 3.43, respectively, which exceeds the state-of-the-art model.

Channel and temporal-frequency attention UNet for monaural speech enhancement

The presence of noise and reverberation significantly impedes speech clarity and intelligibility. To mitigate these effects, numerous deep learning-based network models have been proposed for speech enhancement tasks aimed at improving speech quality. In this study, we propose a monaural speech enhancement model called the channel and temporal-frequency attention UNet (CTFUNet). CTFUNet takes the noisy spectrum as input and produces a complex ideal ratio mask (cIRM) as output. To improve the speech enhancement performance of CTFUNet, we employ multi-scale temporal-frequency processing to extract input speech spectrum features. We also utilize multi-conv head channel attention and residual channel attention to capture temporal-frequency and channel features. Moreover, we introduce the channel temporal-frequency skip connection to alleviate information loss between down-sampling and up-sampling. On the blind test set of the first deep noise suppression challenge, our proposed CTFUNet has better denoising performance than the champion models and the latest models. Furthermore, our model outperforms recent models such as Uformar and MTFAA in both denoising and dereverberation performance.

Ideal ratio mask estimation based on cochleagram for audio-visual monaural speech enhancement

In this paper, the estimation of the ideal ratio mask (IRM) has been carried out based on speech cochleagram and visual cues using Audio-Visual Multichannel Convolutional Neural Network (AVMCNN) to enhance the speech signal. Recently several researchers have shown that speech enhancement using visual data as an additional input along with the audio data is more effective in minimizing the acoustic noise present in the speech signal. This work proposes a novel CNN-based audio-visual IRM estimation model. In the proposed audio-visual IRM estimation model, the dynamics of both audio and visual signal features are extracted using multichannel CNN and contextually combined for speech enhancement. The enhanced speech obtained using the proposed model is evaluated based on speech quality and intelligibility. The evaluation results signify that the proposed audio-visual mask estimation model shows improved performance over the audio-only, visual-only, and existing audio-visual mask estimation models. In turn, the proposed AVMCNN model proves its effectiveness in combining the dynamics of the audio features with the visual speech features for speech enhancement.

A novel target decoupling framework based on waveform-spectrum fusion network for monaural speech enhancement

Speech enhancement refers to the technique of restoring clean speech from the corrupted noisy speech to reduce or suppress noise and interference. In recent years, the target decoupling strategy in the time-frequency (T-F) domain is proposed to alleviate the compensation problem between magnitude and phase. However, such target decoupling methods lack the utilization of waveform representations, so the neural networks cannot learn the complementary advantages between time domain and T-F domain. In this paper, we propose a cross-domain target decoupling system leveraging both waveform-domain and T-F domain semantic information to resynthesize the estimated spectrum. In order to fully use the information from both domains, the final reconstruction scheme is extended to each middle layer through lightweight semantic aggregation modules. Based on these above methods, the encoder-decoder-based waveform-spectrum fusion network (WSFNet) is proposed to concretely achieve cross-domain causal monaural speech enhancement. In WSFNet, the dual-path RNN blocks are embedded into the bottleneck layer to model both intra-frame and inter-frame long-range contextual correlations. Extensive experimental results demonstrate that our WSFNet outperforms the most advanced target decoupling systems and other previous state-of-the-art (SOTA) systems in terms of speech quality and intelligibility.

Convolutional gated recurrent unit networks based real-time monaural speech enhancement

Convolutional gated recurrent unit networks based real-time monaural speech enhancement

Convolutional Gated Recurrent Unit Networks Based Real-time Monaural Speech Enhancement

International Journal on Artificial Intelligence ToolsAccepted Papers No AccessConvolutional Gated Recurrent Unit Networks Based Real-time Monaural Speech EnhancementS. Vanambathina, V. Anumolu, P. Tejasree, and R. DivyaS. Vanambathina Search for more papers by this author , V. Anumolu Search for more papers by this author , P. Tejasree Search for more papers by this author , and R. Divya Search for more papers by this author https://doi.org/10.1142/S0218213023500392Cited by:0 PreviousNext AboutSectionsPDF/EPUB ToolsAdd to favoritesDownload CitationsTrack CitationsRecommend to Library ShareShare onFacebookTwitterLinked InRedditEmail Remember to check out the Most Cited Articles! Check out Notable Titles in Artificial Intelligence. FiguresReferencesRelatedDetails Recommended Accepted Papers Metrics History Received 13 May 2022 Accepted 17 August 2022 PDF download

GAN-in-GAN for Monaural Speech Enhancement

GAN-in-GAN for Monaural Speech Enhancement

Harmonic Attention for Monaural Speech Enhancement

To further improve the quality of the enhanced speech, it is appealing that more profound articulatory and auditory knowledge should be introduced into the speech enhancement model. Among these, harmonics seriously affect speech timbre and play a crucial role in speech intelligibility. Especially in the frequency domain, harmonics appear as the local maximum peaks of energy, which could be expected to serve as anchors to recover the distorted speech. In this paper, an explicit modeling method, harmonic attention, is presented, patching the harmonics with the help of residual ones. In order to maintain the spectral structure of speech during the processing and to enable the network to support harmonic modeling, a harmonic attention-based progressive enhancement network (HAPNet) is applied, which gradually approaches clean speech with stacked modules of harmonic attention. In addition, to make enhanced speech more consistent with hearing, a loss function based on the loudness power compression (LC-SNR) is used, which measures both magnitude and phase values with appropriate auditory effects. The experimental visualization indicates that the harmonic attention can capture and recover the harmonics of speech. And the objective evaluations show that the presented HAPNet and LC-SNR outperform the referenced methods. Furthermore, the presented model trained on 100 hours of data achieves competitive results with the referenced models trained on 3000+ hours of data, and one trained on 500 hours of data yields the state-of-the-art performance.

Multi‐stage attention network for monaural speech enhancement

AbstractAlthough current attention‐based speech enhancement methods have been proven to be capable of significantly improving the noise reduction performance, a bottleneck has arisen in juggling both detailed features and high‐level features: The more attention paid to such performance indicators as speech intelligibility index and articulation index will lead to the loss of subtle features such as syllable continuity and timbre distortion. To tackle such a problem, we divide the speech enhancement model into multiple stages, with a special attention mechanism introduced in each stage so that the detailed speech information is retained and the features of high levels are captured. In this research, a Multi‐Stage Attention Network (MSANet) is implemented in cascade by using three different attention modules: Self Attention, Channel Attention and Spatial Attention. The attention module in each stage needs only to focus on the features of its corresponding stage, thus giving full play to their respective advantages without affecting or damaging the features of other stages, helping decouple the feature extraction process and obtaining the features capable of complementing one another. The comparisons and ablation experiments show that the performance of MSANet is superior to those of current mainstream time domain or time‐frequency domain state‐of‐the‐art methods. Compared to the baseline, PESQ score of their model (3.11) has increased by 5%, CSIG (4.44) has increased by 2.5%, CBAK (3.63) has increased by 2.8% and COVL (3.81) has increased by 3.8%, which demonstrates the potential of MSANet as speech feature extraction backbones. An implementation of the PyTorch version is available here: http://www.msp‐lab.cn:1436/msp/MSANet.

First coarse, fine afterward: A lightweight two-stage complex approach for monaural speech enhancement

Deep neural network-based speech enhancement systems have achieved promising results. However, the state-of-the-art (SOTA) models usually have too many parameters and require too much computational work to be used on devices for practical applications. In this paper, we propose a novel lightweight complex spectral mask-based neural network with a two-stage pipeline for monaural speech enhancement. The network utilizes the idea of decoupling a primary problem into several simple sub-problems, which reduces the computational burden and model parameters. Specifically, the network contains two mask-based sub-networks, i.e., CoarseNet, and FineNet, implemented in the complex domain to improve the enhancement performances progressively. The CoarseNet takes the coarse-grained compact features as input and estimates the corresponding full-band complex mask. The FineNet focuses on further removing residual noises in the low-frequency components of CoarseNet output by predicting a fine-grained mask. The transforms between coarse- and fine-scale are based on a novel learnable complex-valued rectangular bandwidth (LCRB) filter bank. Furthermore, we also propose a lightweight and general complex-valued attention mechanism to improve the modeling capability of convolutional encoder/decoder of the network and uses cross-stage skip connections (CSSC) between sub-networks to facilitate information flowing between sub-networks. Extensive experiments on two standard corpora demonstrate that our proposed approach achieves better performances over previous SOTA systems under various conditions while maintaining relatively small model sizes and low computational complexity.

Joint waveform and magnitude processing for monaural speech enhancement

Speech enhancement is an essential task for improving the quality and intelligibility of speech signals corrupted by noise. Current deep neural network-based speech enhancement methods have achieved remarkable results. However, most of them only work in the time domain or the time–frequency domain, which do not fully use the complementary advantages of the two domains. In this paper, we propose a framework with joint waveform and magnitude processing for single-channel speech enhancement, which can realize the complementary advantages of the time-domain and time–frequency features. Specifically, the proposed network adopts a triple-stage training strategy. In the first two stages, the two sub-networks take waveform and magnitude as input features to generate two pre-enhanced speech signals, respectively. In the third stage, a fusion sub-network is used to fuse the two pre-enhanced speech signals, which further improves the quality and intelligibility of speech. All three sub-networks are encoder-decoder-based architecture with skip connections, and an additional temporal convolutional network is inserted between the encoder and the decoder. In order to improve the information flow of the network, we introduce the gating mechanism into the temporal convolutional network, which we refer to as the gated temporal convolutional network. In addition, the recently popular group communication strategy is also introduced into the network, which significantly reduces the number of trainable parameters and obtains on-par or better enhancement performance. Experimental results demonstrate that the proposed method consistently outperforms other advanced baselines in terms of objective speech quality and intelligibility metrics. Moreover, the proposed model also achieves outstanding cross-corpus and cross-language generalization capabilities.