Speech Enhancement Method Research Articles

From simulation to reality: tackling data mismatches in speech enhancement with unsupervised pre-training

In this study, we introduce an innovative speech enhancement methodology that ingeniously combines unsupervised pre-training with supervised fine-tuning. This hybrid approach directly addresses the prevalent data mismatch challenge inherent in traditional supervised speech enhancement methods. Our technique distinctly utilizes unpaired noisy and clean speech data and incorporates varied noises during the pre-training phase. This strategy effectively simulates the benefits of supervised learning, eliminating the need for paired data. Inspired by contrastive learning techniques prevalent in computer vision, our model is adept at preserving essential speech features amidst noise interference. At the heart of our method lies a sophisticated Generative Adversarial Network (GAN) architecture. This includes a generator that proficiently processes both magnitude and complex-domain features, alongside a discriminator meticulously designed to optimize specific evaluation metrics. Through rigorous experimental evaluations, we validate the robustness and versatility of our approach. It consistently delivers superior speech quality, demonstrating remarkable efficacy in real-world scenarios, which are often characterized by complex and unpredictable noise environments.

Deep-learning-based speech enhancement under rough-focus conditions with optical laser microphone

Optical laser microphones have attracted attention for acoustic systems capable of recording the target speech from a distance. An optical laser microphone measures the speech-induced vibration by focusing the laser beam on the surface of the vibrating object. A recording method called rough-focus recording using an unfocused laser beam, enables wide-range recording and robust recording against changes in focal length. However, with rough-focus recording, the broad laser beam coverage leads to insufficient intensity of the reflected laser for accurate acoustical signal measurement, resulting in speech-quality degradation, such as the inclusion of noise, and attenuation of high-frequency components in the acquired speech. To solve this problem, deep-learning-based speech-enhancement methods for optical laser microphones have been proposed. Such methods require separate models for different focus settings, exhibiting a lack of adaptability to changing focus settings. We propose a speech-enhancement method for training a single model for various focus settings. This model is trained with speech signals recorded across different focus settings to enhance speech recorded in various focus settings. Experimental results indicate that the this model trained with the proposed method performs equivalent to or better than a model trained with the conventional models.

Deep autoencoder with gated convolutional neural networks for improving speech quality in secured communications

In this study, we introduce a speech enhancement method to improve the quality of decrypted speech signals from hand-talk devices, which are highly susceptible to security attacks. Ensuring high-quality decrypted speech is essential because traditional speech enhancement methods struggle with artifacts only present during speech due to the encryption process applied selectively. This situation limits the effectiveness of traditional methods, which assume distortion is constant and can be estimated during silent periods. Our solution involves a deep-learning approach that employs a gated convolutional neural network (GCNN). Unlike typical convolutional neural networks (CNNs) that excel in processing spatial data but falter with temporal changes, our GCNN integrates a gating mechanism to enhance handling of temporal dynamics in speech data. This method directly maps distorted speech to its clean counterpart, bypassing the need for explicit noise estimation. Our experiments indicate that this deep-learning method significantly outperforms traditional speech enhancement techniques and conventional CNNs in several key evaluation metrics, offering a promising advancement in decrypted speech quality enhancement.

A dual-region speech enhancement method based on voiceprint segmentation

Single-channel speech enhancement primarily relies on deep learning models to recover clean speech signals from noise-contaminated speech. These models establish a mapping relationship between noisy and clean speech. However, considering the sparse distribution characteristics of speech energy across the entire time–frequency spectrogram, constructing the mapping relationship from noisy to clean speech exhibits significant differences in regions where speech energy is concentrated and non-concentrated. Utilizing one deep model to simultaneously address these two distinct regression tasks increases the complexity of the mapping relationships, consequently restricting the model’s performance. To validate our hypothesis, we propose a dual-region speech enhancement model based on voiceprint region segmentation. Specifically, we first train a voiceprint segmentation model to classify noisy speech into two regions. Subsequently, we establish dedicated speech enhancement models for each region, with the dual-region models concurrently constructing mapping relationships for noise-corrupted speech to clean speech in distinct regions. Finally, by merging the results, the complete restored speech can be obtained. Experimental results on public datasets demonstrate that our method achieves competitive speech enhancement performance, outperforming the state-of-the-art. Ablation study results confirm the effectiveness of the proposed approach in enhancing model performance.

A neural network approach for speech enhancement and noise-robust bandwidth extension

When processing noisy utterances with varying frequency bandwidths using an enhancement model, the effective bandwidth of the resulting enhanced speech often remains unchanged. However, high-frequency components are crucial for perceived audio quality, underscoring the need for noise-robust bandwidth extension capabilities in speech enhancement networks. In this study, we addressed this challenge by proposing a novel network architecture and loss function based on the CAUNet, which is a state-of-the-art speech enhancement method. We introduced a multi-scale loss and implemented a coordinate embedded upsampling block to facilitate bandwidth extension while maintaining the ability of speech enhancement. Additionally, we proposed a gradient loss function to promote the neural network’s convergence, leading to significant performance improvements. Our experimental results validate these modifications and clearly demonstrate the superiority of our approach over competing methods.

Assessment of Self-Supervised Denoising Methods for Esophageal Speech Enhancement

Esophageal speech (ES) is a pathological voice that is often difficult to understand. Moreover, acquiring recordings of a patient’s voice before a laryngectomy proves challenging, thereby complicating enhancing this kind of voice. That is why most supervised methods used to enhance ES are based on voice conversion, which uses healthy speaker targets, things that may not preserve the speaker’s identity. Otherwise, unsupervised methods for ES are mostly based on traditional filters, which cannot alone beat this kind of noise, making the denoising process difficult. Also, these methods are known for producing musical artifacts. To address these issues, a self-supervised method based on the Only-Noisy-Training (ONT) model was applied, consisting of denoising a signal without needing a clean target. Four experiments were conducted using Deep Complex UNET (DCUNET) and Deep Complex UNET with Complex Two-Stage Transformer Module (DCUNET-cTSTM) for assessment. Both of these models are based on the ONT approach. Also, for comparison purposes and to calculate the evaluation metrics, the pre-trained VoiceFixer model was used to restore the clean wave files of esophageal speech. Even with the fact that ONT-based methods work better with noisy wave files, the results have proven that ES can be denoised without the need for clean targets, and hence, the speaker’s identity is retained.

Air Traffic Control Speech Enhancement Method Based on Improved DNN-IRM

The quality of air traffic control speech is crucial. However, internal and external noise can impact air traffic control speech quality. Clear speech instructions and feedback help optimize flight processes and responses to emergencies. The traditional speech enhancement method based on a deep neural network and ideal ratio mask (DNN-IRM) is prone to distortion of the target speech in a strong noise environment. This paper introduces an air traffic control speech enhancement method based on an improved DNN-IRM. It employs LeakyReLU as an activation function to alleviate the gradient vanishing problem, improves the DNN network structure to enhance the IRM estimation capability, and adjusts the IRM weights to reduce noise interference in the target speech. The experimental results show that, compared with other methods, this method improves the perceptual evaluation of speech quality (PESQ), short-term objective intelligibility (STOI), scale-invariant signal-to-noise ratio (SI-SNR), and speech spectrogram clarity. In addition, we use this method to enhance real air traffic control speech, and the speech quality is also improved.

Objective and subjective evaluation of speech enhancement methods in the UDASE task of the 7th CHiME challenge

Supervised models for speech enhancement are trained using artificially generated mixtures of clean speech and noise signals. However, the synthetic training conditions may not accurately reflect real-world conditions encountered during testing. This discrepancy can result in poor performance when the test domain significantly differs from the synthetic training domain. To tackle this issue, the UDASE task of the 7th CHiME challenge aimed to leverage real-world noisy speech recordings from the test domain for unsupervised domain adaptation of speech enhancement models. Specifically, this test domain corresponds to the CHiME-5 dataset, characterized by real multi-speaker and conversational speech recordings made in noisy and reverberant domestic environments, for which ground-truth clean speech signals are not available. In this paper, we present the objective and subjective evaluations of the systems that were submitted to the CHiME-7 UDASE task, and we provide an analysis of the results. This analysis reveals a limited correlation between subjective ratings and several supervised nonintrusive performance metrics recently proposed for speech enhancement. Conversely, the results suggest that more traditional intrusive objective metrics can be used for in-domain performance evaluation using the reverberant LibriCHiME-5 dataset developed for the challenge. The subjective evaluation indicates that all systems successfully reduced the background noise, but always at the expense of increased distortion. Out of the four speech enhancement methods evaluated subjectively, only one demonstrated an improvement in overall quality compared to the unprocessed noisy speech, highlighting the difficulty of the task. The tools and audio material created for the CHiME-7 UDASE task are shared with the community.

Using deep learning to improve the intelligibility of a target speaker in noisy multi-talker environments for people with normal hearing and hearing loss.

Understanding speech in noisy environments is a challenging task, especially in communication situations with several competing speakers. Despite their ongoing improvement, assistive listening devices and speech processing approaches still do not perform well enough in noisy multi-talker environments, as they may fail to restore the intelligibility of a speaker of interest among competing sound sources. In this study, a quasi-causal deep learning algorithm was developed that can extract the voice of a target speaker, as indicated by a short enrollment utterance, from a mixture of multiple concurrent speakers in background noise. Objective evaluation with computational metrics demonstrated that the speaker-informed algorithm successfully extracts the target speaker from noisy multi-talker mixtures. This was achieved using a single algorithm that generalized to unseen speakers, different numbers of speakers and relative speaker levels, and different speech corpora. Double-blind sentence recognition tests on mixtures of one, two, and three speakers in restaurant noise were conducted with listeners with normal hearing and listeners with hearing loss. Results indicated significant intelligibility improvements with the speaker-informed algorithm of 17% and 31% for people without and with hearing loss, respectively. In conclusion, it was demonstrated that deep learning-based speaker extraction can enhance speech intelligibility in noisy multi-talker environments where uninformed speech enhancement methods fail.

Weak abnormal acoustic signal enhancement and recognition using squeeze-and-excitation attention based denoising convolutional neural network during high-dam flood discharging

Acoustic signals (particularly cavitation acoustic signals) generated during the flood discharge of high dams are highly sensitive to various abnormal situations, whereas weak abnormal signal recognition under strong discharge-noise interference is extremely challenging. Based on the prototype and model experiments, the related abnormal acoustic signals and discharge noise were recorded to construct datasets. Subsequently, using the framework of the deep neural network (DNN) speech enhancement method, a squeeze-and-excitation attention based denoising convolutional neural network (DnCNN) based method for weak abnormal acoustic signal enhancement and recognition was proposed and verified using two case studies of cavitation acoustic signal enhancement and multicategory acoustic signal enhancement and recognition. Compared with the DnCNN method and traditional signal processing methods (such as wavelet, empirical mode decomposition, least mean square, and recursive least square), the proposed method achieved excellent signal enhancement performance after training based on limited prior knowledge of signal and noise. It also demonstrated good generalization ability and robustness in multicategory tasks, which significantly improved the abnormal signal recognition accuracy. This study provides technical support for the practical application of acoustic monitoring based on DNN for safety during the flood discharge of high dams.

PDenoiser: A Personalized Speech Enhancement Neural Network for Pre-hospital Emergency Medical Services.

Pre-hospital emergency medical service (EMS) tasks often come with complex and diverse noise interferences, posing challenges in implementing ASR-based medical technologies and hindering efficient and accurate telephonic communication. Among the different types of noise distortion, interfering speech is especially annoying. To address these issues, our aim is to develop a technology capable of extracting the intended speech content of the target physician from noisy and mixed audio during EMS tasks. In this work, we propose a monoaural personalized speech enhancement (PSE) method called pDenoiser, which is a real-time neural network that operates in the time domain. By leveraging the prior vocalization cues of emergency physicians, pDenoiser selectively enhances target speech components while suppressing noise and nontarget speech components, thereby improving speech quality and speech recognition accuracy under noisy conditions. We demonstrate the potential value of our approach through evaluations on both public general-domain test sets and our self-collected real-world EMS test sets. The experimental results are promising, as our model effectively promotes both speech quality and ASR performance under various conditions and outperforms related methods across multiple evaluation metrics. Our methodology will hopefully elevate EMS efficiency and fortify security against nontarget speech during EMS tasks.

Supervised Attention Multi-Scale Temporal Convolutional Network for monaural speech enhancement

Speech signals are often distorted by reverberation and noise, with a widely distributed signal-to-noise ratio (SNR). To address this, our study develops robust, deep neural network (DNN)-based speech enhancement methods. We reproduce several DNN-based monaural speech enhancement methods and outline a strategy for constructing datasets. This strategy, validated through experimental reproductions, has effectively enhanced the denoising efficiency and robustness of the models. Then, we propose a causal speech enhancement system named Supervised Attention Multi-Scale Temporal Convolutional Network (SA-MSTCN). SA-MSTCN extracts the complex compressed spectrum (CCS) for input encoding and employs complex ratio masking (CRM) for output decoding. The supervised attention module, a lightweight addition to SA-MSTCN, guides feature extraction. Experiment results show that the supervised attention module effectively improves noise reduction performance with a minor increase in computational cost. The multi-scale temporal convolutional network refines the perceptual field and better reconstructs the speech signal. Overall, SA-MSTCN not only achieves state-of-the-art speech quality and intelligibility compared to other methods but also maintains stable denoising performance across various environments.

INCREASING ROBUSTNESS OF I-VECTORS VIA MASKING: A CASE STUDY IN SYNTHETIC SPEECH DETECTION

Ensuring security in speaker recognition systems is crucial. In the past years, it has been demonstrated that spoofing attacks can fool these systems. In order to deal with this issue, spoof speech detection systems have been developed. While these systems have served with a good performance, their effectiveness tends to degrade under noise. Traditional speech enhancement methods are not efficient for improving performance, they even make it worse. In this research paper, performance of the noise mask obtained via a convolutional neural network structure for reducing the noise effects was investigated. The mask is used to suppress noisy regions of spectrograms in order to extract robust i-vectors. The proposed system is tested on the ASVspoof 2015 database with three different noise types and accomplished superior performance compared to the traditional systems. However, there is a loss of performance in noise types that are not encountered during training phase.

Stacked Multiscale Densely Connected Temporal Convolutional Attention Network for Multi-Objective Speech Enhancement in an Airborne Environment

Airborne speech enhancement is always a major challenge for the security of airborne systems. Recently, multi-objective learning technology has become one of the mainstream methods of monaural speech enhancement. In this paper, we propose a novel multi-objective method for airborne speech enhancement, called the stacked multiscale densely connected temporal convolutional attention network (SMDTANet). More specifically, the core of SMDTANet includes three parts, namely a stacked multiscale feature extractor, a triple-attention-based temporal convolutional neural network (TA-TCNN), and a densely connected prediction module. The stacked multiscale feature extractor is leveraged to capture comprehensive feature information from noisy log-power spectra (LPS) inputs. Then, the TA-TCNN adopts a combination of these multiscale features and noisy amplitude modulation spectrogram (AMS) features as inputs to improve its powerful temporal modeling capability. In TA-TCNN, we integrate the advantages of channel attention, spatial attention, and T-F attention to design a novel triple-attention module, which can guide the network to suppress irrelevant information and emphasize informative features of different views. The densely connected prediction module is used to reliably control the flow of the information to provide an accurate estimation of clean LPS and the ideal ratio mask (IRM). Moreover, a new joint-weighted (JW) loss function is constructed to further improve the performance without adding to the model complexity. Extensive experiments on real-world airborne conditions show that our SMDTANet can obtain an on-par or better performance compared to other reference methods in terms of all the objective metrics of speech quality and intelligibility.

Analysis of statistical estimators and neural network approaches for speech enhancement

Speech communicated is adversely affected by environmental noise. It is important to process the speech and reduce noise for better understanding. Speech enhancement or noise reduction is useful to provide comfort for human or machine listening. Traditional algorithms provide better noise reduction and better-quality speech. Due to the non-stationary nature of noise and the quasi-stationary nature of speech, the traditional methods are proven inadequate in achieving high-quality speech. Later statistical estimators based on Gaussian, and super-Gaussian Probability Density Function (PDF) assumption further improved the enhancement performance. But still, non-stationary noise nature introduces artifacts in processed signal and results in decreased performance. It is observed that neural network approaches and the factorization approach provide better performance even under non-stationary noises by proper training and large database. Different features result in variations in output performance under unseen noise and speaker conditions. It is important to understand the importance and advantages of traditional methods, statistical estimators, and neural network approaches performances. To select the suitable method for a required application, it is essential to consider the trade-off between quality and distortion. In this work, the importance of speech enhancement methods is discussed. Performance measures used for understanding the speech enhancement like Signal to Noise Ratio (SNR), Segmental SNR, Log-Likelihood Ratio (LLR), Weighted Spectral Slope (WSS), Perceptual Evaluation of Speech Quality (PESQ), Short-Time Objective Intelligibility (STOI), Signal to Distortion Ratio (SDR) and Mean Opinion Score (MOS), are given. Highlights of important results are discussed for analyzing better speech enhancement methods for the required application. In this work, performance is compared using objective and subjective performance measures. Simulation results show superior performance when neural network is employed in statistical estimators.

Towards an Environmentally Robust Speech Assistant System for Emergency Medical Services.

Automated speech recognition technology with robust performance in various environments is highly needed by emergency clinicians, but there are few successful cases. One main challenge is the wide variety of environmental interference involved during a typical prehospital care emergency service such as background noises and overlapping speech. To solve this problem, we try to establish an environmentally robust speech assistant system with the help of the proposed personalized speech enhancement (PSE) method, which utilizes the target physician's voiceprint feature to suppress non-target signal components. We demonstrate its potential value using both general public test set and our real EMS test set by evaluating the objective speech quality metrics, DNSMOS, and the recognition accuracy. Hopefully, the proposed method will raise EMS efficiency and security against non-target speech.

Decoupling-style monaural speech enhancement with a triple-branch cross-domain fusion network

Monaural speech enhancement aims to remove background noise from noisy speech signals captured by a single microphone. In recent years, several cross-domain monaural speech enhancement methods are developed to leverage both waveform and harmonic information. However, these methods fall short in fully capturing the dependencies between the time domain and time-frequency (T-F) domain, as well as in harnessing the benefits of the target decoupling strategy. This paper proposes a causal encoder-decoder-based Triple-branch Cross-domain Fusion Network (TCF-Net), which effectively processes speeches by leveraging both time domain and T-F domain features. The proposed approach enables the parallel recovery of magnitude and phase information to alleviate the compensation problem between them. TCF-Net forms a triple-branch network by collaboratively reconstructing the enhanced spectrum with a complex spectrum branch and a magnitude spectrum branch, while incorporating time-domain information with a waveform compensation branch. To fully leverage the information from three domains, Triple-domain Fusion Modules (TFMs) are inserted in each intermediate layer of the model to extract and merge the information from two T-F domain branches and one time domain branch. The TFMs generate masks to progressively compensate for the magnitude of the two T-F domain branches and promote information interaction, further restoring the magnitude of the clean speech. Experimental results demonstrate that TCF-Net outperforms state-of-the-art (SOTA) cross-domain methods and target decoupling methods under causal configuration in all evaluation metrics, which validates the importance of the proposed cross-domain information fusion strategy and target decoupling strategy.

A Two-Stage Deep Representation Learning-Based Speech Enhancement Method Using Variational Autoencoder and Adversarial Training

A Two-Stage Deep Representation Learning-Based Speech Enhancement Method Using Variational Autoencoder and Adversarial Training

A speech enhancement method combining beamforming with RNN for hearing aids

Speech enhancement is essential for hearing aids. In recent years, many speech enhancement methods based on deep learning have been proven to be effective. However, these speech enhancement methods rarely consider limited hardware resources and have difficulty meeting real-time requirements, which is very important for hearing aids. To solve the above problems, we propose a method that combines beamforming and speech enhancement methods based on deep learning. Beamforming is used to filter background noise and reduce the complexity of noise. Additionally, a new filter bank used in hearing aids is adopted to reduce the complexity of the system. The system was deployed and tested in resource-constrained hearing aids. The effectiveness of the method was verified by objective experiments using standard evaluation indicators. The results showed that the power was 8.43 mA, the signal-to-noise ratio improved by 9.4394 dB, and the PESQ improved by 0.7350. The presented objective and subjective results show that the proposed method achieves better noise suppression than previous methods.

Robust Spoofed Speech Detection with Denoised I-vectors

Spoofed speech detection is recently gaining attention of the researchers as speaker verification is shown to be vulnerable to spoofing attacks such as voice conversion, speech synthesis, replay, and impersonation. Although various different methods have been proposed to detect spoofed speech, their performances decrease dramatically under the mismatched conditions due to the additive or reverberant noises. Conventional speech enhancement methods fail to recover the performance gap, hence more advanced techniques seem to be necessary to solve the noisy spoofed speech detection problem. In this work, Denoising Autoencoder (DAE) is used to obtain clean estimates of i-vectors from their noisy versions. ASVspoof 2015 database is used in the experiments with five different noise types, added to the original utterances at 0, 10, and 20 dB signal-to-noise ratios (SNR). The experimental results verified that the DAE provides a more robust spoof detection, where the conventional methods fail.