Evaluation Of Speech Quality Research Articles

Refining the evaluation of speech synthesis: A summary of the Blizzard Challenge 2023

The Blizzard Challenge has benchmarked progress in Text-to-Speech (TTS) since 2005. The Challenge has seen important milestones passed, with results suggesting that synthetic speech was indistinguishable from natural speech in terms of intelligibility in 2021 and that by that same year it was perhaps even indistinguishable in naturalness. The high quality of synthetic speech generated by the latest TTS systems has thus revealed limitations with ITU-T P.800.1 Mean Opinion Score (MOS) in detecting the remaining differences between synthetic and natural speech. Yet, it was the only method used in previous Challenges and is still the most popular method in the field for speech synthesis evaluation. In the 2023 Challenge, we addressed observed limitations of past Challenges by incorporating state-of-the-art speech synthesis evaluation techniques to refine the evaluation of speech quality, speaker similarity and intelligibility. For speech quality, a relative comparison of the systems receiving the best MOS was able to discover a greater number of significant differences between systems. Regarding speaker similarity, we demonstrated that there is a strong bias depending on whether the listeners are familiar with the target voice or not. As for intelligibility, the evaluation of language-specific phenomena, such as the pronunciation of homographs, better highlighted system limits compared to global transcription tasks of synthesised utterances. In addition to reporting results for the 18 entries to the 2023 Challenge, we extend the results analysis to type of TTS module to provide some insights on the most recent advances in model design. Overall, this year’s results demonstrate the need for a shift towards new methods for refining TTS evaluation to shed light on increasingly smaller and localised differences between synthesised and natural speech.

Enhanced Speech Processing for Air Traffic Control Using an Optimized Generative Adversarial Network

In order to improve the quality of voice calls during air traffic control, an improved SEGAN air traffic control speech enhancement algorithm is proposed. Aiming at the problem that the traditional speech enhancement algorithm based on generative adversarial network (SEGAN) is drowned under low signal-to-noise ratio conditions, a multi-stage, multi-mapping, multi-dimensional output generator and a multi-scale, multi-discriminator network model are proposed based on the SEGAN network model. First, the speech semantic features are extracted based on the deep neural network structure to complete the semantic segmentation of air traffic control speech. Secondly, multiple generators are set to further optimize the speech signal. Then, a downsampling module is added to the convolution layer to improve the model's utilization of speech information and reduce the loss of speech information. Finally, multi-scale, multiple discriminators are used to learn the distribution law and information of speech samples in multiple directions. The results show that under low signal-to-noise ratio conditions, the improved SEGAN model improves the short-term objective intelligibility (STOI) and the perceptual evaluation of speech quality (PESQ) by and respectively , which can quickly and effectively perform air traffic control speech enhancement and provide preparation for subsequent air traffic control speech recognition.

Impact of Mask Type as Training Target for Speech Intelligibility and Quality in Cochlear-Implant Noise Reduction

The selection of a target when training deep neural networks for speech enhancement is an important consideration. Different masks have been shown to exhibit different performance characteristics depending on the application and the conditions. This paper presents a comprehensive comparison of several different masks for noise reduction in cochlear implants. The study incorporated three well-known masks, namely the Ideal Binary Mask (IBM), Ideal Ratio Mask (IRM) and the Fast Fourier Transform Mask (FFTM), as well as two newly proposed masks, based on existing masks, called the Quantized Mask (QM) and the Phase-Sensitive plus Ideal Ratio Mask (PSM+). These five masks are used to train networks to estimate masks for the purpose of separating speech from noisy mixtures. A vocoder was used to simulate the behavior of a cochlear implant. Short-time Objective Intelligibility (STOI) and Perceptual Evaluation of Speech Quality (PESQ) scores indicate that the two new masks proposed in this study (QM and PSM+) perform best for normal speech intelligibility and quality in the presence of stationary and non-stationary noise over a range of signal-to-noise ratios (SNRs). The Normalized Covariance Measure (NCM) and similarity scores indicate that they also perform best for speech intelligibility/gauging the similarity of vocoded speech. The Quantized Mask performs better than the Ideal Binary Mask due to its better resolution as it approximates the Wiener Gain Function. The PSM+ performs better than the three existing benchmark masks (IBM, IRM, and FFTM) as it incorporates both magnitude and phase information.

Realization of global audio telepresence via a learning-based model-matching approach with an acoustic array system

A Global Audio Telepresence (GOAT) system requires a microphone array to capture the spatial audio signals in the far end and a loudspeaker array to reconstruct the sound field in the near end. This seamlessly immerses near-end users in remote audio scenes with full ambience. In this paper, we use a learning-based GOAT system (L-GOAT) based on the model-matching principle, where a deep neural network (DNN) acts as non-linear filters for the GOAT system. The network training attempts to minimize the matching error between the signals reproduced by the DNN and the desired signals filtered by the far-end acoustic transfer functions (ATFs). Extensive simulations were carried out for multi-source scenarios in two different rooms with different reverberation times. To implement the L-GOAT system, a five-microphone linear array was adopted in the far-end room, while a six-loudspeaker array was utilized in the near-end room. The objective evaluation matrices, including the Perceptual Evaluation of Speech Quality (PESQ), Short-Time Objective Intelligibility (STOI), and the matching errors, were conducted to validate the efficacy of the GOAT systems. The proposed learning-based approach has demonstrated superior performance compared to a conventional digital signal processing (DSP)-based method.

Effect of Modulation Domain Coupled KalmanSpectral Filter on Speech Enhancement over Wireless Voiced Communication System

Noise suppression in speech signal has received great attention in signal processing research community. However, existing methods of noise suppression such as the time-domain Kalman filter suffers from the psychoacoustic and physiological characteristic feature while the spectral subtraction technique performed in the modulation domain is associated with remnant noise in the enhanced speech signal. Therefore, in this paper, a Modulation-Domain coupled Kalman-Spectral Filtering (MD-KSF) in a single channel system is proposed to address the aforementioned shortcomings in the two existing techniques. The simulation of the proposed coupled MD-KSF was performed in MATLAB software environment. The evaluation of the proposed coupled MD-KSF technique in terms of Spectral waveform, Mean Square Error (MSE) and Log Spectral Distance (LSD) was performed. Furthermore, Perceptual Evaluation of Speech Quality (PESQ) and Short-Time Objective Intelligibility (STOI) tests were employed to validate the quality and intelligibility of the proposed coupled MD- KSF using the NOIZEUS corpus data set. The proposed technique shows significant noise suppression over the existing techniques.

Comparative Evaluation of Speech Quality Before and After Functional Palatal Recontouring in Complete Dentures using Artificial Intelligence: A Study Protocol.

Introduction In the realm of dentistry, where precision and understanding of oral structures are paramount, the utilization of advanced technologies has revolutionized diagnostic and treatment methodologies. One such technology, palatography, holds significant importance in understanding the intricate dynamics of the oral cavity, particularly the interaction between the tongue and the palate during speech and swallowing. In the field of prosthodontics, achieving a successful outcome goes beyond simply restoring missing teeth. A well-designed denture or prosthesis should not only function properly but also consider factors like aesthetics, comfort, and speech intelligibility. Methods The speech quality analysis using artificial intelligence has been designed to assist in the palatal recontouring of complete dentures during the trial denture appointment and denture insertion. The study will be conducted on 12 completely edentulous patients, selected based on exclusion and inclusion criteria. Expected results The palatal recontouring assessment is expected to improve speech quality significantly. This could be incorporated into routine complete denture rehabilitation cases where the ease of technology can help enhance the phonetics of the prosthesis. Conclusions Patients undergoing full denture rehabilitation benefit greatly from phonetics. Taking this key factor into play through the use of advanced but readily available technology, palatography can be employed as a routine procedure for rehabilitation through complete dentures. There is definitely a learning curve involved while making it a routine procedure, but it is, nonetheless, a significant addition to holistic rehabilitation.

Noise Reduction Using Sparsity Constrained and Regularized Iterative Thresholding Algorithm and Dictionary

Dictionary-based methods are recognized for their ability to estimate clear speech from speech contaminated with noise. However, these techniques face a limitation in distinguishing between speech and noise components. The presented algorithm addresses this challenge by introducing an innovative solution through an iterative thresholding method. In this method, the thresholding is based on noise characteristics and remains independent of variations in noise power. To determine the stopping criteria for thresholding, the algorithm leverages the structure of the speech signal in time–frequency domains using the Gini Index. Remarkably, this technique adeptly extracts reliable signal components from a noisy time–frequency magnitude spectrum, performing well under non-varying and varying Signal-to-Noise Ratio (SNR) conditions. Importantly, it achieves this without the need for mask functions, voice activity detection techniques, noise or mixture dictionaries, or SNR information, which are essential components in other dictionary-based methods. Following the thresholding process, the clean speech is assessed using a dictionary-based approach to restore perceptual loss. In evaluating its performance, the algorithm is compared with traditional enhancement techniques concerning perceptual evaluation of speech quality and short-time objective intelligibility. The assessment is conducted under various background noise conditions, including babble, white, factory, and Volvo noises. The proposed algorithm exhibits superior performance, particularly in low and varying SNR scenarios.

In-Car Speech Enhancement Based on Source Separation Technique

Purpose: The purpose of this study was to investigate and analyze to increase the quality and intelligibility of speech in cars. The passenger dialogue inside the car, the sound of other equipment, and a wide range of interference effects are major challenges in the task of speech separation in-car environment.Methods: Speech enhancement based on the source separation algorithm has been proposed to enhance the preferred speech signals using a microphone array inside a car. The proposed approach determines the signal direction in the time domain by utilizing the time difference of arrival (TDOA). TDOA signals are processed, and an adaptive least mean square method is used to determine the enhanced preferred signal.Results: Experimental results show that the proposed approach yields an signal-to-noise ratio (SNR) of 7.4, perceptual evaluation of speech quality (PESQ) of 2.33 respectively. The proposed strategy outperforms existing methods in terms of PESQ, and SNR. The PESQ of the proposed method is 2.45%, and 5.22% better than the existing Independent Component Analyses, and Run length Source techniques.Conclusion: Finally, compared to the existing methods, the suggested speech enhancement algorithm is more reliable and flexible, and it can properly identify the exact location of the sound.

Air Traffic Control Speech Enhancement Method Based on Improved DNN-IRM

The quality of air traffic control speech is crucial. However, internal and external noise can impact air traffic control speech quality. Clear speech instructions and feedback help optimize flight processes and responses to emergencies. The traditional speech enhancement method based on a deep neural network and ideal ratio mask (DNN-IRM) is prone to distortion of the target speech in a strong noise environment. This paper introduces an air traffic control speech enhancement method based on an improved DNN-IRM. It employs LeakyReLU as an activation function to alleviate the gradient vanishing problem, improves the DNN network structure to enhance the IRM estimation capability, and adjusts the IRM weights to reduce noise interference in the target speech. The experimental results show that, compared with other methods, this method improves the perceptual evaluation of speech quality (PESQ), short-term objective intelligibility (STOI), scale-invariant signal-to-noise ratio (SI-SNR), and speech spectrogram clarity. In addition, we use this method to enhance real air traffic control speech, and the speech quality is also improved.

Synthesizing Lithuanian voice replacement for laryngeal cancer patients with Pareto-optimized flow-based generative synthesis network

This study presents a Pareto optimized flow-based generative network for speech synthesis - the P-GLOW model in Lithuanian speech synthesis for substituting original voices affected by cancer-related pathologies. Comparing this pure Lithuanian model with an English model trained on Lithuanian phonemes, the study emphasizes the impact of language-specific characteristics on performance, illustrating the advantages of tailored, language-specific models for speech substitution for impaired persons. Our methodology integrates generative models with an architecture based on 12 coupling layers, 1x1 invertible convolutions, and dilated convolutions with gated hyperbolic tangent activation functions. This design facilitates the effective transformation of waveform inputs into Mel spectrograms, enabling the synthesis of high-quality speech audio. The proposed, Pareto optimized, native Lithuanian model achieved a MOS of 4.2 ± 0.1 and an SMOS of 3.9 ± 0.2, while the baseline model (English language model trained on top with Lithuanian phonemes) scored lower with a MOS of 3.8 ± 0.2 and an SMOS of 3.4 ± 0.3. In terms of the Mel Cepstral Distortion (MCD), Voiced/Unvoiced Decision Error (VDE), Glottal-to-Noise Excitation Ratio (GPE), and Frame Fidelity Error (FFE) scores, the proposed method achieved an MCD of 3.2 dB ± 0.2, a VDE of 5.6% ± 1.0, a GPE of 7.9% ± 1.4, and an FFE of 9.7% ± 2.0, outperforming the baseline method which scored higher with an MCD of 4.6 dB ± 0.3, a VDE of 10.8% ± 1.9, a GPE of 14.5% ± 2.4, and an FFE of 18.7% ± 2.8. Log Likelihood Ratio (LLR), Weighted Spectral Slope (WSS), and Perceptual Evaluation of Speech Quality (PESQ) scores also showed strong improvement. The proposed method achieved an LLR of 0.5 ± 0.05, a WSS of 0.3 ± 0.03, and a PESQ of 4.2 ± 0.1, while the baseline method scored lower with an LLR of 0.8 ± 0.08, a WSS of 0.5 ± 0.05, and a PESQ of 3.6 ± 0.2. The Speech Intelligibility Index (SII) and Short-Time Objective Intelligibility (STOI) metrics for restored alaryngeal speech were also high. The proposed method achieved an SII of 0.75 ± 0.05 and an STOI of 0.85 ± 0.03, while the baseline method scored lower with an SII of 0.65 ± 0.05 and an STOI of 0.75 ± 0.04. The evaluation results demonstrate that our approach was able to effectively synthesize high-quality Lithuanian speech audio restoring the input signal from distorted alaryngeal speech.

Research on Oral English Learning System Integrating AI Speech Data Recognition and Speech Quality Evaluation Algorithm

The application of AI speech data recognition (SDR) and speech quality assessment algorithms in spoken English learning systems can help to realize the intelligence and automation of the process related to spoken English learning. To better assist the people concerned with the learning of spoken English, it is necessary to integrate AI speech data recognition technology in the process of learning spoken English to bring learners a learning environment and learning conditions that are not bound by time and space. As an important way to realize the natural communication of information between machines and people, AI speech recognition technology has significant application advantages and application values in eliminating the influence of the native language of English speaking learners and innovating the learning mode in the local area. In addition, the assessment of the reliability of spoken English language learning can be carried out in several dimensions. However, it is still difficult to unify the criteria of many sizes, bringing new application scenarios for AI SDR. Based on this, this paper first analyzes the theoretical basis of AI SDR. Then it investigates the speech data recognition method of the fused AI English-speaking learning mechanism and the process of fused AI English-speaking speech quality assessment. Finally, it designs and verifies the functions of the spoken English learning mechanism incorporating AI speech data identification and quality assessment.

Imperceptible and Reversible Acoustic Watermarking Based on Modified Integer Discrete Cosine Transform Coefficient Expansion

This paper aims to explore an alternative reversible digital watermarking solution to guarantee the integrity of and detect tampering with data of probative importance. Since the payload for verification is embedded in the contents, algorithms for reversible embedding and extraction, imperceptibility, payload capacity, and computational time are issues to evaluate. Thus, we propose a reversible and imperceptible audio information-hiding algorithm based on modified integer discrete cosine transform (intDCT) coefficient expansion. In this work, the original signal is segmented into fixed-length frames, and then intDCT is applied to each frame to transform signals from the time domain into integer DCT coefficients. Expansion is applied to DCT coefficients at a higher frequency to reserve hiding capacity. Objective evaluation of speech quality is conducted using listening quality objective mean opinion (MOS-LQO) and the segmental signal-to-noise ratio (segSNR). The audio quality of different frame lengths and capacities is evaluated. Averages of 4.41 for MOS-LQO and 23.314 [dB] for segSNR for 112 ITU-T test signals were obtained with a capacity of 8000 bps, which assured imperceptibility with the sufficient capacity of the proposed method. This shows comparable audio quality to conventional work based on Linear Predictive Coding (LPC) regarding MOS-LQO. However, all segSNR scores of the proposed method have comparable or better performance in the time domain. Additionally, comparing histograms of the normalized maximum absolute value of stego data shows a lower possibility of overflow than the LPC method. A computational cost, including hiding and transforming, is an average of 4.884 s to process a 10 s audio clip. Blind tampering detection without the original data is achieved by the proposed embedding and extraction method.

Automatic bioacoustics noise reduction method based on a deep feature loss network

Acoustic sensors that collect acoustic data over extended periods and broad ranges are widely used in bioacoustics monitoring. However, in open environments, acoustic data collected using acoustic sensors can be subject to interference from various real-world noises, thereby influencing the subsequent analysis and processing of bioacoustic data. Existing bioacoustic noise reduction methods are limited in their application because of their low efficiency, unsuitability for non-stationary noise, generally unimproved signal-to-noise ratio (SNR) efficacy, and considerable amounts of residual noise. These limitations hinder the effective processing of recorded signals for which extraneous noise overlaps with bird vocalizations. In this study, we propose a bioacoustic noise reduction method based on a deep feature loss network for bird sounds. The method has a rapid denoising speed and can more effectively remove background noise from field recording signals without distorting the bird acoustic spectrum. The denoising effects of the proposed method were compared with those of a speech enhancement generative adversarial network, web real-time communications denoising, and other noise reduction methods. The denoising ability of these methods for different noises was evaluated using spectrograms and objective evaluation measures such as the SNR and perceptual evaluation of speech quality (PESQ). The experimental results revealed that our proposed noise reduction method can obtain higher SNRs and PESQ scores than other noise reduction methods, with the SNR increasing by up to 35.83 dB following denoising.

On intrusive speech quality measures and a global SNR based metric

Measuring the quality of noisy speech signals has been an increasingly important problem in the field of speech processing as more and more speech-communication and human-machine-interface systems are deployed in practical applications. In this paper, we study four widely used classical performance measures: signal-to-distortion ratio (SDR), short-time objective intelligibility (STOI), signal-to-noise ratio (SNR), and perceptual evaluation of speech quality (PESQ). Through analyzing these performance measures under the same framework and identifying the relationship between their core parameters, we convert these measures into the corresponding equivalent SNRs. This conversion enables not only some new insights into different quality measures but also a way to combine these measures into a new metric. In the derivation of the equivalent SNRs, we introduce the widely used masking technique into the computation of correlation coefficients, which is subsequently used to analyze STOI. Furthermore, we propose an attention method to compute the core parameters of PESQ, and also an empirical formula to project the equivalent SNRs into PESQ scores. Experiments are carried out and the results justifies the properties of the derived quality measures.

Analysis of statistical estimators and neural network approaches for speech enhancement

Speech communicated is adversely affected by environmental noise. It is important to process the speech and reduce noise for better understanding. Speech enhancement or noise reduction is useful to provide comfort for human or machine listening. Traditional algorithms provide better noise reduction and better-quality speech. Due to the non-stationary nature of noise and the quasi-stationary nature of speech, the traditional methods are proven inadequate in achieving high-quality speech. Later statistical estimators based on Gaussian, and super-Gaussian Probability Density Function (PDF) assumption further improved the enhancement performance. But still, non-stationary noise nature introduces artifacts in processed signal and results in decreased performance. It is observed that neural network approaches and the factorization approach provide better performance even under non-stationary noises by proper training and large database. Different features result in variations in output performance under unseen noise and speaker conditions. It is important to understand the importance and advantages of traditional methods, statistical estimators, and neural network approaches performances. To select the suitable method for a required application, it is essential to consider the trade-off between quality and distortion. In this work, the importance of speech enhancement methods is discussed. Performance measures used for understanding the speech enhancement like Signal to Noise Ratio (SNR), Segmental SNR, Log-Likelihood Ratio (LLR), Weighted Spectral Slope (WSS), Perceptual Evaluation of Speech Quality (PESQ), Short-Time Objective Intelligibility (STOI), Signal to Distortion Ratio (SDR) and Mean Opinion Score (MOS), are given. Highlights of important results are discussed for analyzing better speech enhancement methods for the required application. In this work, performance is compared using objective and subjective performance measures. Simulation results show superior performance when neural network is employed in statistical estimators.

Deep causal speech enhancement and recognition using efficient long-short term memory Recurrent Neural Network.

Long short-term memory (LSTM) has been effectively used to represent sequential data in recent years. However, LSTM still struggles with capturing the long-term temporal dependencies. In this paper, we propose an hourglass-shaped LSTM that is able to capture long-term temporal correlations by reducing the feature resolutions without data loss. We have used skip connections in non-adjacent layers to avoid gradient decay. In addition, an attention process is incorporated into skip connections to emphasize the essential spectral features and spectral regions. The proposed LSTM model is applied to speech enhancement and recognition applications. The proposed LSTM model uses no future information, resulting in a causal system suitable for real-time processing. The combined spectral feature sets are used to train the LSTM model for improved performance. Using the proposed model, the ideal ratio mask (IRM) is estimated as a training objective. The experimental evaluations using short-time objective intelligibility (STOI) and perceptual evaluation of speech quality (PESQ) have demonstrated that the proposed model with robust feature representation obtained higher speech intelligibility and perceptual quality. With the TIMIT, LibriSpeech, and VoiceBank datasets, the proposed model improved STOI by 16.21%, 16.41%, and 18.33% over noisy speech, whereas PESQ is improved by 31.1%, 32.9%, and 32%. In seen and unseen noisy situations, the proposed model outperformed existing deep neural networks (DNNs), including baseline LSTM, feedforward neural network (FDNN), convolutional neural network (CNN), and generative adversarial network (GAN). With the Kaldi toolkit for automated speech recognition (ASR), the proposed model significantly reduced the word error rates (WERs) and reached an average WER of 15.13% in noisy backgrounds.

Non-Intrusive Speech Quality Assessment Based on Deep Neural Networks for Speech Communication.

Traditionally, speech quality evaluation relies on subjective assessments or intrusive methods that require reference signals or additional equipment. However, over recent years, non-intrusive speech quality assessment has emerged as a promising alternative, capturing much attention from researchers and industry professionals. This article presents a deep learning-based method that exploits large-scale intrusive simulated data to improve the accuracy and generalization of non-intrusive methods. The major contributions of this article are as follows. First, it presents a data simulation method, which generates degraded speech signals and labels their speech quality with the perceptual objective listening quality assessment (POLQA). The generated data is proven to be useful for pretraining the deep learning models. Second, it proposes to apply an adversarial speaker classifier to reduce the impact of speaker-dependent information on speech quality evaluation. Third, an autoencoder-based deep learning scheme is proposed following the principle of representation learning and adversarial training (AT) methods, which is able to transfer the knowledge learned from a large amount of simulated speech data labeled by POLQA. With the help of discriminative representations extracted from the autoencoder, the prediction model can be trained well on a relatively small amount of speech data labeled through subjective listening tests. Fourth, an end-to-end speech quality evaluation neural network is developed, which takes magnitude and phase spectral features as its inputs. This phase-aware model is more accurate than the model using only the magnitude spectral features. A large number of experiments are carried out with three datasets: one simulated with labels obtained using POLQA and two recorded with labels obtained using subjective listening tests. The results show that the presented phase-aware method improves the performance of the baseline model and the proposed model with latent representations extracted from the adversarial autoencoder (AAE) outperforms the state-of-the-art objective quality assessment methods, reducing the root mean square error (RMSE) by 10.5% and 12.2% on the Beijing Institute of Technology (BIT) dataset and Tencent Corpus, respectively. The code and supplementary materials are available at https://github.com/liushenme/AAE-SQA.

High-capacity steganography through audio fusion and fission

<span>Information security is required for two reasons, either to conceal the information completely or to prevent the misuse of the information by adding watermarks or metadata. Audio steganography uses audio signals to hide secret information. In the proposed audio steganography technique, cover audio files and secret audio files are transformed from time domain to wavelet domain using discrete wavelet transform, the secret audio file is transformed in two levels, leading to secure and high-capacity data hiding. 1% of the 2-level compressed secret is fused to 99% of the 1-level compressed cover. “Peak signal to noise ratio and mean squared error, Pearson’s correlation coefficient, spearman’s correlation coefficient, perceptual evaluation of speech quality and short-time objective intelligibility” are considered to assess the similarity of cover audio and stego audio and similarity of secret audio embedded, and secret audio retrieved. Results show that the stego audio signal is perceptually indistinguishable from the cover audio signal. The approach also passed the robustness test.</span>

Dynamic acoustic compensation and adaptive focal training for personalized speech enhancement

Recently, more and more personalized speech enhancement (PSE) systems with excellent performance have been proposed. Compared with traditional speech enhancement systems, PSE systems have a wider range of application scenarios, which can simultaneously remove background noise and interfering speaker's speech. However, two issues still limit the performance and generalization ability of the model: 1) Acoustic environment mismatch between the test noisy speech and clean enrollment speech of target speaker, which limits the performance of personalized speech enhancement system; 2) Hard sample mining and learning. How to improve the performance of hard samples determines the practicality of a personalized speech enhancement system in complex real-world scenarios. In this paper, a dynamic acoustic compensation (DAC) is proposed to alleviate the environment mismatch, by intercepting the acoustic segments from noisy speech and mixing it with enrollment speech. To well exploit the hard samples, we propose an adaptive focal training (AFT) strategy by assigning adaptive loss weights to hard and non-hard samples during training. Both the DAC and AFT are proposed to improve and generalize our previous work, a densely-connected pyramid complex convolutional network with speaker encoder (sDPCCN) for personalized speech enhancement. In addition, a time-frequency multi-loss training is further introduced to enhance the improved sDPCCN. To examine the effectiveness of the proposed methods, we generate the noisy-reverb training and test data by utilizing non-overlapping segments of the 4th Deep Noise Suppression (DNS4) Challenge Dataset. Results show that, DAC effectively alleviates the acoustic environment mismatch and brings large improvements in terms of multiple evaluation metrics, and AFT reduces the hard sample rate significantly. When all proposed methods are applied, the perceptual evaluation of speech quality (PESQ) score is improved from 3.21 to 3.36, and the scale invariant signal-to-noise ratio (SISNR) is improved from 15.11 to 15.89 on the test set, which fully verify their effectiveness and practicality.

SASEGAN-TCN: Speech enhancement algorithm based on self-attention generative adversarial network and temporal convolutional network.

Traditional unsupervised speech enhancement models often have problems such as non-aggregation of input feature information, which will introduce additional noise during training, thereby reducing the quality of the speech signal. In order to solve the above problems, this paper analyzed the impact of problems such as non-aggregation of input speech feature information on its performance. Moreover, this article introduced a temporal convolutional neural network and proposed a SASEGAN-TCN speech enhancement model, which captured local features information and aggregated global feature information to improve model effect and training stability. The simulation experiment results showed that the model can achieve 2.1636 and 92.78% in perceptual evaluation of speech quality (PESQ) score and short-time objective intelligibility (STOI) on the Valentini dataset, and can accordingly reach 1.8077 and 83.54% on the THCHS30 dataset. In addition, this article used the enhanced speech data for the acoustic model to verify the recognition accuracy. The speech recognition error rate was reduced by 17.4%, which was a significant improvement compared to the baseline model experimental results.