Perceptual Speech Quality Research Articles

Speech Quality Perception in Unilateral Cochlear Implant Users With Single-Sided Deafness.

Cochlear implant (CI) users frequently complain about speech quality perception (SQP). In patients undergoing cochlear implantation for single-sided deafness, there is concern that poor SQP from the implanted ear will negatively impact binaural (CI + normal hearing [NH]) SQP. In this study, we investigate if binaural SQP is measurably different than unimplanted NH alone. Cross-sectional study. Tertiary care center. Fifteen unilateral CI users with NH in the contralateral ear completed the validated Columbia Speech Quality Instrument. This instrument consists of 9 audio clips rated across 14 specific speech qualities using a 10-point visual analog scale. SQP was assessed in 3 conditions: CI only, NH only, and CI + NH. Median speech quality scores were worse in the CI only condition compared to the NH only (50.0 vs 72.6, P = .0003) and binaural (50.0 vs 71.0, P = .007) conditions. Median speech quality scores were not significantly different between the NH only and binaural conditions (72.6 vs 71, P = .8). Compared to NH, CI speech quality sounded less clear, less natural, and more mechanical. Compared to NH, SQP is poorer with a CI alone. However, in contrast to expectation, there is no significant difference between NH and binaural SQP. This suggests poorer CI speech perception does not negatively impact binaural SQP in patients undergoing cochlear implantation for single-sided deafness.

From the perspective of perceptual speech quality: The robustness of frequency bands to noise.

Speech quality is one of the main foci of speech-related research, where it is frequently studied with speech intelligibility, another essential measurement. Band-level perceptual speech intelligibility, however, has been studied frequently, whereas speech quality has not been thoroughly analyzed. In this paper, a Multiple Stimuli With Hidden Reference and Anchor (MUSHRA) inspired approach was proposed to study the individual robustness of frequency bands to noise with perceptual speech quality as the measure. Speech signals were filtered into thirty-two frequency bands with compromising real-world noise employed at different signal-to-noise ratios. Robustness to noise indices of individual frequency bands was calculated based on the human-rated perceptual quality scores assigned to the reconstructed noisy speech signals. Trends in the results suggest the mid-frequency region appeared less robust to noise in terms of perceptual speech quality. These findings suggest future research aiming at improving speech quality should pay more attention to the mid-frequency region of the speech signals accordingly.

Speech Enhancement: A Review of Different Deep Learning Methods

Speech enhancement methods differ depending on the degree of degradation and noise in the speech signal, so research in the field is still difficult, especially when dealing with residual and background noise, which is highly transient. Numerous deep learning networks have been developed that provide promising results for improving the perceptual quality and intelligibility of noisy speech. Innovation and research in speech enhancement have been opened up by the power of deep learning techniques with implications across a wide range of real time applications. By reviewing the important datasets, feature extraction methods, deep learning models, training algorithms and evaluation metrics for speech enhancement, this paper provides a comprehensive overview. We begin by tracing the evolution of speech enhancement research, from early approaches to recent advances in deep learning architectures. By analyzing and comparing the approaches to solving speech enhancement challenges, we categorize them according to their strengths and weaknesses. Moreover, we discuss the challenges and future directions of deep learning in speech enhancement, including the demand for parameter-efficient models for speech enhancement. The purpose of this paper is to examine the development of the field, compare and contrast different approaches, and highlight future directions as well as challenges for further research.

Lip landmark-based audio-visual speech enhancement with multimodal feature fusion network

Traditional speech enhancement uses only audio signals to obtain clean speech by removing background noise. Audio-visual speech enhancement uses additional visual information to improve the intelligibility and perceptual quality of noisy speech, which can be applied to video conferencing. However, the original visual cues contain redundant information, which causes high latency and makes the model prone to overfitting. Meanwhile, related methods do not pay much attention to exploiting critical visual features and fusing multimodal audio-visual features. In this paper, we propose an efficient multimodal feature fusion network (MFF-Net) for audio-visual speech enhancement. Specifically, we are the first to use fine-grained 3D lip landmarks to represent visual features. This visual representation can provide refined 3D visual information while protecting privacy. We design a multi-scale enhancement module (MEM) with a multi-branch structure that extracts critical multi-scale features by using dilated convolutions and attention mechanisms. Besides, we propose an audio-visual fusion module (AFM) that uses a mutually reinforcing strategy to effectively fuse visual and audio features. To verify the effectiveness of our method, we construct two speech enhancement datasets based on lip landmarks and conduct related experiments. Extensive experimental results show that our proposed MFF-Net has competitive performance compared to other methods.

Speech Enhancement Performance Based on the MANNER Network Using Feature Fusion

The problems that the multi-view attention network for noise erasure (MANNER) cannot take into account are the local and global features in the speech enhancement of long sequences. An attention and feature fusion MANNER (AF-MANNER) network is proposed, which improves the multi-view attention (MA) module in MANNER and replaces the global and local attention in the module. AF-MANNER also designs the feature-weighted fusion module to fuse the features of flash attention and neighborhood attention to enhance the feature expression of the network. The final ablation studies show that this network exhibits a good performance in speech enhancement and that its structure is valuable for improving the intelligibility and perceptual quality of speech.

Differentiable Mean Opinion Score Regularization for Perceptual Speech Enhancement

Many speech enhancement methods require perceptual quality metrics for evaluation. The “holy grail” of perceptual speech quality assessment is human subjective ratings, known as the mean opinion score. However, acquiring human ratings is time-consuming, laborious, and expensive. Existing objective quality metrics, on the other hand, are efficient and easy to compute but do not correlate well with human ratings. In this paper, we propose a relatively lightweight deep-learning-based model to predict the human ratings of speech signals. Since it is differentiable, it can be easily employed as a perceptual regularization to improve existing deep-learning-based speech enhancement methods. Experimental results demonstrate that the predictions of our proposed model correlate well with human judgments. We present application in speech enhancement and show that, interestingly, while there is a degradation in performance in terms of traditional objective metrics, there is a significant improvement in the perceptual quality and the naturalness of the enhanced speech.

U-Shaped Low-Complexity Type-2 Fuzzy LSTM Neural Network for Speech Enhancement

Speech enhancement (SE) aims to improve the intelligibility and perceptual quality of speech contaminated by noise signals through spectral or temporal changes. Deep learning models achieve speech enhancement and estimate the magnitude spectrum. This paper proposes a novel and computationally efficient deep learning model to enhance noisy speech. The model pre-processes the noisy speech magnitude by redistributing energy from high-energy voiced segments to low-energy unvoiced segments using an adaptive power law transformation while maintaining the total energy of the speech signals constant. A U-shaped fuzzy long short-term memory (UFLSTM) estimates the magnitude of a time-frequency (T-F) mask by using the pre-processed data. Residual connections to the similar-shaped layers are added to avoid gradient decay. Attention process is adopted by modifying the forget gate of UFLSTM. To make a causal speech enhancement system, the processing does not include any future audio frames. We compare the proposed speech enhancement to other deep learning models in different noisy environments with signal-to-noise ratios of 0 dB, 5 dB, and 10 dB. The experiments show that the proposed SE system outscores the competing deep learning models and considerably improves speech intelligibility and quality. In terms of STOI and PESQ, the LibriSpeech database improves results by (0.211) 21.1% and (0.95) 36.39%, respectively, over noisy speech in seen noisy conditions, and by (0.199) 19.9% and (0.94) 35.69% over noisy speech in unseen noisy conditions. Further, the cross-corpus analysis shows that proposed SE system performs better when trained with the DNS dataset as compared to the LibriSpeech, VoiceBank, and TIMIT datasets.

Speech enhancement with noise estimation and filtration using deep learning models

Speech enhancement helps in eliminating the environmental noises from the communication signals. The main intention of the augmentation system is to develop the perceptual quality of communication or speech. For this purpose, various filtering schemes, spectral restoration models and speech models were implemented. In order to improve the odds of reducing noise and restoring the original signal, artificial intelligence (AI) and machine learning algorithms (MLA) were included into every sector. Deep transfer learning was used in this work to remove noise from the data and restore the original signals. This proposed approach includes a filtration scheme instead of using a convolution layer in the RESNET-50 architecture. The filters tested for speech enhanced deep learning models are modified Kalman filter and enhanced wiener filter. The performance metrics were calculated between various algorithms and proposed models to identify which approaches to follow the better way result obtained. The performance metrics compared PESQ, LSD and segSNR for different low signal to noise ratio conditions.

Speech Intelligibility evaluation in the presence of speech masker of cochlear implant in a reverberant room

In this report, we investigated the performance of a cochlear implant (CI) in a reverberant room with the presence of a competing speech masker. We compared Short-Time Objective Intelligibility (STOI) and Perceptual Speech Quality (PESQ) to represent the objective, to percent correct word (PCW) and Mean Opinion Score (MOS) as subjective evaluations, respectively. The target and the masker speech were convoluted with 0.4- and 0.7-s reverberation time, respectively, then processed with a 14-channel vocoder. We set the target to the masker 5dB and 0 dB SNR. We simulated two well-known techniques, namely CIS and SPEAK processors. Ten normal-hearing paid subjects were participated to evaluate the performance in clean, reverberant with no masker and reverberant with masker. The results showed that performance degraded significantly as the reverberation duration increased. Furthermore, the presence of a masker to a target speech in a reverberant room worsened all the evaluated performances compared to the reverberant alone. The results suggested that further investigation is required to reduce the masker effect to improve CI intelligibility.

The effect of speech and noise levels on the quality perceived by cochlear implant and normal hearing listeners

Electrical hearing by cochlear implants (CIs) may be fundamentally different from the acoustic hearing by normal-hearing (NH) listeners, presumably showing unequal speech quality perception in various noise environments. Noise reduction (NR) algorithms used in CIs reduce the noise in favor of signal-to-noise ratio (SNR), regardless of plausible accompanying distortions that may degrade the speech quality perception. To gain a better understanding of CI speech quality perception, the present work aimed at investigating speech quality perception in diverse noise conditions, including factors of speech/noise levels, type of noise, and distortions caused by NR models. Fifteen NH and seven CI subjects participated in this study. Speech sentences were set to two different levels (65 and 75 dB SPL (Sound Pressure Level)). Two types of noise (Cafeteria and Babble) at three levels (55, 65, and 75 dB SPL) were used. Sentences were processed using two NR algorithms to investigate the perceptual sensitivity of CI and NH listeners to the distortion. All sentences processed with the combinations of these sets were presented to CI and NH listeners, and they were asked to rate the sound quality of speech as they perceived. The effect of each factor on the perceived speech quality was investigated based on the group-averaged quality rated by CI and NH listeners. Consistent with previous studies, CI listeners were not as sensitive as NH listeners to the distortion made by NR algorithms. Statistical analysis showed that the speech level has a significant effect on quality perception. At the same SNR, the quality of 65 dB speech was rated higher than that of 75 dB for CI users, but vice versa for NH listeners. Therefore, the present study showed that the perceived speech quality patterns were different between CI and NH listeners in terms of their sensitivity to distortion and speech level in a complex listening environment.

Low-complexity artificial noise suppression methods for deep learning-based speech enhancement algorithms

Deep learning-based speech enhancement algorithms have shown their powerful ability in removing both stationary and non-stationary noise components from noisy speech observations. But they often introduce artificial residual noise, especially when the training target does not contain the phase information, e.g., ideal ratio mask, or the clean speech magnitude and its variations. It is well-known that once the power of the residual noise components exceeds the noise masking threshold of the human auditory system, the perceptual speech quality may degrade. One intuitive way is to further suppress the residual noise components by a postprocessing scheme. However, the highly non-stationary nature of this kind of residual noise makes the noise power spectral density (PSD) estimation a challenging problem. To solve this problem, the paper proposes three strategies to estimate the noise PSD frame by frame, and then the residual noise can be removed effectively by applying a gain function based on the decision-directed approach. The objective measurement results show that the proposed postfiltering strategies outperform the conventional postfilter in terms of segmental signal-to-noise ratio (SNR) as well as speech quality improvement. Moreover, the AB subjective listening test shows that the preference percentages of the proposed strategies are over 60%.

Practical cross-layer testing of HARQ-induced delay variation on IP/RTP QoS and VoLTE QoE

As the PHY/MAC-layer IR-HARQ and RLC-layer ARQ error recovery procedures, adopted in LTE, may impose additional delay when their code-block retransmissions occur, the arising question is whether these significantly contribute to IP and consequently RTP packet delays, and finally degrade the overall application-layer end-to-end QoE, especially when voice is transmitted over LTE? With this regard, we propose and demonstrate a VoLTE QoS and QoE test procedure based on PHY/MAC/RLC/IP/TCP-UDP/RTP cross-layer protocol analysis and perceptual speech quality QoE measurements. We identified monotonic relationship between the paired observations: QoE and HARQ RTT, i.e. between the PESQ voice quality rating and the IP/RTP packet latency, for given BLER of the received MAC/RLC code-blocks. Specifically, we found out that, for the HARQ RTT value of about 8 ms, only up to 2 HARQ retransmissions (and consequently no RLC-ARQ one) is appropriate during any voice packet, otherwise delay accumulation might not be accordingly “smoothed out” by jitter/playback buffers along the propagation path.

Assessment of Extreme Communication Environment With Ultralow SNR: A Benchmark

Accurate estimation of subjective assessment plays an essential role in not only speech quality perception, but also communication environment assessment. Traditional speech quality perception is almost always with an environment, in which the content of speech can be heard clearly. Unlike speech quality assessment, in extreme communication environment with ultralow SNR, like short wave channel with active jamming, speech intelligibility is impaired. Under this condition, subjective assessment as absolute category ranking (ACR) whose scores are made by experienced staffs, and many objective measurements cannot handle this situation. In this paper, we propose information damage level (IDL) to replace subjective ACR as subjective assessment. IDL is the average of scores which is marked on the subjective recognition rate. Under extreme communication environment, it can effectively avoid excessive differences from people to people to some extent. We also provide a new dataset collected in an environment with active jamming, whose speech files are recoded under three different environments, named as indoor simulation environment (EN1), outdoor simulation environment (EN2), and outdoor real environment (EN3). We also benchmark a novel open framework on random forest for direct predicting subjective assessment by combining all possible objective measurements. Experiment results prove the effectiveness of our open framework together with our dataset.

Impact of Languages and Accent on Perceived Speech Quality Predicted by Perceptual Evaluation of Speech Quality (PESQ) and Perceptual Objective Listening Quality Assessment (POLQA): Case of Moore, Dioula, French and English

Impact of Languages and Accent on Perceived Speech Quality Predicted by Perceptual Evaluation of Speech Quality (PESQ) and Perceptual Objective Listening Quality Assessment (POLQA): Case of Moore, Dioula, French and English

Non-intrusive speech quality prediction based on the blind estimation of clean speech and the i-vector framework

Output-based instrumental speech quality assessment relies only on the received (processed) signal to predict quality. Such methods are called non-intrusive and are crucial in speech applications where reference clean signals are not accessible. In this paper, we propose a new non-intrusive instrumental quality measure based on the similarity between two i-vectors. As the reference clean signal is not available, the reference i-vector representation cannot be extracted directly from it. Therefore, we propose the use of a clean speech Gaussian mixture model to estimate the clean speech spectra from its degraded speech spectrum counterpart. Next, the two respective i-vector representations are extracted and either the cosine or Eucledian similarity metrics are computed as a correlate of speech quality. Here, the clean speech model is trained using RASTA-filtered mel-frequency cepstral coefficients extracted from a pool of clean speech files, thus allowing us to attain a model of clean spectrum characteristics. The proposed method is evaluated on noisy, reverberant, and enhanced speech conditions. Experimental results show the proposed system providing higher correlations with perceptual speech quality than several benchmark non-intrusive measures, especially for noisy and enhanced speech.

Dialogue act based expressive speech synthesis in limited domain for the Czech language

This paper deals with expressive speech synthesis in a dialogue. Dialogue acts - discrete expressive categories - are used for expressivity description. The aim of the work is to create a procedure for development of expressive speech synthesis for a dialogue system in a limited domain. The domain is here limited to dialogues between a human and a computer on a given topic of reminiscing about personal photographs. To incorporate expressivity into synthetic speech, modifications of current algorithms used for neutral speech synthesis are made. An expressive speech corpus is recorded, annotated using a predefined set of dialogue acts, and its acoustic analysis is performed. Unit selection and HMM-based methods are used to synthesize expressive speech, and an evaluation using listening tests is presented. The listeners asses two basic aspects of synthetic expressive speech for isolated utterances: speech quality and expressivity perception. The evaluation is also performed for utterances in a dialogue to asses appropriateness of synthetic expressive speech. It can be concluded that synthetic expressive speech is rated positively even though it is of worse quality when comparing with the neutral speech synthesis. However, synthetic expressive speech is able to transmit expressivity to listeners and to improve the naturalness of the synthetic speech.

Speech Quality Classifier Model based on DBN that Considers Atmospheric Phenomena

Current implementations of 5G networks consider higher frequency range of operation than previous telecommunication networks, and it is possible to offer higher data rates for different applications. On the other hand, atmospheric phenomena could have a more negative impact on the transmission quality. Thus, the study of the transmitted signal quality at high frequencies is relevant to guaranty the user ́s quality of experience. In this research, the recommendations ITU-R P.838-3 and ITU-R P.676-11 are implemented in a network scenario, which are methodologies to estimate the signal degradations originated by rainfall and atmospheric gases, respectively. Thus, speech signals are encoded by the AMR-WB codec, transmitted and the perceptual speech quality is evaluated using the algorithm described in ITU-T Rec. P.863, mostly known as POLQA. The novelty of this work is to propose a non-intrusive speech quality classifier that considers atmospheric phenomena. This classifier is based on Deep Belief Networks (DBN) that uses Support Vector Machine (SVM) with radial basis function kernel (RBF-SVM) as classifier, to identify five predefined speech quality classes. Experimental Results show that the proposed speech quality classifier reached an accuracy between 92% and 95% for each quality class overcoming the results obtained by the sole non-intrusive standard described in ITU-T Recommendation P.563. Furthermore, subjective tests are carried out to validate the proposed classifier performance, and it reached an accuracy of 94.8%.

On Learning Spectral Masking for Single Channel Speech Enhancement Using Feedforward and Recurrent Neural Networks

Human speech in real-world environments is typically degraded by the background noise. They have a negative impact on perceptual speech quality and intelligibility which causes performance degradation in various speech-related technological applications, such as hearing aids and automatic speech recognition systems. It also degrades the original phase of the clean speech and introduces perceptual disturbance which leads to the negative impacts on the quality of speech. Therefore, speech enhancement must vigilantly be dealt with in everyday listening environments. In this article, speech enhancement is performed using supervised learning of spectral masking. Deep neural networks (DNN) and recurrent neural networks (RNN) are trained to learn the spectral masking from the magnitude spectrograms of the degraded speech. An iterative procedure is adopted as a post-processing step to deal with the noisy phase. Additionally, an intelligibility improvement filter is also used to incorporate the critical band importance function weights where higher weights contribute more towards intelligibility. Systematic experiments demonstrated that the proposed approaches greatly attenuated the background noise. Also, they led to large improvements of the perceived speech quality and intelligibility, as well as automatic speech recognition. In experiments, TIMIT database is used. The STOI is improved by 17.6% over the noisy speech. Also, SDR and PESQ are improved by 5.22dB and 19% over the noisy speech utterances. These comparisons showed that the proposed speech enhancement approaches outperformed the related speech enhancement methods.

Learning Complex Spectral Mapping with Gated Convolutional Recurrent Networks for Monaural Speech Enhancement.

Phase is important for perceptual quality of speech. However, it seems intractable to directly estimate phase spectra through supervised learning due to their lack of spectrotemporal structure in it. Complex spectral mapping aims to estimate the real and imaginary spectrograms of clean speech from those of noisy speech, which simultaneously enhances magnitude and phase responses of speech. Inspired by multi-task learning, we propose a gated convolutional recurrent network (GCRN) for complex spectral mapping, which amounts to a causal system for monaural speech enhancement. Our experimental results suggest that the proposed GCRN substantially outperforms an existing convolutional neural network (CNN) for complex spectral mapping in terms of both objective speech intelligibility and quality. Moreover, the proposed approach yields significantly higher STOI and PESQ than magnitude spectral mapping and complex ratio masking. We also find that complex spectral mapping with the proposed GCRN provides an effective phase estimate.

Identificeerbaarheid van kinderen met een gehoorbeperking

Abstract Studies on the speech and language development of hearing-impaired children often focus on (deviations in) the children’s speech production. However, it is unclear if listeners also perceive differences between the speech of normally hearing and hearing-impaired children. This contribution wants to fill this void by investigating the overall perceived speech quality of both groups. Three groups of listeners (speech and language pathologists, primary school teachers and inexperienced listeners) judged 126 utterances of seven normally hearing children, seven children with an acoustic hearing aid and seven children with a cochlear implant, in a comparative judgment task. All children were approximately seven years old and received, in the case of the hearing-impaired children, their assistive hearing device before the age of two. The online tool D-PAC was used to administer the comparative judgement task. The listeners compared stimuli in pairs and decided which stimulus sounded best. This method ultimately leads to a ranking in which all stimuli are represented according to their overall perceived speech quality. The main result is that the speech of normally hearing children was preferred by the listeners. This indicates that, even after several years of device use, the speech quality of hearing-impaired children is perceived as different from that of normally hearing children. Within the group of hearing-impaired children, cochlear implanted children were judged to exhibit higher speech quality than acoustically hearing aided children, especially after a longer device use. The speech quality of the latter group, on the other hand, remained practically stable. Listeners, irrespectively of their degree of experience with (hearing-impaired) children’s speech, completed the task similarly. In other words: the difference between the overall perceived speech quality of normally hearing and hearing-impaired children is salient for all listener groups and they all slightly preferred children with a cochlear implant over children with an acoustic hearing aid.