Speech Intelligibility Metrics Research Articles

Non-intrusive speech intelligibility prediction using an auditory periphery model with hearing loss

Speech intelligibility prediction methods are necessary for hearing aid development. However, many such prediction methods are categorized as intrusive metrics because they require reference speech as input, which is often unavailable in real-world situations. Additionally, the processing techniques in hearing aids may cause temporal or frequency shifts, which degrade the accuracy of intrusive speech intelligibility metrics. This paper proposes a non-intrusive auditory model for predicting speech intelligibility under hearing loss conditions. The proposed method requires binaural signals from hearing aids and audiograms representing the hearing conditions of hearing-impaired listeners. It also includes additional acoustic features to improve the method's robustness in noisy and reverberant environments. A two-dimensional convolutional neural network with neural decision forests is used to construct a speech intelligibility prediction model. An evaluation conducted with the first Clarity Prediction Challenge dataset shows that the proposed method performs better than the baseline system.

Using a single-channel reference with the MBSTOI binaural intelligibility metric

In order to assess the intelligibility of a target signal in a noisy environment, intrusive speech intelligibility metrics are typically used. They require a clean reference signal to be available which can be difficult to obtain especially for binaural metrics like the modified binaural short time objective intelligibility metric (MBSTOI). We here present a hybrid version of MBSTOI that incorporates a deep learning stage that allows the metric to be computed with only a single-channel clean reference signal. The models presented are trained on simulated data containing target speech, localised noise, diffuse noise, and reverberation. The hybrid output metrics are then compared directly to MBSTOI to assess performances. Results show the performance of our single channel reference vs MBSTOI. The outcome of this work offers a fast and flexible way to generate audio data for machine learning (ML) and highlights the potential for low level implementation of ML into existing tools.

Predicting Speech Intelligibility for People with a Hearing Loss: The Clarity Challenges

Objective speech intelligibility metrics are used to reduce the need for time consuming listening tests. They are used in the design of audio systems; room acoustics and signal processing algorithms. Most published speech intelligibility metrics have been developed using young adults with so-called 'normal hearing', and therefore do not work well for those with different hearing characteristics. One of the most common causes of aural diversity is sensorineural hearing loss. While partially restoring perception through hearing aids is possible, results are mixed. This has led to the Clarity Project, which is running an open series of Enhancement Challenges to improve the processing of speech-in-noise for hearing aids. To enable this, objective metrics of speech intelligibility are needed, which work from signals produced by hearing aids for diverse listeners. For this reason, Clarity is also running Prediction Challenges to improve speech intelligibility models. Competitors are given a set of audio signals produced by hearing aid algorithms, and challenged to predict how many words a listener with a particular hearing characteristic will achieve. Drawing on the learning from the challenge, we will outline what has been learnt about improving intelligibility metrics for those with a hearing impairment.

Joint waveform and magnitude processing for monaural speech enhancement

Speech enhancement is an essential task for improving the quality and intelligibility of speech signals corrupted by noise. Current deep neural network-based speech enhancement methods have achieved remarkable results. However, most of them only work in the time domain or the time–frequency domain, which do not fully use the complementary advantages of the two domains. In this paper, we propose a framework with joint waveform and magnitude processing for single-channel speech enhancement, which can realize the complementary advantages of the time-domain and time–frequency features. Specifically, the proposed network adopts a triple-stage training strategy. In the first two stages, the two sub-networks take waveform and magnitude as input features to generate two pre-enhanced speech signals, respectively. In the third stage, a fusion sub-network is used to fuse the two pre-enhanced speech signals, which further improves the quality and intelligibility of speech. All three sub-networks are encoder-decoder-based architecture with skip connections, and an additional temporal convolutional network is inserted between the encoder and the decoder. In order to improve the information flow of the network, we introduce the gating mechanism into the temporal convolutional network, which we refer to as the gated temporal convolutional network. In addition, the recently popular group communication strategy is also introduced into the network, which significantly reduces the number of trainable parameters and obtains on-par or better enhancement performance. Experimental results demonstrate that the proposed method consistently outperforms other advanced baselines in terms of objective speech quality and intelligibility metrics. Moreover, the proposed model also achieves outstanding cross-corpus and cross-language generalization capabilities.

An intrusive method for estimating speech intelligibility from noisy and distorted signals.

An objective metric that predicts speech intelligibility under different types of noise and distortion would be desirable in voice communication. To date, the majority of studies concerning speech intelligibility metrics have focused on predicting the effects of individual noise or distortion mechanisms. This study proposes an objective metric, the spectrogram orthogonal polynomial measure (SOPM), that attempts to predict speech intelligibility for people with normal hearing under adverse conditions. The SOPM metric is developed by extracting features from the spectrogram using Krawtchouk moments. The metric's performance is evaluated for several types of noise (steady-state and fluctuating noise), distortions (peak clipping, center clipping, and phase jitters), ideal time-frequency segregation, and reverberation conditions both in quiet and noisy environments. High correlation (0.97-0.996) is achieved with the proposed metric when evaluated with subjective scores by normal-hearing subjects under various conditions.

On the Compensation Between Magnitude and Phase in Speech Separation

Deep neural network (DNN) based end-to-end optimization in the complex time-frequency (T-F) domain or time domain has shown considerable potential in monaural speech separation. Many recent studies optimize loss functions defined solely in the time or complex domain, without including a loss on magnitude. Although such loss functions typically produce better scores if the evaluation metrics are objective time-domain metrics, they however produce worse scores on speech quality and intelligibility metrics and usually lead to worse speech recognition performance, compared with including a loss on magnitude. While this phenomenon has been experimentally observed by many studies, it is often not accurately explained and there lacks a thorough understanding on its fundamental cause. This paper provides a novel view from the perspective of the implicit compensation between estimated magnitude and phase. Analytical results based on monaural speech separation and robust automatic speech recognition (ASR) tasks in noisy-reverberant conditions support the validity of our view.

Perceptual and Objective Assessment of Envelope Enhancement for Children With Auditory Processing Disorder.

This paper evaluated the performance of an envelope enhancement (EE) algorithm subjectively by children with auditory processing disorder (APD), and objectively through computational models. Speech intelligibility data was collected from children with APD, for unprocessed and envelope-enhanced speech in the presence of stationary and non-stationary background noise at different signal to noise ratios (SNRs), both with and without noise reduction (NR) algorithms as a front-end to the EE algorithm. Furthermore, intrusive and non-intrusive objective speech intelligibility metrics were derived to predict the perceptual impact of this EE algorithm. Subjective data for stationary noise conditions revealed that the combination of NR and EE algorithms significantly improved the speech intelligibility scores at poor SNRs. In contrast, the same combination was ineffective in improving speech intelligibility in non-stationary noise conditions. Taken together, subjective results suggest that exaggerating the envelope cues improves speech identification scores for children with APD. However, the benefit obtained varies depending upon the type and level of the background noise. Both intrusive and non-intrusive objective speech intelligibility estimators exhibited good correlation with the subjective data, with the intrusive metric demonstrating better generalization capabilities. Implications of these results for hearing aid applications for children with APD is discussed.

Unsupervised Low Latency Speech Enhancement With RT-GCC-NMF

In this paper, we present RT-GCC-NMF: a real-time (RT), two-channel blind speech enhancement algorithm that combines the non-negative matrix factorization (NMF) dictionary learning algorithm with the generalized cross-correlation (GCC) spatial localization method. Using a pre-learned universal NMF dictionary, RT-GCC-NMF operates in a frame-by-frame fashion by associating individual dictionary atoms to target speech or background interference based on their estimated time-delay of arrivals (TDOA). We evaluate RT-GCC-NMF on two-channel mixtures of speech and real-world noise from the Signal Separation and Evaluation Campaign (SiSEC). We demonstrate that this approach generalizes to new speakers, acoustic environments, and recording setups from very little training data, and outperforms all but one of the algorithms from the SiSEC challenge in terms of overall Perceptual Evaluation methods for Audio Source Separation (PEASS) scores and compares favourably to the ideal binary mask baseline. Over a wide range of input SNRs, we show that this approach simultaneously improves the PEASS and signal to noise ratio (SNR)-based Blind Source Separation (BSS) Eval objective quality metrics as well as the short-time objective intelligibility (STOI) and extended STOI (ESTOI) objective speech intelligibility metrics. A flexible, soft masking function in the space of NMF activation coefficients offers real-time control of the trade-off between interference suppression and target speaker fidelity. Finally, we use an asymmetric short-time Fourier transform (STFT) to reduce the inherent algorithmic latency of RT-GCC-NMF from 64 ms to 2 ms with no loss in performance. We demonstrate that latencies within the tolerable range for hearing aids are possible on current hardware platforms.

On the feasibility of using a bispectral measure as a nonintrusive predictor of speech intelligibility

The presence of background noise or nonlinear distortions encountered in real-world situations often reduces the intelligibility of speech signals. Several objective measurements and prediction procedures have been developed to assess speech intelligibility in noise. Most of the existing measures are, however, suitable for only a subset of specified forms of distortion. This study developed a reliable, reference-free speech intelligibility metric that uses the properties of an acoustic signal to predict the effects of a wide range of distortions that influence speech intelligibility in quiet and noisy conditions. The bispectral speech intelligibility metric (BSIM), was developed by extracting the features from the spectrogram of speech signals using the third-order statistics, which are collectively known as the bispectrum. Speech intelligibility scores predicted by the BSIM were compared to behavioral speech intelligibility scores in quiet and noise. The performance of the BSIM was also compared with that of several widely used speech intelligibility metrics. Results showed that the BSIM can successfully predict nonlinear distortions, such as peak-clipping and center-clipping, as well as time domain distortions, such as phase-jitter and reverberation. Unlike existing metrics, such as the articulation index and speech transmission index, the BSIM successfully captured the effect of fluctuating noise on speech intelligibility and predicted the effects of the degradation of noisy speech processed by the ideal time-frequency segregation method. The BSIM presents a reliable, reference-free, and objective measure of speech intelligibility that can provide real-time predictions of the effect of signal processing and acoustics distortion on speech intelligibility in quiet and noise. In addition, the BSIM could be used to analyze algorithms that process noisy speech.

Comparison of the predictive accuracy of different computational models of auditory perception

Computational models of auditory perception offer a time-efficient method of assessing the effects of distortion on speech perception. Several objective metrics have been proposed to predict speech intelligibility, especially when speech is obscured by the presence of background noise. Novel approaches to full-reference and reference-free speech intelligibility metrics have emerged in recent years, but deciphering the best metric for predicting speech intelligibility still requires investigation. This study assessed the predictive accuracy of several reliable, full, and reference-free speech intelligibility metrics. Speech perception scores were measured on listeners with normal hearing and hearing loss in quiet and noise. Acoustic recordings were made of the presented speech stimuli and combined with a computational model of the auditory nerve to simulate behavioral scores using several established metrics such as the STOI, NSIM, SRMR, SII, SNRloss, and BSIM. The estimated speech scores were correlated with behavioral speech recognition scores to assess predictive accuracy of the model simulations. Several of the predicted scores correlated well with behavioral scores. Evaluation of individual phonemes revealed differential sensitivity of the metrics across different phonemic classifications.

A Real-Time Convolutional Neural Network Based Speech Enhancement for Hearing Impaired Listeners Using Smartphone.

This paper presents a Speech Enhancement (SE) technique based on multi-objective learning convolutional neural network to improve the overall quality of speech perceived by Hearing Aid (HA) users. The proposed method is implemented on a smartphone as an application that performs real-time SE. This arrangement works as an assistive tool to HA. A multi-objective learning architecture including primary and secondary features uses a mapping-based convolutional neural network (CNN) model to remove noise from a noisy speech spectrum. The algorithm is computationally fast and has a low processing delay which enables it to operate seamlessly on a smartphone. The steps and the detailed analysis of real-time implementation are discussed. The proposed method is compared with existing conventional and neural network-based SE techniques through speech quality and intelligibility metrics in various noisy speech conditions. The key contribution of this paper includes the realization of CNN SE model on a smartphone processor that works seamlessly with HA. The experimental results demonstrate significant improvements over the state-of-the-art techniques and reflect the usability of the developed SE application in noisy environments.

The Effects of a Choral Singing Intervention on Speech Characteristics in Individuals With Parkinson’s Disease: An Exploratory Study

The purpose of this exploratory study was to examine speech characteristics in individuals with Parkinson’s disease (PD) before and after participation in an 11-week choral singing intervention. Speech characteristics were analyzed using an acoustic measure of vowel space area (VSA) and a standardized speech intelligibility metric. Participants included five males and five females with PD. VSA was derived for the corner vowels (/i/, /u/, /æ/ /α/) produced during a sentence-level task. VSA was expressed as an irregular quadrilateral formed by the first and second formants of these corner vowels. A validated speech intelligibility instrument was administered pre- and posttreatment. VSA significantly increased posttreatment, which suggests greater tongue excursion during vowel articulation. Speech intelligibility scores significantly increased posttreatment, thus reflecting improved speech communication. Results suggest that choral singing may be a viable speech treatment for some individuals with PD. A large-scale randomized controlled trial is warranted.

Critical methodological assessment of the distraction distance used for evaluating room acoustic quality of open-plan offices

ISO 3382-3, the international standard for evaluating the acoustics of open-plan offices, includes metrics based on measuring spatial decay, over workstations, of speech sound pressure level, and speech intelligibility. The speech intelligibility metrics, out of which distraction distance (rD) has recently been shown to be most useful, are derived from speech transmission index (STI) calculations. This study examines how the ISO 3382-3 metrics, especially rD, are affected by certain aspects of STI calculation, by using acoustic measurements of 20 offices. Results show that rD values calculated using ISO 3382-3 speech spectrum were significantly different to those calculated using other standard and proposed speech spectra. The choice of standard octave-band transmission index weights, e.g., male/female, used in standard STI calculation as per IEC 60268-16 (2011), did not significantly affect rD values. However, compared to the standard STI calculation, the pre-standard weights from the STI model from 1985 that is used in some important studies cited in ISO 3382-3 yielded much shorter rD values; RASTI yielded an increase in rD of less than 1 m; whereas STIPA (IEC 60268-16: 2011) did not affect rD. The effect of auditory spectral masking in standard STI calculation was negligible overall, but there were cases where the auditory masking reduced rD by up to 0.5 m. The effects of speech spectrum on two other relevant metrics in ISO 3382-3, namely spatial decay rate of speech, and A-weighted sound pressure level of speech at 4 m distance, were not significant. Overall, sensitivities of rD to several aspects of STI calculation are highlighted, which should contribute to the interpretation of studies that informed ISO 3382-3 development, support the current implied practice of using male STI weights for genderless speech in the calculation of rD, and provide guidance on potential refinements.

Phase-Aware Single-Channel Speech Enhancement With Modulation-Domain Kalman Filtering

We present a speech enhancement algorithm that performs modulation-domain Kalman filtering to track the speech phase using circular statistics, along with the spectral log-amplitudes of speech and noise. In the proposed algorithm, the speech phase posterior is used to create an enhanced speech phase spectrum for the signal reconstruction of speech. The Kalman filter prediction step separately models the temporal inter-frame correlation of the speech and noise spectral log-amplitudes and of the speech phase, while the Kalman filter update step models their nonlinear relations under the assumption that speech and noise add in the complex short-time Fourier transform domain. The phase-sensitive enhancement algorithm is evaluated with speech quality and intelligibility metrics, using a variety of noise types over a range of SNRs. Instrumental measures predict that tracking the speech log-spectrum and phase with modulation-domain Kalman filtering leads to consistent improvements in speech quality, over both conventional enhancement algorithms and other algorithms that perform modulation-domain Kalman filtering.

Electroacoustic assessment of wireless remote microphone systems.

Wireless remote microphones (RMs) transmit the desired acoustic signal to the hearing aid (HA) and facilitate enhanced listening in challenging environments. Fitting and verification of RMs, and benchmarking the relative performance of different RM devices in varied acoustic environments are of significant interest to Audiologists and RM developers. This paper investigates the application of instrumental speech intelligibility and quality metrics for characterizing the RM performance in two acoustic environments with varying amounts of background noise and reverberation. In both environments, two head and torso simulators (HATS) were placed 2 m apart, where one HATS served as the talker and the other served as the listener. Four RM systems were interfaced separately with a HA programmed to match the prescriptive targets for the N4 standard audiogram and placed on the listener HATS. The HA output in varied acoustic conditions was recorded and analyzed offline through computational models predicting speech intelligibility and quality. Results showed performance differences among the four RMs in the presence of noise and/or reverberation, with one RM exhibiting significantly better performance. Clinical implications and applications of these results are discussed.

Features for Masking-Based Monaural Speech Separation in Reverberant Conditions

Monaural speech separation is a fundamental problem in speech and signal processing. This problem can be approached from a supervised learning perspective by predicting an ideal time–frequency mask from features of noisy speech. In reverberant conditions at low signal-to-noise ratios (SNRs), accurate mask prediction is challenging and can benefit from effective features. In this paper, we investigate an extensive set of acoustic–phonetic features extracted in adverse conditions. Deep neural networks are used as the learning machine, and separation performance is evaluated using standard objective speech intelligibility metrics. Separation performance is systematically evaluated in both nonspeech and speech interference, in a variety of SNRs, reverberation times, and direct-to-reverberant energy ratios. Considerable performance improvement is observed by using contextual information, likely due to temporal effects of room reverberation. In addition, we construct feature combination sets using a sequential floating forward selection algorithm, and combined features outperform individual ones. We also find that optimal feature sets in anechoic conditions are different from those in reverberant conditions.

Metrics for evaluating noise suppression in hearing aids

Noise suppression in audio systems such as hearing aids typically involves modifying the speech envelope. Noisy speech segments having poor signal-to-noise ratios are attenuated while those considered to be primarily speech are left at or near full intensity. Both spectral subtraction and ideal binary mask noise suppression work on this principle. Recent work in the development of speech intelligibility and speech quality metrics indicates that the envelope fidelity, computed as the correlation coefficient between the processed noisy speech envelope and the original noise-free speech envelope, is an important factor in determining noise-suppression benefit. This result implies that the best noise suppression system is one in which the envelope of the processed noisy speech is restored to match as closely as possible that of the original noise-free speech. The potential benefits and limitations of envelope restoration for noisy speech are analyzed using the HASPI intelligibility and HASQI quality metrics, and the performance of existing noise suppression processing is compared to that of envelope restoration for both normal-hearing and hearing-impaired listeners.

Reference-Free Assessment of Speech Intelligibility Using Bispectrum of an Auditory Neurogram.

Sensorineural hearing loss occurs due to damage to the inner and outer hair cells of the peripheral auditory system. Hearing loss can cause decreases in audibility, dynamic range, frequency and temporal resolution of the auditory system, and all of these effects are known to affect speech intelligibility. In this study, a new reference-free speech intelligibility metric is proposed using 2-D neurograms constructed from the output of a computational model of the auditory periphery. The responses of the auditory-nerve fibers with a wide range of characteristic frequencies were simulated to construct neurograms. The features of the neurograms were extracted using third-order statistics referred to as bispectrum. The phase coupling of neurogram bispectrum provides a unique insight for the presence (or deficit) of supra-threshold nonlinearities beyond audibility for listeners with normal hearing (or hearing loss). The speech intelligibility scores predicted by the proposed method were compared to the behavioral scores for listeners with normal hearing and hearing loss both in quiet and under noisy background conditions. The results were also compared to the performance of some existing methods. The predicted results showed a good fit with a small error suggesting that the subjective scores can be estimated reliably using the proposed neural-response-based metric. The proposed metric also had a wide dynamic range, and the predicted scores were well-separated as a function of hearing loss. The proposed metric successfully captures the effects of hearing loss and supra-threshold nonlinearities on speech intelligibility. This metric could be applied to evaluate the performance of various speech-processing algorithms designed for hearing aids and cochlear implants.

Using speech intelligibility metrics to predict the performance of hearing-impaired listeners

When fitting a hearing aid to a hearing-impaired listener, it would be very beneficial for an audiologist to be able to use an objective metric to determine the best signal processing algorithm to compensate for the user’s loss and improve the user’s understanding of speech in degraded acoustic environments. Several metrics have been considered to predict impaired listeners’ speech recognition, including modified versions of the Speech Intelligibility Index (SII) [e.g., Ricketts, Ear Hear. 17, 124–132 (1996)] and the Speech Transmission Index (STI) [e.g., Humes et al., J. Speech Hear. Res. 29, 447–462 (1986)]. For impaired listeners using linear amplification hearing aids, there are metrics that work fairly well. For listeners using hearing aids that include nonlinear processing, such as amplitude compression, results have not been as successful. This talk will review the current status of metric capabilities, approaches that have been proposed to mitigate metric limitations and which metric features appe...

Intelligibility of Reverberant Speech with Amplification: Limitation of Speech Intelligibility Metrics, and a Preliminary Examination of an Alternative Approach

This study examines the effect of speech level on intelligibility in different reverberation conditions, and explores the potential of loudness-based reverberation parameters proposed by Lee et al. [J. Acoust. Soc. Am., 131(2), 1194-1205 (2012)] to explain the effect of speech level on intelligibility in various reverberation conditions. Listening experiments were performed with three speech levels (LAeq of 55 dB, 65 dB and 75 dB) and three reverberation conditions (T20 of 1.0 s, 1.9 s and 4.0 s), and subjects listened to speech stimuli through headphones. Collected subjective data were compared with two conventional speech intelligibility parameters (Speech Intelligibility Index and Speech Transmission Index) and two loudness-based reverberation parameters (EDTN and TN). Results reveal that the effect of speech level on intelligibility changes with a room’s reverberation conditions, and that increased level results in reduced intelligibility in highly reverberant conditions. EDTN and TN explain this finding better than do STI and SII, because they consider many psychoacoustic phenomena important for the modeling of the effect of speech level varying with reverberation.