Speech Intelligibility Prediction Research Articles

A speech pre-processing method to reduce overlap masking in reverberant environments

In daily life, we are often exposed to speech that is rendered over loudspeakers in a reverberant acoustical environment (examples are public-address systems used in a train station or conference halls). Whereas the early reflections can support speech intelligibility, the late reflections smear the speech in time which will result in an overlap of consecutive speech portions and an effective low-pass filtering of the speech-specific modulation spectrum. This study proposes a perceptually motivated pre-processing approach, based on previous work of Hodoshima et al. [J. Acoust. Soc. 119, 4055-4064 (2006)], which reduces the detrimental effects of reverberation by suppressing steady-state portions and emphasizing potentially inaudible/masked segments of speech before it is emitted in the reverberant environment. For pre-processing, the impulse response is separated into a direct and a reverberant path to decide whether speech segments are inaudible and can be neglected or should be emphasized. A speech intelligibility prediction model is used to select the optimal pre-processing parameters for each specific acoustical environment. Listening tests showed, that this pre-processing approach is able to partially compensate the detrimental effects of reverberation leading to a reduction in speech reception thresholds of about 2 to 5 dB measured in speech shaped noise.

Prediction of speech intelligibility of children with cerebral palsy from characteristics at 2 years of age

Prediction of speech intelligibility of children with cerebral palsy from characteristics at 2 years of age

Modeling and Designing Acoustical Conditions of the Interior - Case Study

Abstract The primary aim of this research study was to model acoustic conditions of the Courtyard of the Gdańsk University of Technology Main Building, and then to design a sound reinforcement system for this interior. First, results of measurements of the parameters of the acoustic field are presented. Then, the comparison between measured and predicted values using the ODEON program is shown. Collected data indicate a long reverberation time which results in poor speech intelligibility. Then, a thorough analysis is perform to improve the acoustic properties of the model of the interior investigated. On the basis of the improved acoustic model two options of a sound reinforcement system for this interior are proposed, and then analyzed. After applying sound absorbing material it was noted that the predicted speech intelligibility increased from bad/poor rating to good category.

Envelope and intensity based prediction of psychoacoustic masking and speech intelligibility.

Human auditory perception and speech intelligibility have been successfully described based on the two concepts of spectral masking and amplitude modulation (AM) masking. The power-spectrum model (PSM) [Patterson and Moore (1986). Frequency Selectivity in Hearing, pp. 123-177] accounts for effects of spectral masking and critical bandwidth, while the envelope power-spectrum model (EPSM) [Ewert and Dau (2000). J. Acoust. Soc. Am. 108, 1181-1196] has been successfully applied to AM masking and discrimination. Both models extract the long-term (envelope) power to calculate signal-to-noise ratios (SNR). Recently, the EPSM has been applied to speech intelligibility (SI) considering the short-term envelope SNR on various time scales (multi-resolution speech-based envelope power-spectrum model; mr-sEPSM) to account for SI in fluctuating noise [Jørgensen, Ewert, and Dau (2013). J. Acoust. Soc. Am. 134, 436-446]. Here, a generalized auditory model is suggested combining the classical PSM and the mr-sEPSM to jointly account for psychoacoustics and speech intelligibility. The model was extended to consider the local AM depth in conditions with slowly varying signal levels, and the relative role of long-term and short-term SNR was assessed. The suggested generalized power-spectrum model is shown to account for a large variety of psychoacoustic data and to predict speech intelligibility in various types of background noise.

Spatial Acoustic Scenarios in Multichannel Loudspeaker Systems for Hearing Aid Evaluation.

Field tests and guided walks in real environments show that the benefit from hearing aid (HA) signal processing in real-life situations is typically lower than the predicted benefit found in laboratory studies. This suggests that laboratory test outcome measures are poor predictors of real-life HA benefits. However, a systematic evaluation of algorithms in the field is difficult due to the lack of reproducibility and control of the test conditions. Virtual acoustic environments that simulate real-life situations may allow for a systematic and reproducible evaluation of HAs under more realistic conditions, thus providing a better estimate of real-life benefit than established laboratory tests. To quantify the difference in HA performance between a laboratory condition and more realistic conditions based on technical performance measures using virtual acoustic environments, and to identify the factors affecting HA performance across the tested environments. A set of typical HA beamformer algorithms was evaluated in virtual acoustic environments of different complexity. Performance was assessed based on established technical performance measures, including perceptual model predictions of speech quality and speech intelligibility. Virtual acoustic environments ranged from a simple static reference condition to more realistic complex scenes with dynamically moving sound objects. HA benefit, as predicted by signal-to-noise ratio (SNR) and speech intelligibility measures, differs between the reference condition and more realistic conditions for the tested beamformer algorithms. Other performance measures, such as speech quality or binaural degree of diffusiveness, do not show pronounced differences. However, a decreased speech quality was found in specific conditions. A correlation analysis showed a significant correlation between room acoustic parameters of the sound field and HA performance. The SNR improvement in the reference condition was found to be a poor predictor of HA performance in terms of speech intelligibility improvement in the more realistic conditions. Using several virtual acoustic environments of different complexity, a systematic difference in HA performance between a simple reference condition and more realistic environments was found, which may be related to the discrepancy between laboratory and real-life HA performance reported previously.

Predicting Speech Intelligibility Decline in Amyotrophic Lateral Sclerosis Based on the Deterioration of Individual Speech Subsystems.

PurposeTo determine the mechanisms of speech intelligibility impairment due to neurologic impairments, intelligibility decline was modeled as a function of co-occurring changes in the articulatory, resonatory, phonatory, and respiratory subsystems.MethodSixty-six individuals diagnosed with amyotrophic lateral sclerosis (ALS) were studied longitudinally. The disease-related changes in articulatory, resonatory, phonatory, and respiratory subsystems were quantified using multiple instrumental measures, which were subjected to a principal component analysis and mixed effects models to derive a set of speech subsystem predictors. A stepwise approach was used to select the best set of subsystem predictors to model the overall decline in intelligibility.ResultsIntelligibility was modeled as a function of five predictors that corresponded to velocities of lip and jaw movements (articulatory), number of syllable repetitions in the alternating motion rate task (articulatory), nasal airflow (resonatory), maximum fundamental frequency (phonatory), and speech pauses (respiratory). The model accounted for 95.6% of the variance in intelligibility, among which the articulatory predictors showed the most substantial independent contribution (57.7%).ConclusionArticulatory impairments characterized by reduced velocities of lip and jaw movements and resonatory impairments characterized by increased nasal airflow served as the subsystem predictors of the longitudinal decline of speech intelligibility in ALS. Declines in maximum performance tasks such as the alternating motion rate preceded declines in intelligibility, thus serving as early predictors of bulbar dysfunction. Following the rapid decline in speech intelligibility, a precipitous decline in maximum performance tasks subsequently occurred.

Spectrotemporal modulation sensitivity as a predictor of speech intelligibility in noise for hearing-impaired listeners

The audiogram accounts for only a portion of the variance in speech-reception performance in noise for hearing-impaired listeners. The remaining variance is often attributed to a combination of suprathreshold auditory distortion and non-auditory factors such as cognitive processing. This talk describes a series of studies demonstrating that a suprathreshold measure of sensitivity to spectro-temporal modulation (STM) can account for individual differences in speech-reception scores that are not predicted by the audiogram. STM stimuli are spectrally rippled noises with spectral-peak frequencies that shift over time, akin to modulations in a speech signal. The results show that STM sensitivity correlates to speech-reception performance in noise; that the correlation is ascribed mainly to the low-frequency portion of the stimulus (<2 kHz); and that STM sensitivity can account for individual differences in speech-reception thresholds for hearing-impaired listeners properly fit with individualized frequency-dependent gain. Hearing loss has the largest impact on STM sensitivity for low temporal rates and low carrier frequencies, suggesting a reduced ability to use temporal fine-structure information to detect slow-moving spectral peaks. STM detection is a fast, simple test of suprathreshold auditory function that complements the high-frequency audiogram to account for a substantial proportion of individual variability in speech reception in noise.

An information-theoretic approach to understanding interrupted speech

Everyday listening conditions seldom present the listener with a completely intact speech signal. Research on understanding interrupted speech spans the last seven decades, yet few efforts have focused on what parts of the speech signal were being interrupted. Kewley-Port, Fogerty, and others explored the importance of consonants versus vowels for understanding speech when either was replaced by noise, but this approach is limited by their inherent differences such as segment duration. The present approach is inspired by Shannon information theory, where information is defined by unpredictability, uncertainty, or change. We developed a metric of biologically relevant spectral change in the speech signal, termed cochlea-scaled entropy (CSE). Sentences with high-CSE intervals (low predictability = high information) replaced by noise were understood more poorly than sentences with an equal number and duration of low-CSE intervals (high predictability = low information) replaced. CSE has been shown to better predict speech intelligibility than various temporal measures and consonant/vowel status. This approach has been validated across wide ranges of acoustic simulations of cochlear implant processing, suggesting that cochlear implant users might also utilize these information-bearing acoustic changes to understand speech. Extensions and future directions for this work will be discussed

Spike-timing and mean-rate coding of the temporal fine structure and envelope cues in real words

A number of studies over the past decade have argued for the importance of temporal fine structure (TFS) cues for the perception of consonants. However, recent investigations indicate that TFS cues from consonants may largely be converted into envelope (ENV) cues by narrowband cochlear filtering, such that these cues are conveyed by the mean-rate response of auditory nerve fibers rather than spike-timing cues. However, these studies used nonsense VCV syllables, and this result may not generalize to real words in which the patterns of ENV and TFS cues may be substantially different, and in which lexical context may play a role. In this study, we used a computational model of the auditory periphery and neural-based speech intelligibility predictors to investigate the TFS and ENV representation of real words from the NU-6 database. Spike-timing and mean-rate cues were evaluated for “auditory chimaeras” created from this database, in which the TFS of one signal is mixed with the ENV of another. The results indicate that the chimaera processing has a bigger impact in general on the mean-rate representation of phonemes than on the spike-timing representation, and inclusion of the spike-timing cues gives better predictions of phoneme perception. [Funded by NSERC of Canada.]

Modeling speech intelligibility with recovered envelope from temporal fine structure stimulus

Temporal envelope and fine structure are two prominent acoustic cues for speech perception. Most existing speech-transmission-index-based metrics make use of the temporal envelope information and discard the temporal fine structure (TFS) cue to predict speech intelligibility. Recent studies have shown that the TFS stimulus synthesized with multiband TFS waveforms contains rich intelligibility information, which is reflected as the recovered envelope from the TFS stimulus. The present study first assessed the performance of using the recovered envelope from the synthesized TFS stimulus to predict the intelligibility of noise-distorted and noise-suppressed speech. The TFS stimulus was synthesized and fed as an input into the conventional normalized covariance measure (NCM) module. The results showed that the recovered envelope from the TFS stimulus predicted the intelligibility as well as the original envelope extracted from the wideband speech signal did. In addition, an additive intelligibility model was designed to combine the envelope from wideband speech and the recovered envelope from the TFS stimulus to predict speech intelligibility. The prediction power was significantly improved when these two envelope waveforms were integrated. The present study suggests that the recovered envelope from the TFS stimulus may be alternative acoustic information for modeling speech intelligibility and improving the prediction power of the conventional NCM-based intelligibility index.

A Binaural Grouping Model for Predicting Speech Intelligibility in Multitalker Environments.

Spatially separating speech maskers from target speech often leads to a large intelligibility improvement. Modeling this phenomenon has long been of interest to binaural-hearing researchers for uncovering brain mechanisms and for improving signal-processing algorithms in hearing-assistive devices. Much of the previous binaural modeling work focused on the unmasking enabled by binaural cues at the periphery, and little quantitative modeling has been directed toward the grouping or source-separation benefits of binaural processing. In this article, we propose a binaural model that focuses on grouping, specifically on the selection of time-frequency units that are dominated by signals from the direction of the target. The proposed model uses Equalization-Cancellation (EC) processing with a binary decision rule to estimate a time-frequency binary mask. EC processing is carried out to cancel the target signal and the energy change between the EC input and output is used as a feature that reflects target dominance in each time-frequency unit. The processing in the proposed model requires little computational resources and is straightforward to implement. In combination with the Coherence-based Speech Intelligibility Index, the model is applied to predict the speech intelligibility data measured by Marrone et al. The predicted speech reception threshold matches the pattern of the measured data well, even though the predicted intelligibility improvements relative to the colocated condition are larger than some of the measured data, which may reflect the lack of internal noise in this initial version of the model.

Microscopic prediction of speech intelligibility in spatially distributed speech-shaped noise for normal-hearing listeners

A binaural and psychoacoustically motivated intelligibility model, based on a well-known monaural microscopic model is proposed. This model simulates a phoneme recognition task in the presence of spatially distributed speech-shaped noise in anechoic scenarios. In the proposed model, binaural advantage effects are considered by generating a feature vector for a dynamic-time-warping speech recognizer. This vector consists of three subvectors incorporating two monaural subvectors to model the better-ear hearing, and a binaural subvector to simulate the binaural unmasking effect. The binaural unit of the model is based on equalization-cancellation theory. This model operates blindly, which means separate recordings of speech and noise are not required for the predictions. Speech intelligibility tests were conducted with 12 normal hearing listeners by collecting speech reception thresholds (SRTs) in the presence of single and multiple sources of speech-shaped noise. The comparison of the model predictions with the measured binaural SRTs, and with the predictions of a macroscopic binaural model called extended equalization-cancellation, shows that this approach predicts the intelligibility in anechoic scenarios with good precision. The square of the correlation coefficient (r(2)) and the mean-absolute error between the model predictions and the measurements are 0.98 and 0.62 dB, respectively.

Predicting the intelligibility of noise-corrupted speech non-intrusively by across-band envelope correlation

Abstract Recent psychoacoustic studies found that across-band envelope correlation (ABEC) carried important information for speech intelligibility. This motivated the present work to propose an ABEC-based intelligibility measure that could be used to non-intrusively predict speech intelligibility in noise using only temporal envelope waveforms extracted from the noise-corrupted speech. The proposed ABEC-based metric (denoted as ABECm) was computed by averaging the correlation coefficients of mean-removed envelope waveforms from adjacent frequency bands of the noise-corrupted speech. The ABECm measures were evaluated with intelligibility scores obtained from normal-hearing listeners presented with sentences corrupted by four types of maskers in a total of 22 conditions. High correlation ( r = 0.96) was obtained between ABECm values and listeners’ sentence recognition scores, and this correlation was comparable to those computed with existing intrusive and non-intrusive intelligibility measures. This suggests that across-band envelope correlation may work as a simple but efficient predictor of speech intelligibility in noise, whose computation does not need access to the clean reference signal.

Objective measures of perceptual quality for predicting speech intelligibility in sensorineural hearing loss.

An objective method to predict speech intelligibility in sensorineural hearing loss of different types and increasing degrees of severity is proposed and validated with experimental data. The novel approach is based on the combined use of acoustic simulations of impaired perception and objective measures of perceptual speech quality (PESQ). Acoustic simulations were obtained after degradation of the original, non distorted, speech waveforms by spectral smearing, expansive nonlinearity, and level scaling. PESQ was used to measure perceptual quality of the acoustic simulations obtained by varying the degree of the simulated hearing loss. A logistic function was applied to transform PESQ scores into predicted intelligibility scores. A set of CV and VC syllables in /a/, /u/, and /i/ contexts was used as reference test material. The method was validated with subjective measures of intelligibility of the degraded speech obtained in a group of 10 normal hearing subjects. Overall, prediction of experimental speech intelligibility through the transformed PESQ measures was good (R(2)=0.7; RMSE=0.08) revealing that the proposed approach could be a valuable aid in real clinical applications.

Learning Spectral Mapping for Speech Dereverberation and Denoising

In real-world environments, human speech is usually distorted by both reverberation and background noise, which have negative effects on speech intelligibility and speech quality. They also cause performance degradation in many speech technology applications, such as automatic speech recognition. Therefore, the dereverberation and denoising problems must be dealt with in daily listening environments. In this paper, we propose to perform speech dereverberation using supervised learning, and the supervised approach is then extended to address both dereverberation and denoising. Deep neural networks are trained to directly learn a spectral mapping from the magnitude spectrogram of corrupted speech to that of clean speech. The proposed approach substantially attenuates the distortion caused by reverberation, as well as background noise, and is conceptually simple. Systematic experiments show that the proposed approach leads to significant improvements of predicted speech intelligibility and quality, as well as automatic speech recognition in reverberant noisy conditions. Comparisons show that our approach substantially outperforms related methods.

Comparisons between simulated and in-situ measured speech intelligibility based on (binaural) room impulse responses

This study systematically compares acoustic simulation and in-situ measurement in terms of speech transmission index (STI), speech intelligibility scores and relationship curves when considering (binaural) room impulse response and four general room conditions, namely, an office, a laboratory, a multimedia lecture hall and a semi-anechoic chamber. The results reveal that STI can be predicted accurately by acoustic simulation (using room acoustics software ODEON) when there is a good agreement between the virtual models and the real rooms and that different reverberation time (RT) and signal-to-noise ratio (SNR) may exert less significant influence on the simulated STI. However, subjective intelligibility may be overestimated when using acoustic simulation due to the head-related transfer function (HRTF) filter used, and the score bias may be minimal and difficult to detect in everyday situations. There is no obvious score tendency caused by different RT, though with the decrease in the SNR, score bias may increase. Overall, considering that the accurate acoustic modelling of rooms is often problematic, it is difficult to obtain accurate speech intelligibility prediction results using a simulation technique, especially when the room has not yet been built.

Subjective and objective measurement of synthesized speech intelligibility in modern telephone conditions

This paper investigates the impact of different telephone channels, represented by impairments as introduced by modern telecommunication networks (e.g. speech coding, bandwidth limitation, packet loss, etc.), on the intelligibility of synthesized speech. Both subjective and objective assessments are used. Two different speech intelligibility prediction models, namely PESQ Intelligibility and POLQA Intelligibility, are evaluated by comparing the predictions with subjectively obtained intelligibility scores. The results show that all the investigated degradations seriously impact the intelligibility of the synthesized speech measured subjectively. Furthermore it is shown that PESQ Intelligibility provides too low correlations between predicted objective measurements and subjective scores for accurate prediction of speech intelligibility while POLQA Intelligibility is capable of providing good intelligibility predictions in the case that a closed response experimental set up is used.

Prediction of Speech Intelligibility Using a Neurogram Orthogonal Polynomial Measure (NOPM)

Sensorineural hearing loss (SNHL) is an increasingly prevalent condition, resulting from damage to the inner ear and causing a reduction in speech intelligibility. This paper proposes a new speech intelligibility prediction metric, the neurogram orthogonal polynomial measure (NOPM). This metric applies orthogonal moments to the auditory neurogram to predict speech intelligibility for listeners with and without hearing loss. The model simulates the responses of auditory-nerve fibers to speech signals under quiet and noisy conditions. Neurograms were created using a physiologically based computational model of the auditory periphery. A well-known orthogonal polynomial measure, Krawtchouk moments, was applied to extract features from the auditory neurogram. The predicted intelligibility scores were compared to subjective results, and NOPM showed a good fit with the subjective scores for normal listeners and also for listeners with hearing loss. The proposed metric has a realistic and wider dynamic range than corresponding existing metrics, such as mean structural similarity index measure and neurogram similarity index measure, and the predicted scores are also well-separated as a function of hearing loss. The application of this metric could be extended for assessing hearing-aid and speech-enhancement algorithms.

Envelope-power based prediction of auditory masking and speech intelligibility

In natural surroundings, people are often part of challenging acoustical scenarios (noise) interferers and reverberation. Amplitude modulations (AM) are fundamental acoustic features used to extract information in such situations. It has been shown that the envelope power spectrum model [EPSM; Ewert and Dau, J. Acoust. Soc. Am. 108, 1181 (2000)] can account for psychoacoustic AM detection and masking data by considering the long-term envelope signal-to-noise-ratios (SNR) at the output of modulation filters. Recently, this concept was extended to consider short-term envelope SNRs to predict speech intelligibility in fluctuating maskers and in reverberation [mr-sEPSM; Jørgensen et al., J. Acoust. Soc. Am. 134, 1475 (2013)]. In this study, the EPSM was further extended to consider envelope power SNRs (AM cues) on various time scales and “classic” power SNRs (intensity cues). The goal was to develop an auditory model for jointly predicting psychoacoustic masking and speech intelligibility. The model was demonstrated to account for a broad variety of psychoacoustic effects as AM detection, AM discrimination, AM masking, spectral masking, forward masking, intensity just-noticeable-differences (JNDs) and hearing threshold. Speech intelligibility was predicted comparable to Jørgensen et al.. Implications for future modeling including applications to audio and speech quality assessment are discussed.

Comparing NAL-NL1 and DSL v5 in Hearing Aids Fit to Children with Severe or Profound Hearing Loss: Goodness of Fit-to-Targets, Impacts on Predicted Loudness and Speech Intelligibility.

An important goal of providing amplification to children with hearing loss is to ensure that hearing aids are adjusted to match targets of prescriptive procedures as closely as possible. The Desired Sensation Level (DSL) v5 and the National Acoustic Laboratories' prescription for nonlinear hearing aids, version 1 (NAL-NL1) procedures are widely used in fitting hearing aids to children. Little is known about hearing aid fitting outcomes for children with severe or profound hearing loss. The purpose of this study was to investigate the prescribed and measured gain of hearing aids fit according to the NAL-NL1 and the DSL v5 procedure for children with moderately severe to profound hearing loss; and to examine the impact of choice of prescription on predicted speech intelligibility and loudness. Participants were fit with Phonak Naida V SP hearing aids according to the NAL-NL1 and DSL v5 procedures. The Speech Intelligibility Index (SII) and estimated loudness were calculated using published models. The sample consisted of 16 children (30 ears) aged between 7 and 17 yr old. The measured hearing aid gains were compared with the prescribed gains at 50 (low), 65 (medium), and 80 dB SPL (high) input levels. The goodness of fit-to-targets was quantified by calculating the average root-mean-square (RMS) error of the measured gain compared with prescriptive gain targets for 0.5, 1, 2, and 4 kHz. The significance of difference between prescriptions for hearing aid gains, SII, and loudness was examined by performing analyses of variance. Correlation analyses were used to examine the relationship between measures. The DSL v5 prescribed significantly higher overall gain than the NAL-NL1 procedure for the same audiograms. For low and medium input levels, the hearing aids of all children fit with NAL-NL1 were within 5 dB RMS of prescribed targets, but 33% (10 ears) deviated from the DSL v5 targets by more than 5 dB RMS on average. For high input level, the hearing aid fittings of 60% and 43% of ears deviated by more than 5 dB RMS from targets of NAL-NL1 and DSL v5, respectively. Greater deviations from targets were associated with more severe hearing loss. On average, the SII was higher for DSL v5 than for NAL-NL1 at low input level. No significant difference in SII was found between prescriptions at medium or high input level, despite greater loudness for DSL v5 than for NAL-NL1. Although targets between 0.25 and 2 kHz were well matched for both prescriptions in commercial hearing aids, gain targets at 4 kHz were matched for NAL-NL1 only. Although the two prescriptions differ markedly in estimated loudness, they resulted in comparable predicted speech intelligibility for medium and high input levels.