Speech Intelligibility Prediction Research Articles

Multilingual non-intrusive binaural intelligibility prediction based on phone classification

Speech intelligibility (SI) prediction models are a valuable tool for the development of speech processing algorithms for hearing aids or consumer electronics. For the use in realistic environments it is desirable that the SI model is non-intrusive (does not require separate input of original and degraded speech, transcripts or a-priori knowledge about the signals) and does a binaural processing of the audio signals. Most of the existing SI models do not fulfill all of these criteria. In this study, we propose an SI model based on phone probabilities obtained from a deep neural net. The model comprises a binaural enhancement stage for prediction of the speech recognition threshold (SRT) in realistic acoustic scenes. In the first part of the study, SRT predictions in different spatial configurations are compared to the results from normal-hearing listeners. On average, our approach produces lower errors and higher correlations compared to three intrusive baseline models. In the second part, we explore if measures relevant in spatial hearing, i.e., the intelligibility level difference (ILD) and the binaural ILD (BILD), can be predicted with our modeling approach. We also investigate if a language mismatch between training and testing the model plays a role when predicting ILD and BILD. This point is especially important for low-resource languages, where not thousands of hours of language material are available for training. Binaural benefits are predicted by our model with an error of 1.5 dB. This is slightly higher than the error with a competitive baseline MBSTOI (1.1 dB), but does not require separate input of original and degraded speech. We also find that good binaural predictions can be obtained with models that are not specifically trained with the target language.

SAMI: an M-Health application to telemonitor intelligibility and speech disorder severity in head and neck cancers.

Perceptual measures, such as intelligibility and speech disorder severity, are widely used in the clinical assessment of speech disorders in patients treated for oral or oropharyngeal cancer. Despite their widespread usage, these measures are known to be subjective and hard to reproduce. Therefore, an M-Health assessment based on an automatic prediction has been seen as a more robust and reliable alternative. Despite recent progress, these automatic approaches still remain somewhat theoretical, and a need to implement them in real clinical practice rises. Hence, in the present work we introduce SAMI, a clinical mobile application used to predict speech intelligibility and disorder severity as well as to monitor patient progress on these measures over time. The first part of this work illustrates the design and development of the systems supported by SAMI. Here, we show how deep neural speaker embeddings are used to automatically regress speech disorder measurements (intelligibility and severity), as well as the training and validation of the system on a French corpus of head and neck cancer. Furthermore, we also test our model on a secondary corpus recorded in real clinical conditions. The second part details the results obtained from the deployment of our system in a real clinical environment, over the course of several weeks. In this section, the results obtained with SAMI are compared to an a posteriori perceptual evaluation, conducted by a set of experts on the new recorded data. The comparison suggests a high correlation and a low error between the perceptual and automatic evaluations, validating the clinical usage of the proposed application.

Personalized prediction of speech intelligibility for hearing-impaired listeners using a physiological model of the human ear

Due to the problems of occlusion effect, insufficient high-frequency gain, and acoustic feedback in conventional hearing aids, middle ear implants as a new type of hearing device have become an important compensatory tool for the treatment of patients with moderate to severe hearing loss. However, current speech intelligibility (SI) models are inadequate for predicting SI in hearing-impaired (HI) listeners after middle ear implants implantation. Moreover, the compensatory performance of middle ear implants before implantation remains unknown due to the invasive nature of the surgical procedure. Therefore, this study proposes a novel SI model that can predict the compensatory effects on SI after middle ear implants implantation. The model combines a physiologically nonlinear auditory preprocessing front-end with a short-term correlation analysis back-end. In normal-hearing (NH) listeners, the model accurately predicts speech reception thresholds (SRTs) and masking release under steady-state and fluctuating noise conditions. For HI listeners, the model modifies the parameters of outer hair cells and inner hair cells in the preprocessing front-end to simulate the patient’s audiogram, achieving excellent predictive capability for HI listeners’ test data. Overall, the proposed SI model can be used for the optimal design and algorithm fitting of middle ear implants tailored to patients with varying degrees of hearing loss, offering valuable insights for the clinical treatment of HI listeners.

Speech intelligibility prediction using generalized ESTOI with fine-tuned parameters

In this article, a lightweight and interpretable speech intelligibility prediction network is proposed. It is based on the ESTOI metric with several extensions: learned modulation filterbank, temporal attention, and taking into account robustness of a given reference recording. The proposed network is differentiable, and therefore it can be applied as a loss function in speech enhancement systems. The method was evaluated using the Clarity Prediction Challenge dataset. Compared to MB-STOI, the best of the systems proposed in this paper reduced RMSE from 28.01 to 21.33. It also outperformed best performing systems from the Clarity Challenge, while its training does not require additional labels like speech enhancement system and talker. It also has small memory and requirements, therefore, it can be potentially used as a loss function to train speech enhancement system. As it would consume less resources, the saved ones can be used for a larger speech enhancement neural network.

Impact of perceptual-learning strategies and background noise on disordered speech intelligibility

Objective: There is a plethora of research showing reduced speech intelligibility for a variety of voice disorders (i.e., dysphonia, alaryngeal). Therapeutic approaches to improve intelligibility typically involve targeting the speaker (e.g., clear speech, reduced rate) with minimal attention to the listener. Therefore, there were three purposes of this study: 1) to determine the impact of background noise on the speech intelligibility of disordered speakers; 2) to determine the impact of providing listeners with perceptual-learning strategies on the speech intelligibility of speakers with a voice disorder; and 3) to determine if subjective ratings of voice quality can predict speech intelligibility. Methods: Sentences were recorded from 12 speakers (2 typical, 3 alaryngeal, 7 dysphonic). Sentences were divided into one of three groups of signal-to-noise ratios (SNR: quiet, +5 dB SNR, and 0 dB SNR) and individually presented to 129 healthy listeners divided into one of three groups (i.e., control, acknowledgment of disorder, cognitive-perceptual strategies). Orthographic transcription was used to assess speech intelligibility. In addition, three expert listeners provided subjective voice quality ratings of all speakers. Results: Listeners had significantly more intelligibility errors with increased background noise (p <.001) and providing strategies to listeners did not result in a statistical improvement level F(6, 486) = 1.53, p = .17, η2p = 0.02. Regression analysis showed that the subjective voice quality overall severity was able to predict speech intelligibility in the noisy condition (0 dB SNR), accounting for 37% of the variance, R2 = .365, F(1,10) = 5.759, p = .037. Conclusion: Results suggest that increased background noise has a deleterious effect on the speech intelligibility of those with a voice disorder but that providing listeners with strategies in hopes of improving speaker intelligibility was not successful. Results did provide support, however, for the use of subjective voice quality ratings as a potential index of speech intelligibility.

Syllable-rate-adjusted-modulation (SRAM) predicts clear and conversational speech intelligibility.

Objectively predicting speech intelligibility is important in both telecommunication and human-machine interaction systems. The classic method relies on signal-to-noise ratios (SNR) to successfully predict speech intelligibility. One exception is clear speech, in which a talker intentionally articulates as if speaking to someone who has hearing loss or is from a different language background. As a result, at the same SNR, clear speech produces higher intelligibility than conversational speech. Despite numerous efforts, no objective metric can successfully predict the clear speech benefit at the sentence level. We proposed a Syllable-Rate-Adjusted-Modulation (SRAM) index to predict the intelligibility of clear and conversational speech. The SRAM used as short as 1 s speech and estimated its modulation power above the syllable rate. We compared SRAM with three reference metrics: envelope-regression-based speech transmission index (ER-STI), hearing-aid speech perception index version 2 (HASPI-v2) and short-time objective intelligibility (STOI), and five automatic speech recognition systems: Amazon Transcribe, Microsoft Azure Speech-To-Text, Google Speech-To-Text, wav2vec2 and Whisper. SRAM outperformed the three reference metrics (ER-STI, HASPI-v2 and STOI) and the five automatic speech recognition systems. Additionally, we demonstrated the important role of syllable rate in predicting speech intelligibility by comparing SRAM with the total modulation power (TMP) that was not adjusted by the syllable rate. SRAM can potentially help understand the characteristics of clear speech, screen speech materials with high intelligibility, and convert conversational speech into clear speech.

Data-Driven Non-Intrusive Speech Intelligibility Prediction Using Speech Presence Probability

Data-Driven Non-Intrusive Speech Intelligibility Prediction Using Speech Presence Probability

Non-intrusive Speech Intelligibility Prediction Method for Reverberant Speech Using Neural Network-Based Frequency Segmentation and Masking Front-end

In conventional non-intrusive speech intelligibility estimation, reverberation is extracted from the time-frequency representation of the input by explicit filter bank processing or spectral masking. However, these filter banks and masking processes are not always optimal. We replaced these processes with convolutional neural networks using rectangular kernels restricted to the frequency direction and masking such as a self-attention mechanism. We believe that this will enable feature extraction that is optimal for intelligibility estimation and will enable its estimation with high accuracy that generalizes well to input under various conditions. We further applied this front-end CNN to a previously proposed prediction model using speech enhancement. As a result, the estimation accuracy was improved compared to conventional front-ends using fixed filter banks, and this prediction showed a correlation coefficient with the subjective evaluation of 0.84 compared to 0.80 with the fixed filter bank.

Non-intrusive speech intelligibility prediction using an auditory periphery model with hearing loss

Speech intelligibility prediction methods are necessary for hearing aid development. However, many such prediction methods are categorized as intrusive metrics because they require reference speech as input, which is often unavailable in real-world situations. Additionally, the processing techniques in hearing aids may cause temporal or frequency shifts, which degrade the accuracy of intrusive speech intelligibility metrics. This paper proposes a non-intrusive auditory model for predicting speech intelligibility under hearing loss conditions. The proposed method requires binaural signals from hearing aids and audiograms representing the hearing conditions of hearing-impaired listeners. It also includes additional acoustic features to improve the method's robustness in noisy and reverberant environments. A two-dimensional convolutional neural network with neural decision forests is used to construct a speech intelligibility prediction model. An evaluation conducted with the first Clarity Prediction Challenge dataset shows that the proposed method performs better than the baseline system.

Variability of speech spectra in offices by room types and communication methods

The acoustic characteristics of room environments can significantly influence speech levels and spectra, an aspect often overlooked in the current method for predicting speech privacy. This paper presents an investigation into the variability of speech spectra in various office contexts, with a specific focus on the influence of room type, communication medium, and language. The study involved over 70 workers in different office room types in Quebec, Canada, who participated in measuring speech spectra within those spaces. Two communication methods, in-person and video-conferencing scenarios, were utilized with participants using either English or French languages. The real-world scenarios shed light on the significant impact of various parameters on speech characteristics within office settings. Comparing the results with standardized speech levels and spectra from ASTM and ISO standards, derived from controlled environments like anechoic chambers, the study reveals significant differences when applying the standard speech spectra to actual office environments. Based on these findings, the study proposes potential modifications to the existing standards, aiming to enhance the accuracy of speech privacy and intelligibility predictions.

Objective Assessment of Binaural Benefit from Acoustical Treatment in Real Primary School Classrooms.

Providing students with an adequate acoustic environment is crucial for ensuring speech intelligibility in primary school classrooms. Two main approaches to control acoustics in educational facilities consist of reducing background noise and late reverberation. Prediction models for speech intelligibility have been developed and implemented to evaluate the effects of these approaches. In this study, two versions of the Binaural Speech Intelligibility Model (BSIM) were used to predict speech intelligibility in realistic spatial configurations of speakers and listeners, considering binaural aspects. Both versions shared the same binaural processing and speech intelligibility backend processes but differed in the pre-processing of the speech signal. An Italian primary school classroom was characterized in terms of acoustics before (reverberation, T20 = 1.6 ± 0.1 s) and after (T20 = 0.6 ± 0.1 s) an acoustical treatment to compare BSIM predictions to well-established room acoustic measures. With shorter reverberation time, speech clarity and definition improved, as well as speech recognition thresholds (SRTs) (by up to ~6 dB), particularly when the noise source was close to the receiver and an energetic masker was present. Conversely, longer reverberation times resulted (i) in poorer SRTs (by ~11 dB on average) and (ii) in an almost non-existent spatial release from masking at an angle (SRM).

Delta-band neural envelope tracking predicts speech intelligibility in noise in preschoolers

Behavioral tests are currently the gold standard in measuring speech intelligibility. However, these tests can be difficult to administer in young children due to factors such as motivation, linguistic knowledge and cognitive skills. It has been shown that measures of neural envelope tracking can be used to predict speech intelligibility and overcome these issues. However, its potential as an objective measure for speech intelligibility in noise remains to be investigated in preschool children. Here, we evaluated neural envelope tracking as a function of signal-to-noise ratio (SNR) in 14 5-year-old children. We examined EEG responses to natural, continuous speech presented at different SNRs ranging from −8 (very difficult) to 8 dB SNR (very easy). As expected delta band (0.5-4 Hz) tracking increased with increasing stimulus SNR. However, this increase was not strictly monotonic as neural tracking reached a plateau between 0 and 4 dB SNR, similarly to the behavioral speech intelligibility outcomes. These findings indicate that neural tracking in the delta band remains stable, as long as the acoustical degradation of the speech signal does not reflect significant changes in speech intelligibility. Theta band tracking (4-8 Hz), on the other hand, was found to be drastically reduced and more easily affected by noise in children, making it less reliable as a measure of speech intelligibility. By contrast, neural envelope tracking in the delta band was directly associated with behavioral measures of speech intelligibility. This suggests that neural envelope tracking in the delta band is a valuable tool for evaluating speech-in-noise intelligibility in preschoolers, highlighting its potential as an objective measure of speech in difficult-to-test populations.

Effects of speech intelligibility and reverberation time on the serial recall task in Chinese open-plan offices: A laboratory study

Speech intelligibility is an essential index for evaluating acoustic performance in open-plan offices. Both speech-to-noise ratio (SNR) and reverberation time (RT) are critical parameters for determining the Speech Transmission Index (STI). STI is an important parameter for the objective prediction of speech intelligibility. Many studies explored the effects of speech intelligibility on work performance and acoustic environmental perceptions in open-plan offices by changing the SNR to obtain various STI conditions. However, few studies research how RT affects speech intelligibility and then influences work performance and perceptions of acoustic environments in open-plan offices. This study conducted a laboratory experiment to identify the changing trends of work performance and acoustic environment perceptions with the increase in STI under different RT conditions and to explore how room RT affects work performance and perceptions of the acoustic environment under the same STI condition. The acoustic conditions tested in this study varied in speech intelligibility (STI of 0.21, 0.42, and 0.61) and reverberation time (RT of 0.4 s and 1.4 s). The main outcome of this study is that occupants have less mental workload, faster task completion speed, and higher acoustic adaptability in a long RT environment compared to a short RT environment at an STI of 0.42. Furthermore, the data show a decreased work performance and an increased speech disturbance with the increase in STI in the short RT environment, while that trend was not observed in the long RT environment. The effects of STI conditions on occupants may differ by gender and noise sensitivity.

Modeling the effect of linguistic predictability on speech intelligibility prediction.

Many existing speech intelligibility prediction (SIP) algorithms can only account for acoustic factors affecting speech intelligibility and cannot predict intelligibility across corpora with different linguistic predictability. To address this, a linguistic component was added to five existing SIP algorithms by estimating linguistic corpus predictability using a pre-trained language model. The results showed improved SIP performance in terms of correlation and prediction error over a mixture of four datasets, each with a different English open-set corpus.

Vowel Acoustics as Predictors of Speech Intelligibility in Dysarthria.

This study sought to determine if alternative vowel space area (VSA) measures (i.e., novel trajectory-based measures: vowel space hull area and vowel space density) predicted speech intelligibility to the same extent as two traditional vowel measures (i.e., token-based measures: VSA and corner dispersion) in speakers with dysarthria. Additionally, this study examined if the strength of the relationship between acoustic vowel measures and intelligibility differed based on how intelligibility was measured (i.e., orthographic transcriptions [OTs] and visual analog scale [VAS] ratings). The Grandfather Passage was read aloud by 40 speakers with dysarthria of varying etiologies, including Parkinson's disease (n = 10), amyotrophic lateral sclerosis (n = 10), Huntington's disease (n = 10), and cerebellar ataxia (n = 10). Token- and trajectory-based acoustic vowel measures were calculated from the passage. Naïve listeners (N = 140) were recruited via crowdsourcing to provide OTs and VAS intelligibility ratings. Hierarchical linear regression models were created to model OTs and VAS intelligibility ratings using the acoustic vowel measures as predictors. Traditional VSA was the sole significant predictor of speech intelligibility for both the OTs (R 2 = .259) and VAS (R 2 = .236) models. In contrast, the trajectory-based measures were not significant predictors of intelligibility. Additionally, the OTs and VAS intelligibility ratings conveyed similar information. The findings suggest that traditional token-based vowel measures better predict intelligibility than trajectory-based measures. Additionally, the findings suggest that VAS methods are comparable to OT methods for estimating speech intelligibility for research purposes.

Predicting Speech Intelligibility for People with a Hearing Loss: The Clarity Challenges

Objective speech intelligibility metrics are used to reduce the need for time consuming listening tests. They are used in the design of audio systems; room acoustics and signal processing algorithms. Most published speech intelligibility metrics have been developed using young adults with so-called 'normal hearing', and therefore do not work well for those with different hearing characteristics. One of the most common causes of aural diversity is sensorineural hearing loss. While partially restoring perception through hearing aids is possible, results are mixed. This has led to the Clarity Project, which is running an open series of Enhancement Challenges to improve the processing of speech-in-noise for hearing aids. To enable this, objective metrics of speech intelligibility are needed, which work from signals produced by hearing aids for diverse listeners. For this reason, Clarity is also running Prediction Challenges to improve speech intelligibility models. Competitors are given a set of audio signals produced by hearing aid algorithms, and challenged to predict how many words a listener with a particular hearing characteristic will achieve. Drawing on the learning from the challenge, we will outline what has been learnt about improving intelligibility metrics for those with a hearing impairment.

Speech Intelligibility in Reverberation is Reduced During Self-Rotation.

Speech intelligibility in cocktail party situations has been traditionally studied for stationary sound sources and stationary participants. Here, speech intelligibility and behavior were investigated during active self-rotation of standing participants in a spatialized speech test. We investigated if people would rotate to improve speech intelligibility, and we asked if knowing the target location would be further beneficial. Target sentences randomly appeared at one of four possible locations: 0°, ± 90°, 180° relative to the participant's initial orientation on each trial, while speech-shaped noise was presented from the front (0°). Participants responded naturally with self-rotating motion. Target sentences were presented either without (Audio-only) or with a picture of an avatar (Audio-Visual). In a baseline (Static) condition, people were standing still without visual location cues. Participants' self-orientation undershot the target location and orientations were close to acoustically optimal. Participants oriented more often in an acoustically optimal way, and speech intelligibility was higher in the Audio-Visual than in the Audio-only condition for the lateral targets. The intelligibility of the individual words in Audio-Visual and Audio-only increased during self-rotation towards the rear target, but it was reduced for the lateral targets when compared to Static, which could be mostly, but not fully, attributed to changes in spatial unmasking. Speech intelligibility prediction based on a model of static spatial unmasking considering self-rotations overestimated the participant performance by 1.4 dB. The results suggest that speech intelligibility is reduced during self-rotation, and that visual cues of location help to achieve more optimal self-rotations and better speech intelligibility.

Further evaluation of the binaural loudness ratio to predict speech intelligibility in vehicles and rooms

This study compared the predictions of speech reception threshold (SRT) across conditions involving spatially distributed speech and noise in a vehicle and in a room, obtained with two prediction methods: the binaural speech-to-noise loudness ratio (BLR) metric proposed by Samardzic and Moore (2021) and the binaural speech intelligibility model (LAV) proposed by Lavandier et al. (2012) used here as a reference. Both methods provide SRT predictions with similar good accuracy for the vehicle experiments that involved background noise generated in a vehicle dynamometer test chamber and on road. The BLR model does not provide accurate SRT predictions for the room experiments that involved both single and multiple stationary noise sources simulated at different positions, for which the LAV model gives accurate predictions. Further analyses indicated that monaural information at the best ear is sufficient to describe the data well for the experiment with background noise generated in a vehicle dynamometer. They also revealed that such monaural analysis is not sufficient for accurate SRT predictions for the experiment with background noise generated on road, nor in the room conditions. Overall, the results indicated that the BLR method does not account for the better-ear listening and binaural unmasking effects underlying the corresponding variations in SRT, particularly in the room experiments.

Characterization and Prediction of Speech Intelligibility at the Output of Hearing Aids in a Noisy Working Environment.

Hearing aids are more and more technically advanced, but do not necessarily guarantee the reproduction of useful signals in all working environments. This is particularly the case for speech intelligibility. This study focuses on the prediction of hearing aid performance in the case of a moderate deafness setting, in service and industrial work environments. To improve intelligibility, hearing aids propose signal processing options such as noise reduction and compression. These processes can transform hearing aids into nonlinear systems. The aim of this study is to develop a nonlinear method for the characterization of hearing aids. The method is based on the synchronized swept sine (SSS) signal method.[16] The SSS method is applied for determining hearing aid frequency responses fitted according to the present methodology and several processing options. The characterization of hearing aid's program containing the noise reduction function is specifically analyzed. Indeed, to be fully active and efficient, the hearing aid, with the noise reduction feature activated, needs to be immersed in a noisy environment which does not allow nonlinear characterization. A linear approach is taken to study this feature. Three hearing aids commonly sold by hearing care professionals are studied here; all of them have three different programs. The characterization for each program is discussed. The statistical study showed that the intelligibility, assessed using the speech transmission index in these sound environments, is well estimated for every program, although certain differences are observed when the compression effect is too high in the service work sector. The characterizations of hearing aids using the programs studied did not highlight the presence of frequency nonlinearities. The characterization method could not take into account amplitude nonlinearities when there is too much gain compression in the hearing process. Globally, all the hearing aid programs provided a very significant improvement in intelligibility in service and industrial work contexts.

AI-IoT Platform for Blind Estimation of Room Acoustic Parameters Based on Deep Neural Networks

Room acoustical parameters have been widely used to describe sound perception in indoor environments, such as concert halls, conference rooms, etc. Many of them have been standardized and often have a high computational demand. With the increasing presence of deep learning approaches in automatic monitoring systems, wireless acoustic sensor networks (WASNs) offer great potential to facilitate the estimation of such parameters. In this scenario, convolutional neural networks (CNNs) offer significant reductions in the computational requirements for in-node parameter predictions, enabling the so-called Artificial Intelligence-Internet of Things (AI-IoT). In this article, we describe the design and analysis of a CNN trained to predict simultaneously a set of common room acoustical parameters directly from speech signals, without the need for specific impulse response measurements. The results show that the proposed CNN-based prediction of room acoustical parameters and speech intelligibility achieves a relative error rate of less than a 5.5%, accompanied by a computational speedup factor close to 250 with respect to the conventional signal processing approach.