Unprocessed Speech Research Articles

Effect of cochlear implant n-of-m strategy on signal-to-noise ratio below which noise hinders speech recognition.

Speech recognition was measured in 24 normal-hearing subjects for unprocessed speech and for speech processed by a cochlear implant Advanced Combination Encoder (ACE) coding strategy in quiet and at various signal-to noise ratios (SNRs). All signals were low- or high-pass filtered to avoid ceiling effects. Surprisingly, speech recognition performance plateaus at approximately 22 dB SNR for both speech types, implying that ACE processing has no effect on the upper limit of the effective SNR range. Speech recognition improved significantly above 15 dB SNR, suggesting that the upper limit used in the Speech Intelligibility Index should be reconsidered.

Effects of aging on voice emotion recognition in cochlear implant users and normally hearing adults listening to spectrally degraded speech

Voice emotion recognition declines with age in normally hearing (NH) adults. However, the effects of aging on voice emotion recognition in NH listeners with spectrally degraded stimuli are neither known nor the effects of aging on adult cochlear implant (CI) users. This study explored age-related effects on voice emotion recognition in NH adults listening to spectrally degraded speech and aimed to compare these changes to age-related changes in adult CI users’ voice emotion recognition with unprocessed speech. Participants listened to 12 emotion-neutral sentences, each spoken with 5 emotions (happy, sad, scared, angry, and neutral) by a male and female talker. Participants identified which emotion they heard while listening to sentences that were either unprocessed or CI-simulated. Preliminary results indicate declines in overall percent correct scores and increased reaction time with both age and increasing spectral degradation in NH adults. Results also suggest age-related effects on percent correct scores and reaction times within the CI group alone. These results have important implications in the aging population of adults with NH and with CIs because limitations in the quality peer to peer interactions have been associated with a decrease in perceived quality of life. [Work supported by R01 DC014233 and P20 GM109023.]Voice emotion recognition declines with age in normally hearing (NH) adults. However, the effects of aging on voice emotion recognition in NH listeners with spectrally degraded stimuli are neither known nor the effects of aging on adult cochlear implant (CI) users. This study explored age-related effects on voice emotion recognition in NH adults listening to spectrally degraded speech and aimed to compare these changes to age-related changes in adult CI users’ voice emotion recognition with unprocessed speech. Participants listened to 12 emotion-neutral sentences, each spoken with 5 emotions (happy, sad, scared, angry, and neutral) by a male and female talker. Participants identified which emotion they heard while listening to sentences that were either unprocessed or CI-simulated. Preliminary results indicate declines in overall percent correct scores and increased reaction time with both age and increasing spectral degradation in NH adults. Results also suggest age-related effects on percent correct sco...

Understanding auditory representations of emotional expressions with neural networks

In contrast to many established emotion recognition systems, convolutional neural networks do not rely on handcrafted features to categorize emotions. Although achieving state-of-the-art performances, it is still not fully understood what these networks learn and how the learned representations correlate with the emotional characteristics of speech. The aim of this work is to contribute to a deeper understanding of the acoustic and prosodic features that are relevant for the perception of emotional states. Firstly, an artificial deep neural network architecture is proposed that learns the auditory features directly from the raw and unprocessed speech signal. Secondly, we introduce two novel methods for the analysis of the implicitly learned representations based on data-driven and network-driven visualization techniques. Using these methods, we identify how the network categorizes an audio signal as a two-dimensional representation of emotions, namely valence and arousal. The proposed approach is a general method to enable a deeper analysis and understanding of the most relevant representations to perceive emotional expressions in speech.

Evaluation of a near-end listening enhancement algorithm by combined speech intelligibility and listening effort measurements.

Previous studies showed that near-end listening enhancement (NELE) algorithms can significantly improve speech intelligibility in noisy environments. This study investigates the benefit of the NELE algorithm AdaptDRC in normal-hearing listeners at signal-to-noise ratios (SNRs) for which speech intelligibility is at ceiling, by evaluating listening effort for processed and unprocessed speech in the presence of speech-shaped and cafeteria noise. The results suggest that the NELE algorithm is able to reduce listening effort over a wide range of SNRs. Hence, listening effort seems to be applicable for evaluating NELE algorithms over a much wider SNR range than speech intelligibility.

Interaction between speech variations and background noise on speech intelligibility by Mandarin-speaking cochlear implant patients

Cochlear implant (CI) users have been shown to be more susceptible to the variations in speech production encountered in everyday listening, in which speaking rate, amplitude, duration, and voice pitch information may be quite variable, depending on the production context. Such variations may be further enlarged by the background noise, especially dynamic noise. The limited spectral resolution provided by the CI limits perception of voice pitch, which is an important cue for speech prosody and for tonal languages such as Mandarin Chinese. In this study, the effect of varying speaking rates and styles and background noise on speech understanding was investigated in Mandarin-speaking CI and normal-hearing (NH) listeners. Thirteen (5 male and 8 female, age 19–62 years) Mandarin-speaking, post-lingually deafened adult CI patients using their clinical processors and 9 (5 male and 4 female, age 23–59 years) NH subjects listening to unprocessed speech. Five different types of speech variations, including 3 speaking rates (slow, normal, fast) and 2 speaking styles (emotional, shouted) were presented with two masking noises (speech-shaped steady state noise-SSN or six-talker babble). Speech reception threshold, defined as the signal-to-noise ratio producing 50% correct word-in-sentence recognition using Mandarin Speech Perception materials was measured. NH listeners performed significantly better (16.7 dB) than CI patients across all conditions regardless of speech variations and noise types. CI patients’ performance deficit was highly dependent on speech rate and noise type; the deficit was smallest (11.7 dB) when slowly-spoken speech was presented in SSN and largest (20.6 dB) when shouted speech was presented in six-talker speech babble. NH listeners performed significantly better in speech babble than in SSN for all speech variations, while CI patients performed similarly in both noise types. The use of clear and slowly-spoken speech in the laboratory setting may largely underestimate CI patients’ performance deficits in real-world listening conditions, where acoustic variations introduced by speech variations and dynamic noise may present additional challenges.

Effects of Training on Lateralization for Simulations of Cochlear Implants and Single-Sided Deafness.

While cochlear implantation has benefitted many patients with single-sided deafness (SSD), there is great variability in cochlear implant (CI) outcomes and binaural performance remains poorer than that of normal-hearing (NH) listeners. Differences in sound quality across ears—temporal fine structure (TFS) information with acoustic hearing vs. coarse spectro-temporal envelope information with electric hearing—may limit integration of acoustic and electric patterns. Binaural performance may also be limited by inter-aural mismatch between the acoustic input frequency and the place of stimulation in the cochlea. SSD CI patients must learn to accommodate these differences between acoustic and electric stimulation to maximize binaural performance. It is possible that training may increase and/or accelerate accommodation and further improve binaural performance. In this study, we evaluated lateralization training in NH subjects listening to broad simulations of SSD CI signal processing. A 16-channel vocoder was used to simulate the coarse spectro-temporal cues available with electric hearing; the degree of inter-aural mismatch was varied by adjusting the simulated insertion depth (SID) to be 25 mm (SID25), 22 mm (SID22) and 19 mm (SID19) from the base of the cochlea. Lateralization was measured using headphones and head-related transfer functions (HRTFs). Baseline lateralization was measured for unprocessed speech (UN) delivered to the left ear to simulate SSD and for binaural performance with the acoustic ear combined with the 16-channel vocoders (UN+SID25, UN+SID22 and UN+SID19). After completing baseline measurements, subjects completed six lateralization training exercises with the UN+SID22 condition, after which performance was re-measured for all baseline conditions. Post-training performance was significantly better than baseline for all conditions (p < 0.05 in all cases), with no significant difference in training benefits among conditions. Given that there was no significant difference between the SSD and the SSD CI conditions before or after training, the results suggest that NH listeners were unable to integrate TFS and coarse spectro-temporal cues across ears for lateralization, and that inter-aural mismatch played a secondary role at best. While lateralization training may benefit SSD CI patients, the training may largely improve spectral analysis with the acoustic ear alone, rather than improve integration of acoustic and electric hearing.

Perceiving foreign-accented speech with decreased spectral resolution in single- and multiple-talker conditions.

To determine the effect of reduced spectral resolution on the intelligibility of foreign-accented speech, vocoder-processed sentences from native and Mandarin-accented English talkers were presented to listeners in single- and multiple-talker conditions. Reduced spectral resolution had little effect on native speech but lowered performance for foreign-accented speech, with a further decrease in multiple-talker conditions. Following the initial exposure, foreign-accented speech with reduced spectral resolution was less intelligible than unprocessed speech in both single- and multiple-talker conditions. Intelligibility improved with extended exposure, but only for single-talker conditions. Results indicate a perceptual impairment when perceiving foreign-accented speech with reduced spectral resolution.

Intelligibility Enhancement of Telephone Speech Using Gaussian Process Regression for Normal-to-Lombard Spectral Tilt Conversion

Noise in the environment can decrease the quality and intelligibility of a telephone conversation. This study focuses on the intelligibility enhancement of narrowband telephone speech in a near-end noise scenario using a postprocessing method based on normal-to-Lombard spectral tilt conversion. The proposed technique uses nonparallel, conversational normal, and Lombard speech together with Gaussian process regression in order to mimic the flattening of the spectral tilt that occurs in the production of natural speech in noisy conditions. The performance of the proposed method was evaluated in comparison to two reference methods, a fixed high-pass filter, and a baseline spectral tilt conversion, as well as in comparison to unprocessed speech in terms of intelligibility and listening preference in noisy conditions and in terms of pressedness in silent conditions. The results indicate that while the proposed technique provides a similar benefit in terms of intelligibility as fixed high-pass filtering, it is also able to produce a notable increase in pressedness. This suggests that the developed processing of the spectral tilt can compete with fixed high-pass filtering in intelligibility enhancement, but it is also able to convert speech to become perceptually closer to natural Lombard speech.

The Influence of Noise Reduction on Speech Intelligibility, Response Times to Speech, and Perceived Listening Effort in Normal-Hearing Listeners.

Single-microphone noise reduction leads to subjective benefit, but not to objective improvements in speech intelligibility. We investigated whether response times (RTs) provide an objective measure of the benefit of noise reduction and whether the effect of noise reduction is reflected in rated listening effort. Twelve normal-hearing participants listened to digit triplets that were either unprocessed or processed with one of two noise-reduction algorithms: an ideal binary mask (IBM) and a more realistic minimum mean square error estimator (MMSE). For each of these three processing conditions, we measured (a) speech intelligibility, (b) RTs on two different tasks (identification of the last digit and arithmetic summation of the first and last digit), and (c) subjective listening effort ratings. All measurements were performed at four signal-to-noise ratios (SNRs): −5, 0, +5, and +∞ dB. Speech intelligibility was high (>97% correct) for all conditions. A significant decrease in response time, relative to the unprocessed condition, was found for both IBM and MMSE for the arithmetic but not the identification task. Listening effort ratings were significantly lower for IBM than for MMSE and unprocessed speech in noise. We conclude that RT for an arithmetic task can provide an objective measure of the benefit of noise reduction. For young normal-hearing listeners, both ideal and realistic noise reduction can reduce RTs at SNRs where speech intelligibility is close to 100%. Ideal noise reduction can also reduce perceived listening effort.

Evaluating near-end listening enhancement in noise for normal-hearing and hearing-impaired listeners

Speech announcements in public spaces are often impaired by environmental noise, e.g., in train stations. In such conditions it is usually not possible to reduce the noise at the listener’s location, but the target speech can be pre-processed to enhance speech intelligibility (near-end listening enhancement, NELE). This contribution presents a series of listening tests to evaluate a NELE algorithm with normal-hearing and hearing-impaired listeners. The algorithm consists of several processing stages including dynamic range compression and frequency shaping. The algorithm either works under an equal-rms-power constraint (same rms level of processed and unprocessed speech) or additionally includes an adaptively controlled increase in rms level at constant peak level. Speech intelligibility was measured using the Oldenburg sentence test with unprocessed and processed stimuli in different types of background noise and SNRs. The results indicate that large benefits can be obtained even for constant speech level, but this benefit strongly depends on background noise type and SNR. The benefit is generally smaller for hearing-impaired than for normal-hearing listeners, but interindividual differences are quite large. All subjects benefit from the adaptive rms level increase, indicating that NELE algorithms with equal-rms-power constraints may not be sufficient for hearing-impaired listeners.

Can dual-carrier processing restore masking release in vocoded speech?

Speech processed to replace the original temporal fine structure (TFS) with tones or noise carriers (vocoder processing) are generally less intelligible than natural or unprocessed speech, especially if a background noise is present. Moreover, the poorer intelligibility associated with vocoder processing is typically larger if the background fluctuates over time. This deleterious effect of vocoder processing has led to the postulate that TFS cues play a critical role when listening into the dips in the background. Recently, we have proposed a technique to reintroduce synthetic TFS cues in vocoded speech using one carrier for the target and one carrier for the background. This “dual-carrier” approach allows sentence intelligibility with a speech masker to reach a level almost comparable to that of natural speech. The goal of the present study was to investigate the extent to which dual-carrier processing generally improves speech recognition in various noises or if it truly compensates for the loss of TFS cues, therefore engendering masking release, as does natural speech. Results comparing masking release for three processing conditions (single-carrier, dual-carrier, and natural speech) in five backgrounds (speech-shaped noise, speech-modulated noise, and 1, 2, or 8 talkers) will be discussed.

Evaluation of near-end speech enhancement under equal-loudness constraint for listeners with normal-hearing and mild-to-moderate hearing loss

Four algorithms designed to enhance the intelligibility of speech when noise is added after processing were evaluated under the constraint that the speech should have the same loudness before and after processing, as determined using a loudness model. The algorithms applied spectral modifications and two of them included dynamic-range compression. On average, the methods with dynamic-range compression required the least level adjustment to equate loudness for the unprocessed and processed speech. Subjects with normal-hearing (experiment 1) and mild-to-moderate hearing loss (experiment 2) were tested using unmodified and enhanced speech presented in speech-shaped noise (SSN) and a competing speaker (CS). The results showed (a) the algorithms with dynamic-range compression yielded the largest intelligibility gains in both experiments and for both types of background; (b) the algorithms without dynamic-range compression either yielded benefit only with the SSN or yielded no consistent benefit; (c) speech reception thresholds for unprocessed speech were higher for hearing-impaired than for normal-hearing subjects, by about 2 dB for the SSN and 6 dB for the CS. It is concluded that the enhancement methods incorporating dynamic-range compression can improve intelligibility under the equal-loudness constraint for both normal-hearing and hearing-impaired subjects and for both steady and fluctuating backgrounds.

Speech intelligibility improvement in noisy environments based on energy correlation in frequency bands

A new speech processing algorithm is proposed to improve speech intelligibility in noisy environments without increasing speech energy. The method improves the near-end speech intelligibility by optimizing the frame-based spectral energy correlation between clean speech and noisy modified speech with a power constraint. This algorithm is developed based on short-time objective intelligibility (STOI) measure, which predicts speech intelligibility in background noise according to the correlation of clean and noisy speeches. The proposed method is compared with unprocessed speech and two baseline methods using two objective intelligibility measures and an intelligibility listening test under various noisy conditions. Results show large intelligibility improvements with the proposed method over the unprocessed noisy speech. In addition, compared with the baseline methods, the proposed algorithm provides the best intelligibility scores for all noisy conditions in the STOI measure and for low signal-to-noise ratios in the speech intelligibility index. The word recognition results also show that the proposed algorithm performs better than the unprocessed speech and the reference methods. An objective quality measure is applied to investigate the speech quality of the introduced method, which is proven to insignificantly affect speech quality.

Single channel noise reduction system in low SNR

We propose a two stage noise reduction system for reducing background noise using single-microphone recordings in very low signal-to-noise ratio (SNR) based on Wiener filtering and ideal binary masking. The proposed system contains two stages. In first stage, the Wiener filtering with improved a priori SNR is applied to noisy speech for background noise reduction. In second stage, the ideal binary mask is estimated at every time---frequency channel by using pre-processed first stage speech and comparing the time---frequency channels against a pre-selected threshold T to reduce the residual noise. The time---frequency channels satisfying the threshold are preserved whereas all other time---frequency channels are attenuated. The results revealed substantial improvements in speech intelligibility and quality over that accomplished with the traditional noise reduction algorithms and unprocessed speech.

Near and Far Field Speech-in-Noise Intelligibility Improvements Based on a Time–Frequency Energy Reallocation Approach

An algorithm designed to enhance the intelligibility of speech signals before they are presented in noisy environments was evaluated. The processed and unprocessed speech had the same root-mean square level. Spectral energy was redistributed to increase the signal-to-noise ratio (SNR) in the mid- and high-frequency bands, while the softer segments of speech were increased in level by applying time-domain dynamic range compression. Noise level adaptation was introduced to increase the subjective quality of signals at high SNRs. Evaluations were conducted both in near field (headphones) and in far field (outdoor) conditions using listeners with normal hearing and two types of background. The results showed: (a) In the near field test, the proposed algorithm yielded significant intelligibility improvements relative to the unprocessed speech for both stationary and nonstationary backgrounds; (b) In the far field test, the proposed algorithm increased the intelligibility of unprocessed speech by a factor of seven at 200-m distance from the sound source; and (c) When the background level did not alter intelligibility, listeners preferred the quality of the speech processed by the noise-dependent version of the algorithm.

Adaptation to foreign-accented speech with decreased spectral resolution

Listening to foreign-accented speech requires greater effort, and intelligibility is lower than for native speech. We have recently shown that speech processed by a vocoder to simulate reduced spectral resolution imposes a substantial additional cost to intelligibility. The present study investigated perceptual adaptation to foreign-accented speech in conditions with reduced spectral resolution using a 9-channel tone vocoder compared to unprocessed speech. Preliminary results suggest that listeners can quickly adapt to foreign-accented speech with decreased spectral resolution. Native American English listeners achieved performance levels similar to those obtained with initial exposure to the same unprocessed speech from multiple talkers of Mandarin-accented English. However, with increased exposure, performance did not improve to the same degree found for unprocessed foreign-accented speech. Results indicate that the adaptation process is negatively impacted by the interaction of foreign accent and reduced spectral resolution, and suggests that special consideration should be given to the difficulties experienced by listeners with cochlear implants (whose devices provide limited spectral resolution) when they communicate with speakers who have a foreign accent.

Discrete cosine transform‐derived spectrum‐based speech enhancement algorithm using temporal‐domain multiband filtering

Conventional multiband speech enhancement involves splitting the spectrum into various frequency bins and performing speech enhancement in each band independently. However, owing to the pole-interaction problem in the spectral domain, estimation of clean speech from the formants, suppressed by the influence of the formants in the neighbouring bands, may result in poor quality. To reduce the influence of stronger formants over the neighbouring bands, in the current work, clean speech is estimated by filtering unprocessed speech in the temporal domain into various equivalent rectangular bandwidth based subbands followed by discrete cosine transform (DCT) based spectral speech enhancement in each band using spectral subtraction/minimum mean square error (MMSE). To further enhance speech, a spectral subtraction-based approach that incorporates band-specific weighting factor obtained using respective band signal-to-noise ratio (SNR), and an MMSE estimator that calculates apriori speech presence/absence probability based on local and global apriori SNR rather than a fixed/equiprobable value are proposed. The performance of the algorithms is evaluated using perceptual evaluation of speech quality and composite speech quality measure. It is observed that DCT-derived spectrum based temporal-domain multiband speech enhancement algorithm outperforms the existing techniques for car, babble, train, white, and factory noise in the 0–10 dB SNR levels.

Modulation masking and glimpsing of natural and vocoded speech during single-talker modulated noise: Effect of the modulation spectrum.

Compared to notionally steady-state noise, modulated maskers provide a perceptual benefit for speech recognition, in part due to preserved speech information during the amplitude dips of the masker. However, overlap in the modulation spectrum between the target speech and the competing modulated masker may potentially result in modulation masking, and thereby offset the release from energetic masking. The current study investigated masking release provided by single-talker modulated noise. The overlap in the modulation spectra of the target speech and the modulated noise masker was varied through time compression or expansion of the competing masker. Younger normal hearing adults listened to sentences that were unprocessed or noise vocoded to primarily limit speech recognition to the preserved temporal envelope cues. For unprocessed speech, results demonstrated improved performance with masker modulation spectrum shifted up or down compared to the target modulation spectrum, except for the most extreme time expansion. For vocoded speech, significant masking release was observed with the slowest masker rate. Perceptual results combined with acoustic analyses of the preserved glimpses of the target speech suggest contributions of modulation masking and cognitive-linguistic processing as factors contributing to performance.

Phase modification for increasing the intelligibility of telephone speech in near-end noise conditions – evaluation of two methods

In this study, two intelligibility-increasing post-processing methods based on the modification of the phase spectrum of speech are proposed for near-end noise conditions. One of the algorithms aims to reduce the dynamic range of the signal and take advantage of the energy gain resulting from amplitude normalization to increase the loudness, while the other algorithm is designed to sharpen the high-amplitude peaks in the time-domain signal generated by the periodic glottal excitation to make the speech sound more clear. Both methods are based on first modifying only the phase spectrum, after which the time-domain signal is computed using the inverse Fourier transform. Finally, the time-domain signal is amplitude normalized by scaling its sample values so that they occupy the original amplitude range of the processed frame. The performance of the proposed methods was evaluated by first comparing them to unprocessed speech using objective quality measures as well as subjective loudness and listening preference tests. Based on the results of these evaluations, the phase-modification methods were further compared to unprocessed speech and dynamic range compression using subjective word-error rate and quality tests. Both narrowband and wideband speech from several talkers were included in both evaluations. Both of the methods were able to increase loudness in some bandwidth conditions as well as outperform unprocessed speech and dynamic range compression in terms of intelligibility in high-noise levels. Both of the methods were rated lower in quality than unprocessed speech in clean conditions. In background noise, however, where intelligibility enhancement algorithms are mostly used, both methods achieved similar results to unprocessed speech in terms of listening preference in some of the bandwidth conditions tested.

Gender and rate effects on speech intelligibility

Older adults seeking hearing help often complain of particular difficulty understanding female voices. This contrasts with studies using young listeners with normal hearing in which female talkers have been found to be generally more intelligible than male talkers (e.g., Bradlow et al., 1996). Could some factor in addition to talker gender be causing older adults to have increased difficulty understanding female voices? Speech that has been time-compressed has been shown to be less intelligible than unprocessed speech (e.g., Gordon-Salant and Friedman, 2011), but few data exist to show whether an increased presentation rate causes an equal loss of intelligibility for male and female talkers. The present study will explore whether an increased playback rate has a greater negative effect on the intelligibility of speech produced by female versus male talkers. Subjects will listen to sentences produced by two female and two male speakers from the Utah Speaking Style Corpus presented at either their original rate or at an increased rate (1.5 times faster) and type out what they heard. The resulting data will show whether, for young normal-hearing listeners, an increased rate of speech affects the intelligibility of female talkers more than it affects the intelligibility of male talkers.