Band-limited Speech Research Articles

A lightweight speech enhancement network fusing bone- and air-conducted speech.

Air-conducted (AC) microphones capture the high-quality desired speech and ambient noise, whereas bone-conducted (BC) microphones are immune to ambient noise but only capture band limited speech. This paper proposes a speech enhancement model that leverages the merits of BC and AC speech. The proposed model takes the spectrogram of BC and AC speech as input and fuses them by an attention-based feature fusion module. The backbone network of the proposed model uses the fused signals to estimate mask of the target speech, which is then applied to the noisy AC speech to recover the target speech. The proposed model adopts a lightweight design of densely gated convolutional attention network (DenGCAN) as the backbone network, which contains encoder, bottleneck layers, and decoder. Furthermore, this paper improves an attention gate and integrates it into skip-connections of DenGCAN, which allows the decoder to focus on the key areas of the feature map extracted by the encoder. As the DenGCAN adopts self-attention mechanism, the proposed model has the potential to improve noise reduction performance at the expense of an increased input-output latency. Experimental results demonstrate that the enhanced speech of the proposed model achieves an average 1.870 wideband-PESQ improvement over the noisy AC speech.

Spectral degradations in the TIMIT, QuickSIN, NU-6, and other popular bandlimited speech materials

Many popular speech corpora were recorded decades ago using transducers, recording techniques, or post-processing techniques that resulted in spectral degradations. The use of such materials very likely has consequences for measures of speech perception, but this is not often considered in studies that use these materials. We analyzed several of these popular corpora to ascertain spectral degradations that may be present. Spectral degradations we frequently observed included limited extended high-frequency (EHF) (>8 kHz) content, and, for a few corpora, substantial low-frequency roll-off <500 Hz. Two corpora analyzed here, the TIMIT/PRESTO and NU-6, exhibited both high- and low-frequency degradations. Such spectral degradations will likely have an effect on behavioral measures because they do not match the high-fidelity speech signals listeners encounter in their everyday lives. This should be considered when selecting stimuli for speech perception experiments. [Work supported by NIH Grant R01-DC019745]

On the use of the TIMIT, QuickSIN, NU-6, and other widely used bandlimited speech materials for speech perception experiments.

The use of spectrally degraded speech signals deprives listeners of acoustic information that is useful for speech perception. Several popular speech corpora, recorded decades ago, have spectral degradations, including limited extended high-frequency (EHF) (>8 kHz) content. Although frequency content above 8 kHz is often assumed to play little or no role in speech perception, recent research suggests that EHF content in speech can have a significant beneficial impact on speech perception under a wide range of natural listening conditions. This paper provides an analysis of the spectral content of popular speech corpora used for speech perception research to highlight the potential shortcomings of using bandlimited speech materials. Two corpora analyzed here, the TIMIT and NU-6, have substantial low-frequency spectral degradation (<500 Hz) in addition to EHF degradation. We provide an overview of the phenomena potentially missed by using bandlimited speech signals, and the factors to consider when selecting stimuli that are sensitive to these effects.

The Influence of Sensation Level on Speech-Evoked Envelope Following Responses.

To evaluate sensation level (SL)-dependent characteristics of envelope following responses (EFRs) elicited by band-limited speech dominant in low, mid, and high frequencies. In 21 young normal hearing adults, EFRs were elicited by 8 male-spoken speech stimuli-the first formant, and second and higher formants of /u/, /a/ and /i/, and modulated fricatives, /∫/ and /s/. Stimulus SL was computed from behaviorally measured thresholds. At 30 dB SL, the amplitude and phase coherence of fricative-elicited EFRs were ~1.5 to 2 times higher than all vowel-elicited EFRs, whereas fewer and smaller differences were found among vowel-elicited EFRs. For all stimuli, EFR amplitude and phase coherence increased by roughly 50% for every 10 dB increase in SL between ~0 and 50 dB. Stimulus and frequency dependency in EFRs exist despite accounting for differences in audibility of speech sounds. The growth rate of EFR characteristics with SL is independent of stimulus and its frequency.

Envelope frequency following responses are stronger for high-pass than low-pass filtered vowels

Background: To assess hearing in response to speech, the envelope frequency following response (FFR) can be observed at the fundamental frequency of a vowel stimulus and its harmonics. FFRs are complex non-linear phenomena, which require better understanding for allowing robust inferences on the assessment of hearing and hearing aid fitting.Objectives: To evaluate the effect of stimulus bandwidth on FFR detection rates using filtered vowel stimuli with equal sound levels.Design: FFRs were collected whilst presenting repeated vowels (in consonant-vowel-consonant format) filtered into different bandwidths. Eighty stimuli per word were presented at 70 dB SPL LAeq through insert earphones with an inter-stimulus interval of 1 s. Responses were detected using frequency-domain Hotelling's T2 (HT2) tests for individual multiples of the fundamental frequency (F0) and for combinations of F0 multiples.Study sample: A total of 11 native English-speaking subjects with normal hearing thresholds.Results: Average detection rates are highest (69%) with stimuli high-pass filtered >1000 Hz, and significantly lower for low-pass filtered stimuli (40%).Conclusions: High-pass filtered vowels therefore appear to elicit stronger FFRs than low-pass filtered vowels at the same dB SPL LAeq. For testing hearing using band-limited speech, filtering effects (due to hearing loss, hearing aid setting or stimulus choice) on responses must be considered.

Patterns of Aided Loudness Growth in Experienced Adult Listeners with Early-Onset Severe-Profound Hearing Loss.

Individuals with early-onset severe-profound bilateral hearing loss (S/PHL) manifest diverse levels of benefit and satisfaction with hearing aids (HAs), even with prescriptive HA fitting. Such fittings incorporate normal loudness values, but little is known about aided loudness outcomes in this population and how those outcomes affect benefit or satisfaction. To describe aided loudness growth and satisfaction with aided listening in experienced adult HA users with S/PHL. The Contour Test of loudness perception was administered to listeners with S/PHL in the aided sound field using broadband speech, band-limited speech, and warble tones. Patterns and slopes of resultant loudness growth functions were referenced to sound field results from listeners with normal hearing (NH). S/PHL listeners also rated their aided listening satisfaction. It was expected that (1) most S/PHL listeners would demonstrate steeper than normal aided loudness growth, (2) loudness normalization would be associated with better high-frequency detection thresholds and speech recognition, and (3) closer approximation to normal would yield greater satisfaction. Participants were paid college-student volunteers: 23 with S/PHL, long-term aided listening experience, and new HAs; 15 with NH. Participants rated loudness on four ascending runs per stimulus (5-dB increments) using categories defined in 1997 by Cox and colleagues. The region between the 10th and 90th percentiles of the NH distribution constituted local norms against which location and slope of the S/PHL functions were examined over the range from Quiet to Loud-but-OK. S/PHL functions were categorized on the basis of their configurations (locations/slopes) relative to the norms. Pattern of aided loudness was normalized or within 5 dB of the normal region on 37% of trials with sufficient data for analysis. Only one of the 23 S/PHL listeners did not demonstrate Normal/Near-normal loudness on any trials. Four nonnormal patterns were identified: Steep (recruitment-like; 38% of trials); Shifted right, with normal growth rate (10%); Hypersensitive, with most intensities louder than normal (10%); and Shallow, with decreasing growth rate (7%). Listeners with high-frequency average thresholds above 100 dB hearing loss or no phonemic-based speech-discrimination skill were less likely to display normalized loudness. Slope was within norms for 52% of S/PHL trials, most also having a Normal/Near-normal growth pattern. Regardless of measured loudness results, all but four listeners with S/PHL reported satisfactory hearing almost always or most of the time with their HAs in designated priority need areas. The variety of aided loudness growth patterns identified reflects the diversity known to characterize individuals with early-onset S/PHL. Loudness rating at the validation stage of HA fit with these listeners is likely to reveal nonnormal loudness, signaling need for further HA adjustment. High satisfaction, however, despite nonnormal loudness growth, suggests that listeners with poor auditory speech recognition may benefit more from aided loudness that supports pattern perception (via the time-intensity waveform of speech), different from most current-day prescription fits.

Noise Robust Formant Frequency Estimation Method Based on Spectral Model of Repeated Autocorrelation of Speech

In this paper, a noise robust formant frequency estimation scheme is developed based on a spectral model matching algorithm. Considering the vocal tract as an autoregressive system, a spectral model of repeated autocorrelation function RACF of band-limited speech signal is proposed. It is shown that because of the repeated autocorrelation operation on band-limited signal, the proposed model can exhibit prominent formant characteristics. First from given noisy speech observations, an adaptive band selection criterion is developed. Next, on each resulting band-limited noisy speech signal, a repeated autocorrelation operation is carried out, which not only reduces the effect of noise but also strengthens the dominant poles corresponding to the formant frequencies. Finally, spectrum of the RACF is computed and instead of direct spectral peak picking, a model fitting scheme is introduced to find out model parameters which lead to formant estimation. The proposed algorithm has been tested on natural vowels as well as some naturally spoken sentences in the presence of different environmental noises. It is found that the proposed scheme provides better formant estimation accuracy in comparison to some of the existing methods at low levels of signal-to-noise ratio.

Estimation of glottal closure instants from degraded speech using a phase-difference-based algorithm

The estimation of glottal closure instants (GCIs) plays a vital role in several glottal synchronous applications, and may not be restricted to clean speech. This necessitates the development of a GCI estimation algorithm that performs as well on degraded speech as on clean speech. Degradations in speech may be in the form of spectral or temporal perturbations. This could result in several spurious discontinuities, as in noisy speech, excitations that are not well-defined, as in band-limited speech, aperiodicity and variations in amplitude, as in pathological speech, thereby making the task of identifying GCIs more challenging. In this regard, a conditional group-delay/phase-difference-based (PD) algorithm that was initially proposed for use on clean speech is extended to operate on degraded speech, specifically telephone, noisy, and pathological speech. The performance of this algorithm is compared with six existing algorithms, in terms of identification, false alarm, and miss rates, and identification accuracy. It is observed that the PD algorithm is robust to degradations in speech and performs better than or on par with existing algorithms in all cases considered. Further, it is also observed that unlike existing algorithms, the PD algorithm scarcely estimates GCIs in non-voiced regions and this is verified in terms of a new metric proposed in the paper, namely, the spurious instants rate in non-voiced regions.

Objective and subjective assessment of bandlimited signaling speech quality

Путем вычисления коэффициентов корреляции и построения карт соответствия, сопоставлены результаты объективного и субъективного оценивания качества речевых сигналов, искаженных из-за ограничения полосы частот. Показано, что карты соответствия, дополненные информацией о способе их построения, более полезны в прикладном плане, нежели коэффициенты корреляции, поскольку, во-первых, выполняют функцию калибровочного коэффициента, а во-вторых, допускают возможность нелинейной зависимости между объективными и субъективными оценками качества речевого сигнала. Библ. 7, рис. 5, табл. 3.

A new framework for robust speech recognition in complex channel environments

Channel distortion is one of the major factors which degrade the performances of automatic speech recognition (ASR) systems. Current compensation methods are generally based on the assumption that the channel distortion is a constant or slowly varying bias in an utterance or globally. However, this assumption is not sustained in a more complex circumstance, when the speech records being recognized are from many different unknown channels and have parts of the spectrum completely removed (e.g. band-limited speech). On the one hand, different channels may cause different distortions; on the other, the distortion caused by a given channel varies over the speech frames when parts of the speech spectrum are removed completely. As a result, the performance of the current methods is limited in complex environments. To solve this problem, we propose a unified framework in which the channel distortion is first divided into two subproblems, namely, spectrum missing and magnitude changing. Next, the two types of distortions are compensated with different techniques in two steps. In the first step, the speech bandwidth is detected for each utterance and the acoustic models are synthesized with clean models to compensate for spectrum missing. In the second step, the constant term of the distortion is estimated via the expectation-maximization (EM) algorithm and subtracted from the means of the synthesized model to further compensate for magnitude changing. Several databases are chosen to evaluate the proposed framework. The speech in these databases is recorded in different channels, including various microphones and band-limited channels. Moreover, to simulate more types of spectrum missing, various low-pass and band-pass filters are used to process the speech from the chosen databases. Although these databases and their filtered versions make the channel conditions more challenging for recognition, experimental results show that the proposed framework can substantially improve the performance of ASR systems in complex channel environments.

Behavioral measures of cochlear compression and temporal resolution as predictors of speech masking release in hearing-impaired listeners

Hearing-impaired (HI) listeners often show less masking release (MR) than normal-hearing listeners when temporal fluctuations are imposed on a steady-state masker, even when accounting for overall audibility differences. This difference may be related to a loss of cochlear compression in HI listeners. Behavioral estimates of compression, using temporal masking curves (TMCs), were compared with MR for band-limited (500-4000 Hz) speech and pure tones in HI listeners and age-matched, noise-masked normal-hearing (NMNH) listeners. Compression and pure-tone MR estimates were made at 500, 1500, and 4000 Hz. The amount of MR was defined as the difference in performance between steady-state and 10-Hz square-wave-gated speech-shaped noise. In addition, temporal resolution was estimated from the slope of the off-frequency TMC. No significant relationship was found between estimated cochlear compression and MR for either speech or pure tones. NMNH listeners had significantly steeper off-frequency temporal masking recovery slopes than did HI listeners, and a small but significant correlation was observed between poorer temporal resolution and reduced MR for speech. The results suggest either that the effects of hearing impairment on MR are not determined primarily by changes in peripheral compression, or that the TMC does not provide a sufficiently reliable measure of cochlear compression.

Allowance of Concave Part in Amplitude Envelope of Band-limited Speech Waveforms

Allowance of Concave Part in Amplitude Envelope of Band-limited Speech Waveforms

Time–Frequency Correlation-Based Missing-Feature Reconstruction for Robust Speech Recognition in Band-Restricted Conditions

Band-limited speech represents one of the most challenging factors for robust speech recognition. This is especially true in supporting audio corpora from sources that have a range of conditions in spoken document retrieval requiring effective automatic speech recognition. The missing-feature reconstruction method has a problem when applied to band-limited speech reconstruction, since it assumes the observations in the unreliable regions are always greater than the latent original clean speech. The approach developed here depends only on reliable components to calculate the posterior probability to mitigate the problem. This study proposes an advanced method to effectively utilize the correlation information of the spectral components across time and frequency axes in an effort to increase the performance of missing-feature reconstruction in band-limited conditions. We employ an F1 area window and cutoff border window in order to include more knowledge on reliable components which are highly correlated with the cutoff frequency band. To detect the cutoff regions for missing-feature reconstruction, blind mask estimation is also presented, which employs the synthesized band-limited speech model without secondary training data. Experiments to evaluate the performance of the proposed methods are accomplished using the SPHINX3 speech recognition engine and the TIMIT corpus. Experimental results demonstrate that the proposed time-frequency (TF) correlation based missing-feature reconstruction method is significantly more effective in improving band-limited speech recognition accuracy. By employing the proposed TF-missing feature reconstruction method, we obtain up to 14.61% of average relative improvement in word error rate (WER) for four available bandwidths with cutoff frequencies 1.0, 1.5, 2.0, and 2.5 kHz, respectively, compared to earlier formulated methods. Experimental results on the National Gallery of the Spoken Word (NGSW) corpus also show the proposed method is effective in improving band-limited speech recognition in real-life spoken document conditions.

Feature Compensation Techniques for ASR on Band-Limited Speech

Band-limited speech (speech for which parts of the spectrum are completely lost) is a major cause for accuracy degradation of automatic speech recognition (ASR) systems particularly when acoustic models have been trained with data with a different spectral range. In this paper, we present an extensive study of the problem of ASR of band-limited speech with full-bandwidth acoustic models. Our focus is mainly on band-limited feature compensation, covering even the case of time-varying band-limiting distortions, but we also compare this approach to more common model-side techniques (adaptation and retraining) and explore the combination of feature-based and model-side approaches. The feature compensation algorithms proposed are organized in a unified framework supported by a novel mathematical model of the impact of such distortions on Mel-frequency cepstral coefficient (MFCC) features. A crucial and novel contribution is the analysis made of the relative correlation of different elements in the MFCC feature vector for the cases of full-bandwidth and limited-bandwidth speech, which justifies an important modification in the feature compensation scheme. Furthermore, an intensive experimental analysis is provided. Experiments are conducted on real telephone channels, as well as artificial low-pass and bandpass filters applied over TIMIT data, and results are given for different experimental constraints and variations of the feature compensation method. Results for other well-known robustness approaches, such as cepstral mean normalization (CMN), model retraining, and model adaptation are also given for comparison. ASR performance with our approach is similar or even better than model adaptation, and we argue that in particular cases such as rapidly varying distortions, or limited computational or memory resources, feature compensation is more convenient. Furthermore, we show that feature-side and model-side approaches may be combined, outperforming any of those approaches alone.

Telephone Speech Perception by Mandarin-Speaking Cochlear Implantees

To evaluate Mandarin-speaking cochlear implant patients' understanding of telephone speech. Telephone speech was simulated by band-limiting broadband speech stimuli (300-3200 Hz) and adding Gaussian noise (35 dB signal-to-noise ratio). Recognition of multitalker vowels, consonants, voice gender, and Chinese tones was measured for both simulated telephone speech and broadband speech in fifteen Mandarin-speaking cochlear implant patients. Results showed no significant difference in Chinese tone recognition scores between broadband and telephone speech. However, mean recognition scores for vowels, consonants and voice gender were significantly lower with telephone speech. The effect of the limited telephone bandwidth on speech recognition was highly variable among subjects. Some subjects were more sensitive to high-frequency speech cues, resulting in a significant drop in performance with band-limited telephone speech, while other subjects were less sensitive to high-frequency speech cues, resulting in similar performance between broadband and band-limited telephone speech. These results suggest that the limited bandwidth negatively affects cochlear implant patients' understanding of telephone speech. Because the effect of band-limited speech was highly variable among subjects, the results also suggest that the contribution of high frequency information to speech recognition may vary significantly among cochlear implant patients. For patients who receive little benefit from high-frequency speech cues, speech processor adjustments may be necessary to access the additional cues provided in broadband speech.

Blind Feature Compensation for Time-Variant Band-Limited Speech Recognition

Mismatch in speech bandwidth between training and real operation greatly affects automatic speech recognition. This letter extends previous work on feature compensation of band-limited speech to establish a framework for blind compensation of speech data of unknown bandwidth, valid even when the distortion (band-limiting channel) changes rapidly and continuously in time. The available bandwidth of the input speech signal is automatically detected, and the band-limited feature vectors are compensated prior to being compared against full-bandwidth acoustic models. For a fixed bandwidth limitation, phoneme recognition performance using the proposed method is similar to that achieved by models adapted to match the distortion. However, compared to model adaptation approaches, this new approach can seamlessly be extended to rapidly time-varying conditions while maintaining low computational and memory costs

Recognition of Simulated Telephone Speech by Cochlear Implant Users

To evaluate cochlear implant users' understanding of telephone speech. Telephone speech was simulated by band-limiting broadband speech stimuli. Multitalker vowel, consonant, and sentence recognition was measured for both simulated telephone speech and broadband speech in 10 postlingually deafened adult cochlear implant users. The study was approved by the St. Vincent's Hospital institutional review board, and signed, informed consent was obtained from all participants. There was no significant difference in vowel recognition scores between broadband and telephone speech. However, mean consonant and sentence recognition scores were significantly poorer with telephone speech. The limited telephone bandwidth significantly reduced cochlear implant users' understanding of telephone speech. The effect of band-limited speech was highly variable, suggesting that the contribution of high-frequency information to speech recognition varied significantly among the cochlear implant users.

On artificial bandwidth extension of telephone speech

We present a signal processing algorithm to convert speech signals with “standard telephone” quality into 7 kHz wideband speech. A statistical approach based on a hidden Markov model (HMM) is used, which takes into account several features of the band-limited speech. The narrowband input signal is classified into a limited number of speech sounds (HMM states) for which the wideband spectral envelope is estimated using the pre-trained model. The enhanced speech exhibits a significantly improved quality without objectionable artifacts.

Generation of broadband speech from narrowband speech based on linear mapping

AbstractIn this paper, a method for generating broadband speech from band‐limited speech using spectral linear mapping is proposed. This method is based on LPC analysis and synthesis. It first extracts sound source information (residual waveforms) and vocal tract information (spectral envelopes) from input speech, performs linear mapping of the vocal tract information and nonlinear processing of the sound source information for broadband conversion, and finally generates broadband speech from both by LPC synthesis. The spectral envelope is made broadband by linear mapping, the spectral space is divided into a number of subspaces, and a narrowband spectrum is converted into a broadband spectrum by a conversion (convert) matrix of each subspace. A conversion (convert) matrix is estimated using training speech such that the mean square error between the postconversion spectrum and the target broadband spectrum is minimized. Spectral distortions of the proposed method were compared experimentally with codebook mapping and neural network methods and it has been verified that the proposed method with linear mapping gives performance not inferior to that of the other two methods. In addition, the proposed method has been verified to have the effect of giving a band feeling in subject evaluation tests. © 2002 Wiley Periodicals, Inc. Electron Comm Jpn Pt 2, 85(8): 44–53, 2002; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecjb.10065

A statistical model for robust integration of narrowband cues in speech

We investigate a statistical model for integrating narrowband cues in speech. The model is inspired by two ideas in human speech perception: (i) Fletcher’s hypothesis (1953) that independent detectors, working in narrow frequency bands, account for the robustness of auditory strategies, and (ii) Miller and Nicely’s analysis (1955) that perceptual confusions in noisy bandlimited speech are correlated with phonetic features. We apply the model to detecting the phonetic feature [+/−sonorant] that distinguishes vowels, approximants, and nasals (sonorants) from stops, fricatives, and affricates (obstruents). The model is represented by a multilayer probabilistic network whose binary hidden variables indicate sonorant cues from different parts of the frequency spectrum. We derive the Expectation-Maximization algorithm for estimating the model’s parameters and evaluate its performance on clean and corrupted speech.