Speech Audio Process Research Articles

Deep learning restores speech intelligibility in multi-talker interference for cochlear implant users

Cochlear implants (CIs) do not offer the same level of effectiveness in noisy environments as in quiet settings. Current single-microphone noise reduction algorithms in hearing aids and CIs only remove predictable, stationary noise, and are ineffective against realistic, non-stationary noise such as multi-talker interference. Recent developments in deep neural network (DNN) algorithms have achieved noteworthy performance in speech enhancement and separation, especially in removing speech noise. However, more work is needed to investigate the potential of DNN algorithms in removing speech noise when tested with listeners fitted with CIs. Here, we implemented two DNN algorithms that are well suited for applications in speech audio processing: (1) recurrent neural network (RNN) and (2) SepFormer. The algorithms were trained with a customized dataset (∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim$$\\end{document} 30 h), and then tested with thirteen CI listeners. Both RNN and SepFormer algorithms significantly improved CI listener’s speech intelligibility in noise without compromising the perceived quality of speech overall. These algorithms not only increased the intelligibility in stationary non-speech noise, but also introduced a substantial improvement in non-stationary noise, where conventional signal processing strategies fall short with little benefits. These results show the promise of using DNN algorithms as a solution for listening challenges in multi-talker noise interference.

Speech emotion detection based on MFCC and CNN-LSTM architecture

Emotion detection techniques have been applied to multiple cases mainly from facial image features and vocal audio features, of which the latter aspect is disputed yet not only due to the complexity of speech audio processing but also the difficulties of extracting appropriate features. Part of the SAVEE and RAVDESS datasets are selected and combined as the dataset, containing seven sorts of common emotions (i.e. happy, neutral, sad, anger, disgust, fear, and surprise) and thousands of samples. Based on the Librosa package, this paper processes the initial audio input into waveplot and spectrum for analysis and concentrates on multiple features including MFCC as targets for feature extraction. The hybrid CNN-LSTM architecture is adopted by virtue of its strong capability to deal with sequential data and time series, which mainly consists of four convolutional layers and three long short-term memory layers. As a result, the architecture achieved an accuracy of 61.07% comprehensively for the test set, among which the detection of anger and neutral reaches a performance of 75.31% and 71.70% respectively. It can also be concluded that the classification accuracy is dependent on the properties of emotion to some extent, with frequently-used and distinct-featured emotions having less probability to be misclassified into other categories. Emotions like surprise whose meaning depends on the specific context are more likely to confuse with positive or negative emotions, and negative emotions also have a possibility to get mixed with each other.

Speech spectrogram display in pseudo-color based on non-linear transform method

Concerning that the human eye to color resolution is much higher than gray-scale difference,this paper proposed a new method to show the spectrogram by C# procedure,which focused on the mapping transformation function and built up the relationship between spectral magnitude and color,to make pseudo-color display come true,and also used it for speech spectrogram display and audio signal processing interface design.Meanwhile in order to achieve better analysis and learning effects,the authors added audio play function so as to watch the spectrogram while listening to the corresponding audio.This method,with simple programming,flexible interface design and less demand on the equipment,can be realized on ordinary PC.

Relevant spectro‐temporal modulations for robust speech and nonspeech classification.

Robust speech/non‐speech classification is an important step in a variety of speech processing applications. For example, in speech and speaker recognition systems designed to work in real world environments, a robust discrimination of speech from other sounds is an essential pre‐processing step. Auditory‐based features at multiple‐scales of time and spectral resolution have been shown to be very useful for the speech/non‐speech classification task [Mesgarani et al., IEEE Trans. Speech Audio Process. 10, 504–516 (2002)]. The features used are computed using a biologically inspired auditory model that maps a given sound to a high‐dimensional representation of its spectro‐temporal modulations (mimicking the various stages taking place along the auditory pathway from the periphery all the way to the primary auditory cortex). In this work, we analyze the contribution of different temporal and spectral modulations for robust speech/non‐speech classification. The results suggest the temporal modulations in the range 12–22 Hz, and spectral modulations in the range 1.5–4 cycles/octave are particularly useful to achieve the robustness in highly noisy and reverberant environments.

An improved new error estimation algorithm for optimal filter lengths for stereophonic acoustic echo cancellation

In this paper, we propose an new error estimate algorithm (NEEA) for stereophonic acoustic echo cancellation (SAEC) that is based on the error estimation algorithm (EEA) in [Nguyen-Ky T, Leis J, Xiang W. An improved error estimate algorithm for stereophonic acoustic echo cancellation system. In: International conference on signal processing and communication systems, ICSPCS’2007, Australia; December 2007]. In the EEA and NEEA, with the minimum error signal fixed, we compute the filter lengths so that the error signal may approximate the minimum error signal. When the echo paths change, the adaptive filter automatically adjusts the filter lengths to the optimum values. We also investigate the difference between the adaptive filter lengths. In contrast with the conclusions in [Khong AWH, Naylor PA. Stereophonic acoustic echo cancellation employing selective-tap adaptive algorithms. IEEE Trans Audio, Speech, Lang Process 2006;14(3):785–96, Gansler T, Benesty J. Stereophonic acoustic echo cancellation and two channel adaptive filtering: an overview. Int J Adapt Control Signal Process 2000;4:565–86, Benesty J, Gansler T. A multichannel acoustic echo canceler double-talk detector based on a normalized cross-correlation matrix. Acoust Echo Noise Control 2002;13(2):95–101, Gansler T, Benesty J. A frequency-domain double-talk detector based on a normalized cross-correlation vector. Signal Process 2001;81:1783–7, Eneroth P, Gay SL, Gansler T, Benesty J. A real-time implementation of a stereophonic acoustic echo canceler. IEEE Trans. Speech Audio Process 2001;9(5):513–23, Gansler T, Benesty J. New insights into the stereophonic acoustic echo cancellation problem and an adaptive nonlinearity solution. IEEE Trans. Speech Audio Process 2002; 10(5):257–67, Benesty J, Gansler T, Morgan DR, Sondhi MM, Gay SL. Advances in network and acoustic echo cancellation. Berlin: Springer-Verlag; 2001], our simulation results have shown that the filter lengths can be different. Our simulation results also confirm that the NEEA is better than EEA and SM-NLMS algorithm in terms of echo return loss enhancement.

Cepstral domain interpretations of line spectral frequencies

In this paper, we derive a relationship between a linear prediction (LP) polynomial and its corresponding line spectral frequencies (LSFs) in cepstral domain. We first obtain the cepstral representations of the symmetric and the antisymmetric polynomials constructed from the LP polynomial. The sum of these cepstra corresponds to the pseudo-cepstrum [H.K. Kim, S.H. Choi, H.S. Lee, On approximating line spectral frequencies to LPC cepstral coefficients, IEEE Trans. Acoust. Speech Audio Process. 8(2) (2000) 195–199]. Finally, LSFs can be computed by matching these two cepstra. Additionally, we derive the recursive relations between LSFs and the LPC coefficients, as well as between LSFs and LPC cepstral coefficients.

Speech intelligibility improvement using convolutive blind source separation assisted by denoising algorithms

The present study is concerned with the blind source separation (BSS) of speech and speech-shaped noise sources. All recordings were carried out in an anechoic chamber using a dummy head (two microphones, one in each ear). The program which implements the algorithm for BSS of convolutive mixtures introduced by Parra and Spence [Parra, L., Spence, C., 2000a. Convolutive blind source separation of non-stationary sources. IEEE Trans. Speech Audio Process. 8(3), 320–327 (US Patent US6167417)] was used to separate out the signals. In the postprocessing phase two different denoising algorithms were used. The first was based on a minimum mean-square error log-spectral amplitude estimator [Ephraim, E., Malah, D., 1985. Speech enhancement using a minimum mean-square error log-spectral amplitude estimator. IEEE Trans. Speech Audio Process. ASSP-33(2), 443–445], while the second one was based on Wiener filter in which the concept of an a priori signal-to-noise estimation presented by Ephraim (as mentioned above) was applied [Scalart, P., Filho, J.V., 1996. Speech enhancement based on a priori signal to noise estimation. IEEE Internat. Conf. Acoust. Speech Signal Process. 1, 629–632]. Non-sense word tests were used as a target speech in both cases while one or two disturbing sources were used as interferences. The speech intelligibility before and after the BSS was measured for three subjects with audiologically normal hearing. Next the speech signal after BSS was denoised and presented to the same listeners. The results revealed some ambiguities caused by the insufficient number of microphones compared to the number of sound sources. For one disturbance only, the intelligibility improvement was significant. However, when there were two disturbances and the target speech, the separation was much poorer. The additional denoising, as could be expected, raises the intelligibility slightly. Although the BSS method requires more research on optimization, the results of the investigation imply that it may be applied to hearing aids in the future.

Speech enhancement by residual domain constrained optimization

A new algorithm for the enhancement of speech corrupted by additive noise is proposed. This algorithm estimates the linear prediction residuals of the clean speech using a constrained optimization criterion. The signal distortion is minimized in the residual domain subject to a constraint on the average power of the noise residuals. Enhanced speech is obtained by exciting the time-varying all-pole synthesis filter with the estimated residuals of the clean speech. The proposed method was tested with speech corrupted by both white Gaussian and colored noise. The enhancement performances were evaluated in terms of segmental signal-to-noise ratio (SNR) and ITU-PESQ scores. Experimental results indicate our method yields better enhancement results than a former residual-weighting scheme [Yegnanarayana, B., Avendano, C., Hermansky, H., Murthy P.S., 1999. Speech enhancement using linear prediction residual. Speech Commun. 28, 25–42]. The proposed method also achieves better noise reduction than the time-domain subspace method [Ephraim, Y., Van Trees, H.L., 1995. A signal subspace approach for speech enhancement. IEEE Trans. Speech Audio Process. 3, 251–266] on real world colored noise.

Language identification using acoustic log-likelihoods of syllable-like units

Automatic spoken language identification (LID) is the task of identifying the language from a short utterance of the speech signal uttered by an unknown speaker. The most successful approach to LID uses phone recognizers of several languages in parallel [Zissman, M.A., 1996. Comparison of four approaches to automatic language identification of telephone speech. IEEE Trans. Speech Audio Process. 4 (1), 31–44]. The basic requirement to build a parallel phone recognition (PPR) system is segmented and labeled speech corpora. In this paper, a novel approach is proposed for the LID task which uses parallel syllable-like unit recognizers, in a frame work similar to the PPR approach in the literature. The difference is that the sub-word unit models for each of the languages to be recognized are generated in an unsupervised manner without the use of segmented and labeled speech corpora. The training data of each of the languages is first segmented into syllable-like units and language-dependent syllable-like unit inventory is created. These syllable-like units are then clustered using an incremental approach. This results in a set of syllable-like units models for each language. Using these language-dependent syllable-like unit models, language identification is performed based on accumulated acoustic log-likelihoods. Our initial results on the Oregon Graduate Institute Multi-language Telephone Speech Corpus [Muthusamy, Y.K., Cole, R.A., Oshika, B.T., 1992. The OGI multi-language telephone speech corpus. In: Proceedings of Internat. Conf. Spoken Language Process., October 1992, pp. 895–898] show that the performance is 72.3%. We further show that if only a subset of syllable-like unit models that are unique (in some sense) are considered, the performance improves to 75.9%.

A comparison of front-ends for bitstream-based ASR over IP

Automatic speech recognition (ASR) is called to play a relevant role in the provision of spoken interfaces for IP-based applications. However, as a consequence of the transit of the speech signal over these particular networks, ASR systems need to face two new challenges: the impoverishment of the speech quality due to the compression needed to fit the channel capacity and the inevitable occurrence of packet losses. In this framework, bitstream-based approaches that obtain the ASR feature vectors directly from the coded bitstream, avoiding the speech decoding process, have been proposed ([S.H. Choi, H.K. Kim, H.S. Lee, Speech recognition using quantized LSP parameters and their transformations in digital communications, Speech Commun. 30 (4) (2000) 223–233. A. Gallardo-Antolín, C. Pelàez-Moreno, F. Díaz-de-María, Recognizing GSM digital speech, IEEE Trans. Speech Audio Process., to appear. H.K. Kim, R.V. Cox, R.C. Rose, Performance improvement of a bitstream-based front-end for wireless speech recognition in adverse environments, IEEE Trans. Speech Audio Process. 10 (8) (2002) 591–604. C. Peláez-Moreno, A. Gallardo-Antolín, F. Díaz-de-María, Recognizing voice over IP networks: a robust front-end for speech recognition on the WWW, IEEE Trans. Multimedia 3(2) (2001) 209–218], among others) to improve the robustness of ASR systems. LSP (Line Spectral Pairs) are the preferred set of parameters for the description of the speech spectral envelope in most of the modern speech coders. Nevertheless, LSP have proved to be unsuitable for ASR, and they must be transformed into cepstrum-type parameters. In this paper we comparatively evaluate the robustness of the most significant LSP to cepstrum transformations in a simulated VoIP (voice over IP) environment which includes two of the most popular codecs used in that network (G.723.1 and G.729) and several network conditions. In particular, we compare ‘pseudocepstrum’ [H.K. Kim, S.H. Choi, H.S. Lee, On approximating Line Spectral Frequencies to LPC cepstral coefficients, IEEE Trans. Speech Audio Process. 8 (2) (2000) 195–199], an approximated but straightforward transformation of LSP into LP cepstral coefficients, with a more computationally demanding but exact one. Our results show that pseudocepstrum is preferable when network conditions are good or computational resources low, while the exact procedure is recommended when network conditions become more adverse.

Auditory analysis for speech recognition based on physiological models

To address the limitations of traditional cepstrum or LPC based front-end processing methods for automatic speech recognition, more elaborate methods based on physiological models of the human auditory system may be used to achieve more robust speech recognition in adverse environments. For this purpose, a modified version of a model of the primary auditory cortex featuring a three dimensional mapping of auditory spectra [Wang and Shamma, IEEE Trans. Speech Audio Process. 3, 382–395 (1995)] is adopted and investigated for its use as an improved front-end processing method. The study is conducted in two ways: first, by relating the model’s redundant representation to traditional spectral representations and showing that the former not only encompasses information provided by the latter, but also reveals more relevant information that makes it superior in describing the identifying features of speech signals; and second, by observing the statistical features of the representation for various classes of sound to show how different identifying features manifest themselves as specific patterns on the cortical map, thereby becoming a place-coded data set on which detection theory could be applied to simulate auditory perception and cognition.

Relative intelligibility of dynamically extracted transient versus steady-state components of speech

Consonants are recognized to dominate higher frequencies of the speech spectrum and to carry more information than vowels, but both demonstrate quasi-steady state and transient components, such as vowel to consonant transitions. Fixed filters somewhat separate these effects, but probably not optimally, given diverse words, speakers, and situations. To enhance the transient characteristics of speech, this study used time-varying adaptive filters [Rao and Kumaresan, IEEE Trans. Speech Audio Process. 8, 240–254 (2000)], following high-pass filtering at 700 Hz (well-known to have minimal effect on intelligibility), to extract predominantly steady-state components of speech material (CVC words, NU-6). The transient component was the difference between the sum of the filter outputs and the original signal. Psychometric functions were determined in five subjects with and without background noise and fitted by ogives. The transient components averaged filtered speech energy, but PBmax was not significantly different (nonparametric ANOVA) from that of either the original or highpass filtered speech. The steady-state components yielded significantly lower PBmax (p 3D 0.003) despite their much greater energy, as expected. These results suggest a potential approach to dynamic enhancement of speech intelligibility. [Work supported by ONR.]

Analysis and enhancement of country singing

The study of human singing has focused extensively on the analysis of voice characteristics. At the same time, a substantial body of work has been under study aimed at modeling and synthesizing the human voice. The work on which we report brings together some key analysis and synthesis principles to create a new model for digitally improving the perceived quality of an average singing voice. The model presented employs an analysis‐by‐synthesis overlap‐add (ABS‐OLA) sinusoidal model, which in the past has been used for the analysis and synthesis of speech, in combination with a spectral model of the vocal tract. The ABS‐OLA sinusoidal model for speech has been shown to be a flexible, accurate, and computationally efficient representation capable of producing a natural‐sounding singing voice [E. B. George and M. J. T. Smith, Trans. Speech Audio Processing 5, 389–406 (1997)]. A spectral model infused in the ABS‐OLA uses Generalized Gaussian functions to provide a simple framework which enables the precise modification of spectral characteristics while maintaining the quality and naturalness of the original voice. Furthermore, it is shown that the parameters of the new ABS‐OLA can accommodate pitch corrections and vocal quality enhancements while preserving naturalness and singer identity. Examples of enhanced country singing will be presented.

An adaptive short-term postfilter based on pseudo-cepstral representation of line spectral frequencies

We propose an adaptive short-term postfilter for speech coders by incorporating the properties of the pseudo-cepstrum [IEEE Trans. Speech Audio Process. 8 (2) (2000) 195]. Since the proposed postfilter implicitly has a characteristic of tilt compensation, it does not require an additional tilt compensation filter as conventional techniques. We derive a relationship between the postfilter parameters so that the postfilter has a minimum phase distortion, which helps simplify the tuning procedure for the parameters. We also show that the postfilter can be successfully implemented with a lower order. By applying this postfilter to several international speech coding standards, we reduce the complexity while obtaining a comparable performance to conventional approaches.

A new distortion measure for spectral quantization based on the LSF intermodel interlacing property

The line spectral frequencies (LSFs) extracted from successive analysis orders are interlaced with each other. This intermodel interlacing property gives a new relationship between the closeness of LSFs and their spectral sensitivities, which motivates a new weighting function for LSF distortion measurement. By applying this new weighting function to LSF quantization, we have achieved a significantly better performance than the conventional heuristic weighting functions in both clean and noise environments. In addition, the proposed weighting function gives better performance than the weighting function based on a high-rate approximation (Gardner weighting (GW)) [W.R. Gardner, B.D. Rao, IEEE Trans. Speech Audio Processing 3 (5) (1995) 367] in noise environments while their performances are comparable in clean environments. Moreover, the complexity of the proposed weighting function is much lower than that of the GW function.

Comments on "Efficient training algorithms for HMMs using incremental estimation"

The paper entitled Efficient training algorithms for HMMs using incremental estimation by Gotoh et al. (IEEE Trans. Speech Audio Processing, vol.6, p.539-48, Nov. 1998) investigated expectation maximization (EM) procedures that increase training speed. The claim of Gotoh et al. that these procedures are generalized EM (Dempster et al. 1977) procedures is shown to be incorrect in the present paper. We discuss why this is so, provide an example of nonmonotonic convergence to a local maximum in likelihood, and outline conditions that guarantee such convergence.

A maximum model distance approach for HMM-based speech recognition

This paper presents a new approach for HMM-training which is based on the maximum model distance (MMD) criterion for different similar utterances. This approach differs from the traditional maximum likelihood (ML) approach in that the ML only considers the likelihood P( O ν | λ ν ) for a single utterance, while the MMD compares the likelihood P( O ν | λ ν ) against those similar utterances and maximizes their likelihood differences. Theoretical and practical issues concerning this approach are investigated. In addition, the corrective training [Bahl, Brown, de Souza and Mercer, IEEE Trans. Speech Audio Process. 1(1), (1993)] of the MMD was also included in this paper and we proved that the corrective training proposed by Bahl et al. (1993) is a special case of our MMD approach. Both speaker-dependent and multi-speaker experiments have been carried out on the Chinese An-set syllables and also the 599 most common utterances from the TIMIT database. Experimental results showed that significant error reduction can be achieved through the proposed approach.

Predicting the perception of words from the perception of phonemes

The present paper explores the degree to which the perception of CVC syllables can be factored into the perception of its constituent phonemes. Categorization experiments with synthetic speech typically manifest such factorability to a remarkable degree (regardless of lexical status of the syllables.) Such factorability is also compatible with intelligibility studies of English CVCs [ e.g., A. Boothroyd and S. Nittrouer, J. Acoust. Soc. Am. 84, 101–114 (1988)]. These show that the identification of nonsense syllables can be predicted as the product of the probabilities of identification of its phoneme parts, while word identification is systematically higher. Simulation studies are reported here involving a ‘‘factored’’ perceptual model that consists of a set of phoneme-likelihood estimators whose outputs are modulated by prior probabilities related to lexical status. This model can approximate the patterns observed in human perception. Simulations were also run with nonfactorable models, where syllables and words involve information about unique stimulus properties that cannot be predicted from their constituent phonemes. Consistent with a conjecture of Allen [J. Allen, IEEE Trans. Speech Audio Process. 2, 567–577 (1994)], such syllable template models do not produce behavior compatible with human performance. [Work supported by SSHRC.]

Robust waveform interpolation for the entire speech signal

In waveform interpolation (WI), a speech signal is reconstructed by the concatenation of infinitesimal segments of an evolving characteristic waveform, which is obtained by interpolation over time [W. B. Kleijn, IEEE Trans. Speech Audio Process. 1, 386–399 (1993)]. WI leads to efficient coding of voiced speech, but current implementations switch to CELP for nonperiodic signals. The WI paradigm is extended to provide an effective basis for the coding of voiced and unvoiced speech and background noise. Prototype waveforms are extracted every 2.5 ms. At this high rate the WI analysis–synthesis system results in transparent speech quality. Each prototype waveform is decomposed into a slowly evolving waveform (SEW), obtained by convolution in time with a 40-ms smoothing window, and a remainder, the rapidly evolving waveform (REW). Because of its low bandwidth, a low bit rate suffices for the SEW (additional processing lowers the bit rate further), while the REW requires only a rough statistical description (e.g., its phase spectrum can be randomized). The new paradigm facilitates efficient, robust speech coding in the range 2–8 kb/s.