Speech Spectral Components Research Articles

Multistage progressive learning based speech enhancement using time-frequency attentive squeezed temporal convolutional networks

Speech enhancement (SE) is an important method for improving speech quality and intelligibility in noisy environments. An effective speech enhancement model depends on precise modelling of the long-range dependencies of noisy speech. Several recent studies have examined ways to enhance speech by capturing the long-term contextual information. For speech enhancement, the time-frequency (T-F) distribution of speech spectral components is also important, but is usually ignored in these studies. The multi-stage learning method is an effective way to integrate various deep-learning modules at the same time. The benefit of multi-stage training is that the optimization target can be iteratively updated stage by stage. In this paper speech enhancement is investigated by multi-stage learning using a multistage structure in which time-frequency attention (TFA) blocks are followed by stacks of squeezed temporal convolutional networks (S-TCN) with exponentially increasing dilation rates. To reinject original information into later stages, a feature fusion (FF) block is inserted at the input of later stages to reduce the possibility of speech information being lost in the early stages. The S-TCN blocks are responsible for temporal sequence modelling task. The time-frequency attention (TFA) is a simple but effective network module that explicitly exploits position information to generate a 2D attention map to characterise the salient T-F distribution of speech by using two branches, time-frame attention and frequency attention in parallel. A set of utterances from the LibriSpeech and Voicebank databases are used to evaluate the performance of the proposed SE. Extensive experiments have demonstrated that the proposed model consistently improves the performance over existing baselines across two widely used objective metrics such as PESQ and STOI. The average PESQ and STOI for proposed model are boosted by a factor of 41.7% and 5.4% for Libri speech dataset, 36.10% and 3.1% for Voice bank dataset as compared to noisy speech. Additionally, we explored the generalization of the proposed TFA-S-TCN model across different speech datasets through cross data base analysis. A significant improvement in system robustness to noise is also shown by our evaluation results using the TFA module.

A Time-Frequency Attention Module for Neural Speech Enhancement

Speech enhancement plays an essential role in a wide range of speech processing applications. Recent studies on speech enhancement tend to investigate how to effectively capture the long-term contextual dependencies of speech signals to boost performance. However, these studies generally neglect the time-frequency (T-F) distribution information of speech spectral components, which is equally important for speech enhancement. In this paper, we propose a simple yet very effective network module, which we term the T-F attention (TFA) module, that uses two parallel attention branches, i.e., time-frame attention and frequency-channel attention, to explicitly exploit position information to generate a 2-D attention map to characterise the salient T-F speech distribution. We validate our TFA module as part of two widely used backbone networks (residual temporal convolution network and Transformer) and conduct speech enhancement with four most popular training objectives. Our extensive experiments demonstrate that our proposed TFA module consistently leads to substantial enhancement performance improvements in terms of the five most widely used objective metrics, with negligible parameter overheads. In addition, we further evaluate the efficacy of speech enhancement as a front-end for a downstream speech recognition task. Our evaluation results show that the TFA module significantly improves the robustness of the system to noisy conditions.

The Modulatory Effects of Visual Speech on Auditory Speech Perception: A Multi-Modal Investigation of How Vision Alters the Temporal, Spatial and Spectral Components of Speech

The Modulatory Effects of Visual Speech on Auditory Speech Perception: A Multi-Modal Investigation of How Vision Alters the Temporal, Spatial and Spectral Components of Speech

Efficient harmonic peak detection of vowel sounds for enhanced voice activity detection

Voice activity detection (VAD) involves discriminating speech segments from background noise and is a critical step in numerous speech-related applications. However, distinguishing speech from noise based on the properties of noise is fallible, because it is difficult to predict and characterise the noise occurring in real life. In this study, the authors instead focus on the intrinsic characteristics of speech. The harmonic peaks of vowel sounds have higher energies than the other spectral components of speech and are the speech features most likely to survive in most cases of severe noise. Therefore, the energy differences between harmonic peaks and other spectral features show promise for enabling robust VAD. To exploit this feature, the harmonic peaks must be accurately located. For this purpose, this study proposes an efficient harmonic peak location detection (HPD) method. Based on extensive experiments conducted in the presence of various noise types and signal-to-noise ratios, we found that VAD with the proposed HPD approach outperforms existing VAD methods and does so with reasonable computational cost and higher robustness.

A GMM/HMM model for reconstruction of missing speech spectral components for continuous speech recognition

This paper presents a method for reconstructing unreliable spectral components of speech signals using the statistical distributions of the clean components. Our goal is to model the temporal patterns in speech signal and take advantage of correlations between speech features in both time and frequency domain simultaneously. In this approach, a hidden Markov model (HMM) is first trained on clean speech data to model the temporal patterns which appear in the sequences of the spectral components. Using this model and according to the probabilities of occurring noisy spectral component at each states, a probability distributions for noisy components are estimated. Then, by applying maximum a posteriori (MAP) estimation on the mentioned distributions, the final estimations of the unreliable spectral components are obtained. The proposed method is compared to a common missing feature method which is based on the probabilistic clustering of the feature vectors and also to a state of the art method based on sparse reconstruction. The experimental results exhibits significant improvement in recognition accuracy over a noise polluted Persian corpus.

Electrophysiologic Assessment of Auditory Training Benefits in Older Adults.

Older adults often exhibit speech perception deficits in difficult listening environments. At present, hearing aids or cochlear implants are the main options for therapeutic remediation; however, they only address audibility and do not compensate for central processing changes that may accompany aging and hearing loss or declines in cognitive function. It is unknown whether long-term hearing aid or cochlear implant use can restore changes in central encoding of temporal and spectral components of speech or improve cognitive function. Therefore, consideration should be given to auditory/cognitive training that targets auditory processing and cognitive declines, taking advantage of the plastic nature of the central auditory system. The demonstration of treatment efficacy is an important component of any training strategy. Electrophysiologic measures can be used to assess training-related benefits. This article will review the evidence for neuroplasticity in the auditory system and the use of evoked potentials to document treatment efficacy.

Two‐microphone subband noise reduction scheme with a new noise subtraction parameter for speech quality enhancement

An improved subband noise reduction technique is proposed for two-microphone voice communication systems. The technique aims to enhance the speech quality by utilising a subband structure with different noise reduction schemes for different frequency bands. In the low-frequency band where dominant cues of speech spectral components are usually located and the noise signals from the two channels are mainly correlated, the spectral subtraction method, together with a new variable noise subtraction parameter, is employed so that the noise attenuation performance and speech distortion are controllable. In the high-frequency band where less-dominant frequency information of speech spectrum is located, the modified cross-spectral subtraction technique is utilised to remove the high-frequency decorrelated noise spectral components. Extensive comparisons among various noise reduction techniques based on computer simulations demonstrate that the proposed two-microphone subband noise reduction scheme achieves excellent noise attenuation performance while preserving the speech quality.

A neural basis of speech-in-noise perception in older adults.

We investigated a neural basis of speech-in-noise perception in older adults. Hearing loss, the third most common chronic condition in older adults, is most often manifested by difficulty understanding speech in background noise. This trouble with understanding speech in noise, which occurs even in individuals who have normal-hearing thresholds, may arise, in part, from age-related declines in central auditory processing of the temporal and spectral components of speech. We hypothesized that older adults with poorer speech-in-noise (SIN) perception demonstrate impairments in the subcortical representation of speech. In all participants (28 adults, age 60-73 yr), average hearing thresholds calculated from 500 to 4000 Hz were ≤ 25 dB HL. The participants were evaluated behaviorally with the Hearing in Noise Test (HINT) and neurophysiologically using speech-evoked auditory brainstem responses recorded in quiet and in background noise. The participants were divided based on their HINT scores into top and bottom performing groups that were matched for audiometric thresholds and intelligent quotient. We compared brainstem responses in the two groups, specifically, the average spectral magnitudes of the neural response and the degree to which background noise affected response morphology. In the quiet condition, the bottom SIN group had reduced neural representation of the fundamental frequency of the speech stimulus and an overall reduction in response magnitude. In the noise condition, the bottom SIN group demonstrated greater disruption in noise, reflecting reduction in neural synchrony. The role of brainstem timing is particularly evident in the strong relationship between SIN perception and quiet-to-noise response correlations. All physiologic measures correlated with SIN perception. Adults in the bottom SIN group differed from the audiometrically matched top SIN group in how speech was neurally encoded. The strength of subcortical encoding of the fundamental frequency appears to be a factor in successful speech-in-noise perception in older adults. Given the limitations of amplification, our results suggest the need for inclusion of auditory training to strengthen central auditory processing in older adults with SIN perception difficulties.

A Novel Mask Estimation Method Employing Posterior-Based Representative Mean Estimate for Missing-Feature Speech Recognition

This paper proposes a novel mask estimation method for missing-feature reconstruction to improve speech recognition performance in various types of background noise conditions. A conventional mask estimation method based on spectral subtraction degrades performance, due to incorrect estimation of the noise signal which fails to accurately represent the variations of background noise during the incoming speech utterance. The proposed mask estimation method utilizes a Posterior-based Representative Mean (PRM) estimate for determining the reliability of the input speech spectral components, which is obtained as a weighted sum of the mean parameters of the speech model using the posterior probability. To obtain the noise-corrupted speech model, a model combination method is employed, which was proposed in our previous study for a feature compensation method. Experimental results demonstrate that the proposed mask estimation method provides more separable distributions for the reliable/unreliable component classifier compared to the conventional mask estimation method. The recognition performance is evaluated using the Aurora 2.0 framework over various types of background noise conditions and the CU-Move real-life in-vehicle corpus. The performance evaluation shows that the proposed mask estimation method is considerably more effective at increasing speech recognition performance in various types of background noise conditions, compared to the conventional mask estimation method which is based on spectral subtraction. By employing the proposed PRM-based mask estimation for missing-feature reconstruction, we obtain +23.41% and +9.45% average relative improvements in word error rate for all four types of noise conditions and CU-Move corpus, respectively, compared to conventional mask estimation methods.

Speech Enhancement Algorithm Using Sub band Two Step Decision Directed Approach with Adaptive Weighting factor and Noise Masking Threshold

Problem statement: Speech Enhancement plays an important role in any of the speech processing systems like speech recognition, mobile communication, hearing aid. Approach: In this work, human perceptual auditory masking effect is incorporated into the single channel speech enhancement algorithm. The algorithm is based on a criterion by which the audible noise may be masked rather than being attenuated and thereby reducing the chance of distortion to speech. The basic decision directed approach is for efficient reduction of musical noise, that includes the estimation of the a priori SNR which is a crucial parameter of the spectral gain, follows the a posteriori SNR with a delay of one frame in speech frames. In this work a simple adaptive speech enhancement technique, using an adaptive sigmoid type function to determine the weighting factor of the TSDD algorithm is employed based on a sub band approach. In turn the spectral estimate is used to obtain a perceptual gain factor. Results: Objective and subjective measures like SNR, MSE, IS distance and were obtained, which shows the ability of the proposed method for efficient enhancement of noisy speech Conclusion/Recommendations: Performance assessment shows that our proposal can achieve a more significant noise reduction and a better spectral estimation of weak speech spectral components from a noisy signal as compared to the conventional speech enhancement algorithm.

Data Hiding in Audio Using Random Orthogonal Matrices

ABSTRACT In this paper, speech scrambling is done to secure the data and at the same time covert information is embedded in scrambled speech that can be transmitted through communication channel. The technique performs encryption by scrambling the spectral components of speech using random orthogonal matrices. At the receiver end, hidden message and original speech are recovered by decoding and decrypting the scrambled signal, respectively. Performance of technique is evaluated in clean and noisy channel environments.

A Speech Enhancement Algorithm Based on a Chi MRF Model of the Speech STFT Amplitudes

A speech enhancement algorithm that takes advantage of the time and frequency dependencies of speech signals is presented in this paper. The above dependencies are incorporated in the statistical model using concepts from the theory of Markov Random Fields. In particular, the speech short-time Fourier transform (STFT) amplitude samples are modeled with a novel Chi Markov Random Field prior, which is then used for the development of an estimator based on the Iterated Conditional Modes method. The novel prior is also coupled with a dasiaharmonicpsila neighborhood, which apart from the immediately adjacent samples on the time frequency plane, also considers samples which are one pitch frequency apart, so as to take advantage of the rich structure of the voiced speech time frames. Additionally, central to the development of the algorithm is the adaptive estimation of the weights that determine the interaction between neighboring samples, which allows the restoration of weak speech spectral components, while maintaining a low level of uniform residual noise. Results that illustrate the improvements achieved with the proposed algorithm, and a comparison with other established speech enhancement schemes are also given.

Integrated Speech Enhancement Method Using Noise Suppression and Dereverberation

This paper proposes a method for enhancing speech signals contaminated by room reverberation and additive stationary noise. The following conditions are assumed. 1) Short-time spectral components of speech and noise are statistically independent Gaussian random variables. 2) A room's convolutive system is modeled as an autoregressive system in each frequency band. 3) A short-time power spectral density of speech is modeled as an all-pole spectrum, while that of noise is assumed to be time-invariant and known in advance. Under these conditions, the proposed method estimates the parameters of the convolutive system and those of the all-pole speech model based on the maximum likelihood estimation method. The estimated parameters are then used to calculate the minimum mean square error estimates of the speech spectral components. The proposed method has two significant features. 1) The parameter estimation part performs noise suppression and dereverberation alternately. (2) Noise-free reverberant speech spectrum estimates, which are transferred by the noise suppression process to the dereverberation process, are represented in the form of a probability distribution. This paper reports the experimental results of 1500 trials conducted using 500 different utterances. The reverberation time RT <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">60</sub> was 0.6 s, and the reverberant signal to noise ratio was 20, 15, or 10 dB. The experimental results show the superiority of the proposed method over the sequential performance of the noise suppression and dereverberation processes.

The a priori SDR Estimation Techniques with Reduced Speech Distortion for Acoustic Echo and Noise Suppression

This paper proposes an enhanced method for estimating the a priori Signal-to-Disturbance Ratio (SDR) to be employed in the Acoustic Echo and Noise Suppression (AENS) system for full-duplex hands-free communications. The proposed a priori SDR estimation technique is modified based upon the Two-Step Noise Reduction (TSNR) algorithm to suppress the background noise while preserving speech spectral components. In addition, a practical approach to determine accurately the Echo Spectrum Variance (ESV) is presented based upon the linear relationship assumption between the power spectrum of far-end speech and acoustic echo signals. The ESV estimation technique is then employed to alleviate the acoustic echo problem. The performance of the AENS system that employs these two proposed estimation techniques is evaluated through the Echo Attenuation (EA), Noise Attenuation (NA), and two speech distortion measures. Simulation results based upon real speech signals guarantee that our improved AENS system is able to mitigate efficiently the problem of acoustic echo and background noise, while preserving the speech quality and speech intelligibility.

Enhancement of speech signals separated from their convolutive mixture by FDICA algorithm

This paper presents a novel method for the enhancement of independent components of mixed speech signal segregated by the frequency domain independent component analysis (FDICA) algorithm. The enhancement algorithm proposed here is based on maximum a posteriori (MAP) estimation of the speech spectral components using generalized Gaussian distribution (GGD) function as the statistical model for the time-frequency series of speech (TFSS) signal. The proposed MAP estimator has been used and evaluated as the post-processing stage for the separation of convolutive mixture of speech signals by the fixed-point FDICA algorithm. It has been found that the combination of separation algorithm with the proposed enhancement algorithm provides better separation performance under both the reverberant and non-reverberant conditions.

Relaxed statistical model for speech enhancement and a priori SNR estimation

In this paper, we propose a statistical model for speech enhancement that takes into account the time-correlation between successive speech spectral components. It retains the simplicity associated with the Gaussian statistical model, and enables the extension of existing algorithms to noncausal estimation. The sequence of speech spectral variances is a random process, which is generally correlated with the sequence of speech spectral magnitudes. Causal and noncausal estimators for the a priori SNR are derived in agreement with the model assumptions and the estimation of the speech spectral components. We show that a special case of the causal estimator degenerates to a decision-directed estimator with a time-varying frequency-dependent weighting factor. Experimental results demonstrate the improved performance of the proposed algorithms.

/spl beta/-order MMSE spectral amplitude estimation for speech enhancement

This paper proposes /spl beta/-order minimum mean-square error (MMSE) speech enhancement approach for estimating the short time spectral amplitude (STSA) of a speech signal. We analyze the characteristics of the /spl beta/-order STSA MMSE estimator and the relation between the value of /spl beta/ and the spectral amplitude gain function of the MMSE method. We further investigate the effectiveness of a range of fixed-/spl beta/ values in estimating STSA based on the MMSE criterion, and discuss how the /spl beta/ value could be adapted using the frame signal-to-noise ratio (SNR). The performance of the proposed speech enhancement approach is then evaluated through spectrogram inspection, objective speech distortion measures and subjective listening tests using several types of noise sources from the NOISEX-92 database. Evaluation results show that our approach can achieve a more significant noise reduction and a better spectral estimation of weak speech spectral components from a noisy signal as compared to many existing speech enhancement algorithms.

Spectral fissioning in phonemic transformations.

Listeners presented with a repeated sequence of brief (30- to 100-msec) steady-state vowels hear phonemic transformations--they cannot identify the vowels, but they perceive two simultaneous utterances that differ in both phonemic content and timbre (Warren, Bashford & Gardner, 1990). These utterances consist of either English words or syllables that occur in English words. In the present study, we attempted to determine whether the two percepts represent alternative interpretations of the same formant structures, or whether different portions of the vowels are used for each verbal organization. It was found that separate spectral regions are employed for each verbal form: Components below 1500 Hz were generally used for one form, and components above 1500 Hz for the other. Hypotheses are offered concerning the processes responsible for the verbal organization of the vowel sequences and for the splitting into two spectrally limited forms. It appears that the tendency to organize spectral regions separately competes with, and can overcome, the tendency to integrate the different spectral components of speech into a single auditory image. A contralateral induction paradigm was used in a procedure designed to quantitatively evaluate these opposing forces of spectral fission and fusion.

The application of principal-components spectral analysis to sensory aids for the hearing impaired

In order to present speech information through an alternate sensory modality—such as touch or vision—extensive processing of the speech signal is required. This processing should represent a large amount of speech information with a small number of slowly varying parameters. These parameters should be relatively speaker independent, robust in the presence of noise, and closely linked to the perceptually important features of speech spectra. One such set of parameters are speech spectral principal components. In this paper we will discuss the application of principal-components analysis of speech spectra to speech recoding for use with sensory aids. A brief overview of the analysis procedure will be included. An interpretation of principal components, and a comparison with other speech parameters such as speech formants, will be presented. Both potential benefits and drawbacks of this analysis procedure will be discussed. In addition, the implementation of a vowel articulation training aid based on principal components will be presented. Although the vowel training aid utilizes a visual display format, with color as a primary information-bearing parameter in the display, the procedure used for obtaining principal components applies equally well to a tactile aid.

Block encoding of speech spectral principal components

A Karhunen‐Loeve (KL) series expansion was used to block encode speech spectral principal components as a function of time. Each of ten principal components was first obtained as a linear combination of 20 speech spectral band energies. Using a fixed block length of ten frames (0.128 s), the KL basis vectors were computed separately for various speakers for each principal component. However, the optimal KL basis vector set was essentially the same for each principal component and for the different speakers. The basis vector set also closely resembled a cosine basis vector set. Approximately 94% of the variance of the principal components was accounted for by five (out of ten) basis vectors. Speech was synthesized using the KL basis vectors for block encoding of ten‐frame blocks of principal components. Informal listening tests indicate that very little information is lost using five basis vectors. These results indicate that speech spectral principal components, particularly the low‐ordered ones which ref...