Speech Spectrum Research Articles

Variation of Formant Frequency with Nepalese Vowels

The regions of speech spectrum in which the frequency corresponds to relatively large amplitude are known as formants. For any vocalic sounds, number of formants may occur in the frequency range 0 to 4000 Hz. The formant frequencies of speech sounds are directly depending up on the shape and size of vocal tract. The aim of study was to study the variation of formant frequency with Nepalese vowels. Ten Nepalese vowels word in initial position /VC/ as spoken three times by 10 male and 10 female Nepali speakers were recorded in system in the free field of partially acoustically treated room. PRRAT software is used to digitize and analyze the data. Linear predictive coding (LPC) spectra were obtained for each of vowels and formant frequencies were measured. By plotting curve between formant frequencies and vowels, explain their variation.

Speech enhancement using a DNN-augmented colored-noise Kalman filter

In this paper, we propose a new speech enhancement system using a deep neural network (DNN)-augmented colored-noise Kalman filter. In our system, both clean speech and noise are modelled as autoregressive (AR) processes, whose parameters comprise the linear prediction coefficients (LPCs) and the driving noise variances. The LPCs are obtained through training a multi-objective DNN that learns the mapping from the noisy acoustic features to the line spectrum frequencies (LSFs), while the driving noise variances are obtained by solving an optimization problem aiming to minimize the difference between the modelled and observed AR spectra of the noisy speech. The colored-noise Kalman filter with DNN estimated parameters is then applied to the noisy speech for denoising. Finally, a post-subtraction technique is adopted to further remove the residual noise in the Kalman-filtered speech. Extensive computer simulations show that the proposed speech enhancement system achieves significant performance gains when compared to conventional Kalman filter based algorithms as well as recent DNN-based methods under both seen and unseen noise conditions.

The Perception and Use of High-Frequency Speech Energy: Clinical and Research Implications

Purpose High-frequency speech energy (above approximately 4–5 kHz) is garnering substantial research attention. This review surveys recent evidence surrounding the presence and use of perceptual information in the high-frequency region. Additionally, clinical and research applications relevant to speech, language, and hearing professionals are discussed. Method Five databases were used during the search (Medline, CINAHL, WorldCat, ERIC, and Google Scholar). Criteria for study inclusion included (a) peer review, (b) utilization of high-frequency energy (above approximately 4 kHz) during the experimental tasks, and (c) were published from 2014 to present. Fifty-seven articles were included for review, and after further inspection, 13 met the inclusion criteria and were retained. Results Thirteen peer-reviewed studies provided evidence to support the supposition that important and useable acoustic cues exist in the high-frequency portion of the speech spectrum. Conclusions Considering the evidence discussed in this document, it is apparent that the high-frequency region contains additional perceptual cues than currently assumed. Specifically, acoustic cues regarding segmental information (vowel and consonant identification), individual speaker identity, and speaker sex are available for use by human listeners and automated machine recognition systems. Additionally, the high-frequency speech region may reduce listening effort and improve speech recognition in noisy listening conditions, particularly when the speech and noise are spatially separated. Therefore, clinicians and researchers should be aware of this information, which can inform clinical practice when fitting amplification devices for various clinical populations and experimental research for speech and hearing scientists.

Multi-objective long-short term memory recurrent neural networks for speech enhancement

Speech-in-noise perception is an important research problem in many real-world multimedia applications. The noise-reduction methods contributed significantly; however rely on a priori information about the noise signals. Deep learning approaches are developed for enhancing the speech signals in nonstationary noisy backgrounds and their benefits are evaluated for the perceived speech quality and intelligibility. In this paper, a multi-objective speech enhancement based on the Long-Short Term Memory (LSTM) recurrent neural network (RNN) is proposed to simultaneously estimate the magnitude and phase spectra of clean speech. During training, the noisy phase spectrum is incorporated as a target and the unstructured phase spectrum is transformed to its derivative that has an identical structure to corresponding magnitude spectrum. Critical Band Importance Functions (CBIFs) are used in training process to further improve the network performance. The results verified that the proposed multi-objective LSTM (MO-LSTM) successfully outscored the standard magnitude-aware LSTM (MA-LSTM), magnitude-aware DNN (MA-DNN), phase-aware DNN (PA-DNN), magnitude-aware GNN (MA-GNN) and magnitude-aware CNN (MA-CNN). Moreover, the proposed speech enhancement considerably improved the speech quality, intelligibility, noise-reduction and automatic speech recognition in changing noisy backgrounds, which is confirmed by the ANalysis Of VAriance (ANOVA) statistical analysis.

Emotion recognition from speech using deep learning on spectrograms

In speech emotion recognition, most emotional corpora generally have problems such as inconsistent sample length and imbalance of sample categories. Considering these problems, in this paper, a variable length input CRNN deep learning model based on Focal Loss is proposed for speech emotion recognition of anger, happiness, neutrality and sadness in IEMOCAP emotional corpus. In this model, Firstly, a variable-length strategy is introduced to input the speech spectra of the filled speech samples into CNN. Then the effective part of the input sequence is preserved and output by masking matrix and convolution layer. Thirdly, the effective output of input sequence is input into BiGRU network for learning. Finally, the focal loss is used for network training to control and adjust the contribution of various samples to the total loss. Compared with the traditional speech emotion recognition model, simulations show that our method can effectively improve the accuracy and performance of emotion recognition.

Non-intrusive speech quality prediction based on the blind estimation of clean speech and the i-vector framework

Output-based instrumental speech quality assessment relies only on the received (processed) signal to predict quality. Such methods are called non-intrusive and are crucial in speech applications where reference clean signals are not accessible. In this paper, we propose a new non-intrusive instrumental quality measure based on the similarity between two i-vectors. As the reference clean signal is not available, the reference i-vector representation cannot be extracted directly from it. Therefore, we propose the use of a clean speech Gaussian mixture model to estimate the clean speech spectra from its degraded speech spectrum counterpart. Next, the two respective i-vector representations are extracted and either the cosine or Eucledian similarity metrics are computed as a correlate of speech quality. Here, the clean speech model is trained using RASTA-filtered mel-frequency cepstral coefficients extracted from a pool of clean speech files, thus allowing us to attain a model of clean spectrum characteristics. The proposed method is evaluated on noisy, reverberant, and enhanced speech conditions. Experimental results show the proposed system providing higher correlations with perceptual speech quality than several benchmark non-intrusive measures, especially for noisy and enhanced speech.

High-frequency spectral shaping enhances glimpsing speech in noise for younger adults with normal hearing

Amplification is necessary to restore audibility to individuals with hearing impairment. However, frequency-specific amplification alters the spectrum of speech in addition to increasing the speech level. The current study investigated the effects of spectrum and speech level on speech recognition by younger adults with normal hearing. Sentence recognition was measured in quiet, speech-modulated noise, and steady-state noise. In addition, acoustic metrics were utilized to quantify the alterations resulting from spectral shaping. Acoustic metrics included (1) the Hearing Aid Speech Perception Index, in order to investigate changes to the temporal envelope, and (2) glimpse analyses, to quantify the proportion of time-frequency units that were glimpsed in a sentence. Results demonstrated that at high speech levels in quiet, spectral shaping was associated with lower sentence recognition. However, in all other conditions, and particularly in noise, spectral shaping enhanced sentence recognition. Acoustic metrics indicated that this enhancement in noise is associated with increases in the proportion and signal-to-noise ratio of glimpses, which resulted in improved preservation of the envelope and temporal fine structure of sentences in comparison to unshaped stimuli. Thus, spectral shaping is beneficial for speech recognition in noise for normal hearing participants due to enhanced speech glimpses.

Hearing and Neurodevelopmental Outcomes of Young Infants with Parechovirus-A and Enterovirus Meningitis: Cohort Study in Singapore Children and Literature Review

Abstract Parechovirus-A (PeV-A) and Enterovirus (EV) commonly cause childhood aseptic meningitis. Bacterial meningitis in children has been associated with devastating long-term sequelae. However, developmental outcomes are unclear in Parechovirus meningitis. This study aims to review the clinical findings and developmental outcomes of infants with PeV-A and EV meningitis. We performed a retrospective study of infants aged 90 days or younger being admitted to our hospital with PeV-A meningitis between November 2015 and July 2017, with positive cerebrospinal fluid (CSF) PeV-A PCR and negative blood and CSF bacterial cultures. Hearing and neurodevelopmental outcomes were compared with a previous cohort of infants aged 90 days or younger with EV meningitis admitted from January 2015 to December 2015. A total of 161 infants were included in our study, of which 68 infants (42.2%) had PeV-A meningitis and 93 infants (57.8%) had EV meningitis. We assessed their developmental outcome at 6 months, 1 year, and 2 years post-meningitis. At 2 years post-meningitis, three infants with PeV-A meningitis had developmental delay (5.5%), whereas none with EV meningitis had developmental delay. One patient had speech delay and autism spectrum disorder, while two had mild speech delay. When compared with our cohort of EV meningitis ≤90 days old, children with PeV-A meningitis ≤90 days old were more likely to have developmental delay 2 years post-meningitis (odds ratio 2.4, 95% confidence interval 2.0–3.0, p = 0.043). None of the patients with PeV-A or EV meningitis had sensorineural hearing loss or neurological sequelae, such as cortical blindness, oropharyngeal dysphagia, hydrocephalus, epilepsy, or cerebral palsy. Infants with PeV-A meningitis had a significant risk of developmental delay 2 years post-meningitis compared with those with EV meningitis. It is important to follow-up the developmental milestones of infants diagnosed with PeV-A meningitis for at least 2 years; and when they develop developmental delay, to ensure that they receive appropriate intervention.

Sinusoidal model-based hypernasality detection in cleft palate speech using CVCV sequence

Hypernasality in the speech of children with cleft palate is a consequence of velopharyngeal insufficiency. The spectral analysis of hypernasal speech shows the presence of nasal formants and anti-formants in the spectrum which affects the harmonic-intensity. The nasal formants increase whereas the anti-formants decrease the magnitude of harmonics around its location of addition. Hence, the spectrum of hypernasal and normal speech is different from each other. To capture the spectral difference, three features namely, normalized harmonic amplitude (NHA), harmonic amplitude ratio (HAR), and prominent harmonics frequency (PHF) are proposed in this work. NHA feature is the magnitude of harmonics after their normalization with respect to the maximum magnitude, HAR feature is the relative magnitude of harmonics with respect to their previous harmonics, and the PHF feature is the frequencies of prominent harmonics in the spectrum. The combination of three features gives an accuracy of 82.46%, 87.89%, 84.25% for /a/, /i/ and /u/ vowels respectively for the detection of hypernasality using support vector machine classifier.

Speech Signal Analysis in Phase Domain

Speech signal analysis based on the Phase Spectrum (PS) in the recent years becomes more popular to the researchers due to its attractive properties. Though some problem arises during its processing but appropriate modification gives the useful result. In this study, we proposed a new method of speech signal analysis based on its phase spectral representation. After analyzing the PS we modified it vocal tract dominated spectrum by utilizing the minimum-phase stable component of the speech spectrum. The modified signal holding only the phase spectral component is used for the parametric modeling of the speech signal to estimate the characteristics of vocal tract perfectly. This extracted feature was found to provide complementary evidence from the magnitude spectrum of speech but with better resolution. According to perceptual analysis, the PS takes precedence over the magnitude spectrum. This research utilizes the Group Delay (GD) spectrum as a representative of the PS because of its meaningful characteristic. All pole modeling is performed on the signal evaluated from the GD spectrum to find the resonances of the vocal tract system accurately. The effectiveness of the method is tested by synthesizing some vowels over a range of pitch periods from low to high pitched speech. The validity of the proposed method is also verified by plotting the formant contour on the spectrogram of a sentence from the TIMIT database and standard F2-F1 plot of natural speech spoken by male and female speakers. The proposed method performs better than the state-of-the-art methods.

Vision perceptually restores auditory spectral dynamics in speech

Visual speech facilitates auditory speech perception, but the visual cues responsible for these benefits and the information they provide remain unclear. Low-level models emphasize basic temporal cues provided by mouth movements, but these impoverished signals may not fully account for the richness of auditory information provided by visual speech. High-level models posit interactions among abstract categorical (i.e., phonemes/visemes) or amodal (e.g., articulatory) speech representations, but require lossy remapping of speech signals onto abstracted representations. Because visible articulators shape the spectral content of speech, we hypothesized that the perceptual system might exploit natural correlations between midlevel visual (oral deformations) and auditory speech features (frequency modulations) to extract detailed spectrotemporal information from visual speech without employing high-level abstractions. Consistent with this hypothesis, we found that the time-frequency dynamics of oral resonances (formants) could be predicted with unexpectedly high precision from the changing shape of the mouth during speech. When isolated from other speech cues, speech-based shape deformations improved perceptual sensitivity for corresponding frequency modulations, suggesting that listeners could exploit this cross-modal correspondence to facilitate perception. To test whether this type of correspondence could improve speech comprehension, we selectively degraded the spectral or temporal dimensions of auditory sentence spectrograms to assess how well visual speech facilitated comprehension under each degradation condition. Visual speech produced drastically larger enhancements during spectral degradation, suggesting a condition-specific facilitation effect driven by cross-modal recovery of auditory speech spectra. The perceptual system may therefore use audiovisual correlations rooted in oral acoustics to extract detailed spectrotemporal information from visual speech.

An Iterative Graph Spectral Subtraction Method for Speech Enhancement

In this paper, we investigate the application of graph signal processing (GSP) theory in speech enhancement. We first propose a set of shift operators to construct graph speech signals, and then analyze their spectrum in the graph Fourier domain. By leveraging the differences between the spectrum of graph speech and graph noise signals, we further propose the graph spectral subtraction (GSS) method to suppress the noise interference in noisy speech. Moreover, based on GSS, we propose the iterative graph spectral subtraction (IGSS) method to further improve the speech enhancement performance. Our experimental results show that the proposed operators are suitable for graph speech signals, and the proposed methods outperform the traditional basic spectral subtraction (BSS) method and iterative basic spectral subtraction (IBSS) method in terms of both signal-to-noise ratios (SNR) and mean Perceptual Evaluation of Speech Quality (PESQ).

One Solution of Extension of Mel-Frequency Cepstral Coefficients Feature Vector for Automatic Speaker Recognition

One extension of feature vector for automatic speaker recognition is considered in this paper. The starting feature vector consisted of 18 mel-frequency cepstral coefficients (MFCCs). Extension was done with two additional features derived from the spectrum of the speech signal. The main idea that generated this research is that it is possible to increase the efficiency of automatic speaker recognition by constructing a feature vector which tracks a real perceived spectrum in the observed speech. Additional features are based on the energy maximums in the appropriate frequency ranges of observed speech frames. In experiments, accuracy and equal error rate (EER) are compared in the case when feature vectors contain only 18 MFCCs and in cases when additional features are used. Recognition accuracy increased by around 3%. Values of EER show smaller differentiation but the results show that adding proposed additional features produced a lower decision threshold. These results indicate that tracking of real occurrences in the spectrum of the speech signal leads to more efficient automatic speaker recognizer. Determining features which track real occurrences in the speech spectrum will improve the procedure of automatic speaker recognition and enable avoiding complex models.

Neural Tracking of Band-Limited Sine-Wave Speech in Normal Hearing and Cochlear Implant Listeners

A sine-wave speech represents complex speech with a limited number of sinusoidal components. Functionally, its strategy draws a close similarity to the signal processing strategy in cochlear implant (CI) which transmits speech envelope information in limited number of channels. In this study, we investigated how synchronous cortical activities to speech envelope relates to the speech intelligibility of sine-wave speech in different bandwidths with both normal hearing (NH) and CI listeners. 12 NH and four CI participants were recruited. We divided our NH participants into two groups: 1) Six NH listeners, 2) six NH listening to CI simulation synthesized using a noise vocoder. In our third group, 3) four CI users, one of them participated in only behavioral tests. Neural tracking was obtained using multi-channel electroencephalogram (EEG) system and cross-checked with behavioural performance, speech perception scores and speech quality ratings. Our result showed that intelligible sine-wave speech can be built with a small number of sinusoidal components selected from the original speech spectrum. Our cross-correlation analysis between cortical activities and speech envelope fluctuation showed an increasing trend in their synchrony with sine-wave speech built using more sinusoidal components for both group 2) and 3). Cortical entrainment to speech envelope is more observable in CI users compared to NH listened to CI simulated sine-wave speech. Our result have implications for understanding of neural tracking in terms of spectrally degraded speech perception in NH and CI individuals. Keywords: Sinusoidal model, electroencephalogram (EEG), cortical entrainment, speech intelligibility, hearing prosthesis. Read more →

A survey on heliumspeech communications in saturation diving

With the development of society and the progress of technology, more and more ocean activities are carried out. It results in booming of deep-sea diving. The use of helium-oxygen mixture (as a kind of breathing gas) solves the physiological problems of divers in saturated diving, but it brings about the Heliumspeech voice communication problem, the drop of speech intelligibility. There is no doubt that the effective speech communication must be provided for supporting the life and work of divers in deep-sea. This paper describes the mechanism of forming heliumspeech, discusses the effects of pressure and helium environment on the speech spectrum, compares the pros and cons of the time-domain and frequency-domain unscrambling techniques, shows the challenges in heliumspeech communications. Finally, it briefly introduces the deep learning, and points out that deep learning/machine learning may be a perfectly unscrambling technique.

3D Multiple Sound Source Localization by Proposed Cuboids Nested Microphone Array in Combination with Adaptive Wavelet-Based Subband GEVD

Sound source localization is one of the applicable areas in speech signal processing. The main challenge appears when the aim is a simultaneous multiple sound source localization from overlapped speech signals with an unknown number of speakers. Therefore, a method able to estimate the number of speakers, along with the speaker’s location, and with high accuracy is required in real-time conditions. The spatial aliasing is an undesirable effect of the use of microphone arrays, which decreases the accuracy of localization algorithms in noisy and reverberant conditions. In this article, a cuboids nested microphone array (CuNMA) is first proposed for eliminating the spatial aliasing. The CuNMA is designed to receive the speech signal of all speakers in different directions. In addition, the inter-microphone distance is adjusted for considering enough microphone pairs for each subarray, which prepares appropriate information for 3D sound source localization. Subsequently, a speech spectral estimation method is considered for evaluating the speech spectrum components. The suitable spectrum components are selected and the undesirable components are denied in the localization process. The speech information is different in frequency bands. Therefore, the adaptive wavelet transform is used for subband processing in the proposed algorithm. The generalized eigenvalue decomposition (GEVD) method is implemented in sub-bands on all nested microphone pairs, and the probability density function (PDF) is calculated for estimating the direction of arrival (DOA) in different sub-bands and continuing frames. The proper PDFs are selected by thresholding on the standard deviation (SD) of the estimated DOAs and the rest are eliminated. This process is repeated on time frames to extract the best DOAs. Finally, K-means clustering and silhouette criteria are considered for DOAs classification in order to estimate the number of clusters (speakers) and the related DOAs. All DOAs in each cluster are intersected for estimating the position of the 3D speakers. The closest point to all DOA planes is selected as a speaker position. The proposed method is compared with a hierarchical grid (HiGRID), perpendicular cross-spectra fusion (PCSF), time-frequency wise spatial spectrum clustering (TF-wise SSC), and spectral source model-deep neural network (SSM-DNN) algorithms based on the accuracy and computational complexity of real and simulated data in noisy and reverberant conditions. The results show the superiority of the proposed method in comparison with other previous works.

Objective Characterization of Phonation Type Using Amplitude of Flow Glottogram Pulse and of Voice Source Fundamental

Phonation type, a phonatory dimension ranging from hypofunctional/breathy to hyperfunctional/pressed, is important both from a clinical and acoustical point of view; hyperfunctional voice can lead to voice disorders and hypofunctional voice reduces text intelligibility. Five male singers sang diminuendo sequences of the syllable /pae/ and three of them also produced speech or singing samples with different phonation types which were rated for phonatory pressedness by a panel of voice experts. The sequences were analyzed by inverse filtering and the associated subglottal pressures were estimated as the oral pressures during the /p/ occlusion. The results showed strong quasi-linear correlations between the peak-to-peak amplitude of the flow glottogram (henceforth pulse amplitude) and mean subglottal pressure, mean airflow, and the level difference between the first and second voice source partials L1-L2. These correlations were found to vary systematically with phonation type. Regardless of phonation type, the correlation between the pulse amplitude and the amplitude of the voice source fundamental frequency was very close to 1.0. The level difference between the first and second spectrum partials L1-L2 in narrow-band long-term-average spectra of speech and singing was found to vary systematically with phonation type in a manner related to voice experts' ratings of perceived degree of pressedness. The findings support the assumption that the combination of subglottal pressure and level of the voice source fundamental is useful for an objective measure of phonation type.

Multi-Task Learning U-Net for Single-Channel Speech Enhancement and Mask-Based Voice Activity Detection

In this paper, a multi-task learning U-shaped neural network (MTU-Net) is proposed and applied to single-channel speech enhancement (SE). The proposed MTU-based SE method estimates an ideal binary mask (IBM) or an ideal ratio mask (IRM) by extending the decoding network of a conventional U-Net to simultaneously model the speech and noise spectra as the target. The effectiveness of the proposed SE method was evaluated under both matched and mismatched noise conditions between training and testing by measuring the perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI). Consequently, the proposed SE method with IRM achieved a substantial improvement with higher average PESQ scores by 0.17, 0.52, and 0.40 than other state-of-the-art deep-learning-based methods, such as the deep recurrent neural network (DRNN), SE generative adversarial network (SEGAN), and conventional U-Net, respectively. In addition, the STOI scores of the proposed SE method are 0.07, 0.05, and 0.05 higher than those of the DRNN, SEGAN, and U-Net, respectively. Next, voice activity detection (VAD) is also proposed by using the IRM estimated by the proposed MTU-Net-based SE method, which is fundamentally an unsupervised method without any model training. Then, the performance of the proposed VAD method was compared with the performance of supervised learning-based methods using a deep neural network (DNN), a boosted DNN, and a long short-term memory (LSTM) network. Consequently, the proposed VAD methods show a slightly better performance than the three neural network-based methods under mismatched noise conditions.

Automatic speaker profiling from short duration speech data

Many paralinguistic applications of speech demand the extraction of information about the speaker characteristics from as little speech data as possible. In this work, we explore the estimation of multiple physical parameters of the speaker from the short duration of speech in a multilingual setting. We explore different feature streams for age and body build estimation derived from the speech spectrum at different resolutions, namely – short-term log-mel spectrogram, formant features and harmonic features of the speech. The statistics of these features over the speech recording are used to learn a support vector regression model for speaker age and body build estimation. The experiments performed on the TIMIT dataset show that each of the individual features is able to achieve results that outperform previously published results in height and age estimation. Furthermore, the estimation errors from these different feature streams are complementary, which allows the combination of estimates from these feature streams to further improve the results. The combined system from short audio snippets achieves a performance of 5.2 cm, and 4.8 cm in Mean Absolute Error (MAE) for male and female respectively for height estimation. Similarly in age estimation the MAE is of 5.2 years, and 5.6 years for male, and female speakers respectively. We also extend the same physical parameter estimation to other body build parameters like shoulder width, waist size and weight along with height on a dataset we collected for speaker profiling. The duration analysis of the proposed scheme shows that the state of the art results can be achieved using only around 1–2 s of speech data. To the best of our knowledge, this is the first attempt to use a common set of features for estimating the different physical traits of a speaker.

A Study on the Korean Conversation Speech Level and Spectrum in Sound-Treated Environment

Purpose: Several studies have reported the differences between Korean and other countries’ speech in long-term average speech spectrum (LTASS) and band importance function. Authors tried to identify the conversational speech level, the resulting spectrum, and the LTASS for Korean. The purpose of this study was to support the production of a Korean-type hearing aid fitting formula to effectively improve the sound quality of hearing aids and the communication abilities with hearing aids, and to standardize the sound stimuli required to measure the performance of hearing aids. Methods: A total of 73 participants with normal hearing and with no specific voice and language deficits was voluntarily recruited from capital, Gyeongsang and Jeolla areas. The conversational speech level was measured by vocalizing ‘soft,’ ‘moderate,’ ‘raised,’ and ‘loud’ at a distance of 1.0 m from the speaker. And LTASS was measured by vocalizing it at a distance of 0.2 m from the speaker. Results: There was a difference in the mean of males and females in the conversation level, but no significant regional differences were shown. The conversational speech level corresponding to 30th, 65th, and 99th percentiles was 59.67, 64.74, and 79.07 dB sound pressure level, respectively. And the speech spectrum of 30th, 65th, and 99th percentile and LTASS showed in different forms from the international speech test signal. Conclusion: The results of this study should help to calculate the Korean type hearing aid fitting formula and should be used as the basic data to determine the characteristics of the sound stimuli when measuring the performance of the hearing aid.