Throat Microphone Speech Research Articles

Enhancement of throat microphone speech using Empirical Mode Decomposition (EMD)

Enhancement of throat microphone speech using Empirical Mode Decomposition (EMD)

Improving phoneme recognition of throat microphone speech recordings using transfer learning

Throat microphones (TM) are a type of skin-attached non-acoustic sensors, which are robust to environmental noise but carry a lower signal bandwidth characterization than the traditional close-talk microphones (CM). Attaining high-performance phoneme recognition is a challenging task when the training data from a degrading channel, such as TM, is limited. In this paper, we address this challenge for the TM speech recordings using a transfer learning approach based on the stacked denoising auto-encoders (SDA). The proposed transfer learning approach defines an SDA-based domain adaptation framework to map the source domain CM representations and the target domain TM representations into a common latent space, where the mismatch across TM and CM is eliminated to better train an acoustic model and to improve the TM phoneme recognition. For the phoneme recognition task, we use the convolutional neural network (CNN) and the hidden Markov model (HMM) based CNN/HMM hybrid system, which delivers better acoustic modeling performance compared to the conventional Gaussian mixture model (GMM) based models. In the experimental evaluations, we observed more than 12% relative phoneme error rate (PER) improvement for the TM recordings with the proposed transfer learning approach compared to baseline performances.

A Spectra-Based Equalization-Generation Combined Framework for Throat Microphone Speech Enhancement

Non-acoustic sensors are widely used in speech signal processing tasks, and their immunity to the background acoustic noise shows great benefits to traditional speech enhancement. To avoid using acoustic speech disturbed by strong noise, spectra mapping from throat microphone (TM) speech to acoustic microphone (AM) speech has been studied. However, there is a distinguished difference between the spectra of the two kinds of speech, and the mapping relationship is different in the low-band and high-band spectra, which limits the performance of the traditional full-band spectra mapping model. In this paper, to improve the perceived quality and intelligibility of TM speech, we investigate the low-band and high-band spectral structure between TM and AM speech, respectively, and propose a spectra-based band-division mapping framework for TM speech enhancement based on the investigation. In the framework, the low-band target spectra of AM speech are mapped based on the equalization method, and the high-band spectra are mapped from the full-band TM speech spectra, which are lack of high-frequency components. The overall framework can be viewed as a combination of spectra equalization in the low band and spectra generation in the high band. Both the objective and subjective evaluation results show clear advantages over the existing TM speech enhancement method based on the full-band spectra mapping.

An analysis of the effect of combining standard and alternate sensor signals on recognition of syllabic units for multimodal speech recognition

This paper studies the effect of combining evidences from multiple modes of speech on the recognition of different categories of sounds. Multimodal speech recognition systems are built by combining the acoustic and visual cues from the (lip radiated) normal microphone speech, throat microphone speech and lip reading for the recognition of the highly confusable 145 consonant-vowel units of the Hindi language. The performance of the multimodal systems are compared with that of the unimodal systems for the recognition of sounds based on their place (POA) and manner of articulation (MOA) as well as their associated vowels. This comparison shows that though the multimodal ASR systems rely on the presence of complimentary speech-related acoustic and visual cues present in the different modes, not all evidences are complimentary. Bimodal systems that combines visual cues from lip reading are shown to improve the recognition of sounds based on POA and MOA, but decrease the recognition of vowels. This study shows that, compared to the standard Automatic Speech Recognition(ASR) system, the best multimodal system that combines the two acoustic cues as well as visual cue improves the recognition of POA category by 11%, MOA category by 3% and vowels by 2%. However, the study shows the need for exploring better fusion techniques to overcome absence of complementary evidences in certain categories of sounds, especially in bimodal systems.

Improving Throat Microphone Speech Recognition by Joint Analysis of Throat and Acoustic Microphone Recordings

We present a new framework for joint analysis of throat and acoustic microphone (TAM) recordings to improve throat microphone only speech recognition. The proposed analysis framework aims to learn joint sub-phone patterns of throat and acoustic microphone recordings through a parallel branch HMM structure. The joint sub-phone patterns define temporally correlated neighborhoods, in which a linear prediction filter estimates a spectrally rich acoustic feature vector from throat feature vectors. Multimodal speech recognition with throat and throat-driven acoustic features significantly improves throat-only speech recognition performance. Experimental evaluations on a parallel TAM database yield benchmark phoneme recognition rates for throat-only and multimodal TAM speech recognition systems as 46.81% and 60.69%, respectively. The proposed throat-driven multimodal speech recognition system improves phoneme recognition rate to 52.58%, a significant relative improvement with respect to the throat-only speech recognition benchmark system.

Mapping Speech Spectra from Throat Microphone to Close-Speaking Microphone: A Neural Network Approach

Speech recorded from a throat microphone is robust to the surrounding noise, but sounds unnatural unlike the speech recorded from a close-speaking microphone. This paper addresses the issue of improving the perceptual quality of the throat microphone speech by mapping the speech spectra from the throat microphone to the close-speaking microphone. A neural network model is used to capture the speaker-dependent functional relationship between the feature vectors (cepstral coefficients) of the two speech signals. A method is proposed to ensure the stability of the all-pole synthesis filter. Objective evaluations indicate the effectiveness of the proposed mapping scheme. The advantage of this method is that the model gives a smooth estimate of the spectra of the close-speaking microphone speech. No distortions are perceived in the reconstructed speech. This mapping technique is also used for bandwidth extension of telephone speech.