Modeling speech production and perception mechanisms and their applications to synthesis, recognition, and coding

Abstract

Summary form only given, as follows. Quantitative models of human speech production and perception mechanisms provide important insights into our cognitive abilities and can lead to high-quality speech synthesis, robust automatic speech recognition and coding schemes, and better speech and hearing prostheses. Some of our research activities in these two areas are described. Our speech production work involved collecting, and analyzing magnetic resonance images (MRI), acoustic recordings, and electropalatography (EPG) data from talkers of American English during speech production. The articulatory database is the largest of its kind in the world and contains the first images of liquids (such as /I/ and /r/) and fricatives (such as /s/ and /sh) for both male and female talkers. MR images are useful for characterizing the 3D geometry of the vocal tract (VT) and for measuring lengths, area functions, and volumes. EPG is used to study inter- and intra-speaker variabilities in the articulatory dynamics, while acoustic recordings are necessary for modeling. Inter- and intra-speaker characteristics of the VT and tongue shapes will be illustrated for various speech sounds, as well as results of acoustic modeling based on the MRI and acoustic data. The implications of our findings on vocal-tract normalization schemes and speech synthesis are also discussed. In the speech perception area, aspects of auditory signal processing and speech perception are parameterized and implemented in a speech recognition system. Our models parameterize the sensitivity to spectral dynamics and local peak frequency positions in the speech signal. These cues remain robust when listening to speech in noise. Recognition evaluations using the dynamic model with a stochastic hidden Markov model (HMM) recognition system showed increased robustness to noise over other state-of-the-art representations. The applications of auditory modeling to speech coding are discussed. We developed an embedded and perceptually-based speech and audio coder. Perceptual metrics are used to ensure that encoding is optimized to the human listener and is based on calculating the signal-to-mask ratio in short-time frames of the input signal. An adaptive bit allocation scheme is employed and the subband energies are then quantized. The coder is variable-rate, noise-robust and suitable for wireless communications.