A HYBRID CONTINUOUS SPEECH RECOGNITION SYSTEM USING SEGMENTAL NEURAL NETS WITH HIDDEN MARKOV MODELS

Abstract

Up till recently, state-of-the-art, large vocabulary, continuous speech recognition (CSR) had employed hidden Markov modeling (HMM) to model speech sounds. In an attempt to improve over HMM we developed a hybrid system that integrates HMM technology with neural networks. We present the concept of a Segmental Neural Net (SNN) for phonetic modeling in CSR. By taking into account all the frames of a phonetic segment simultaneously, the SNN overcomes the well-known conditional-independence limitation of HMMs. We have developed a novel hybrid SNN/HMM system that combines the advantages of SNNs and HMMs using a multiple hypothesis (or N-best) paradigm. In this system, we generate likely phonetic segmentations from the HMM N-best list of word sequences, which are scored by the SNN. The HMM and SNN scores are then combined to optimize performance. In several speaker-independent, 1000-word CSR tests, the error rate for the hybrid system dropped 20% from that of a state-of-the-art HMM system alone.