Maximum entropy modeling for speech recognition

Abstract

Summary form only given. Maximum entropy (maxent) models have become very popular in natural language processing. We begin with a basic introduction of the maximum entropy principle, cover the popular algorithms for training maxent models, and describe how maxent models have been used in language modeling and (more recently) acoustic modeling for speech recognition. Some comparisons with other discriminative modeling methods is made. A substantial amount of time is devoted to the details of a new framework for acoustic modeling using maximum entropy direct models, including practical issues of implementation and usage. Traditional statistical models for speech recognition have all been based on a Bayesian framework using generative models such as hidden Markov models (HMM). The new framework is based on maximum entropy direct modeling, where the probability of a state or word sequence given an observation sequence is computed directly from the model. In contrast to HMM, features can be asynchronous and overlapping, and need not be statistically independent. This model therefore allows for the potential combination of many different types of features. Results from a specific kind of direct model, the maximum entropy Markov model (MEMM) are presented. Even with conventional acoustic features, the approach already shows promising results for phone level decoding. The MEMM significantly outperforms traditional HMM in word error rate when used as stand-alone acoustic models. Combining the MEMM scores with HMM and language model scores shows modest improvements over the best HMM speech recognizer. We give a sense of some exciting possibilities for future research in using maximum entropy models for acoustic modeling.