Enhanced Word Classing for Recurrent Neural Network Language Model

Abstract

Recurrent Neural Network Language Model (RNNLM) has recently been shown to outperform conventional N-gram LM as well as many other competing advanced language model techniques. However, the computation complexity of RNNLM is much higher than the conventional N-gram LM. As a result, the Class-based RNNLM (CRNNLM) is usually employed to speed up both the training and testing phase of RNNLM. In previous work with RNNLM, a simply method based on word frequency has been used to derive word classes. In this paper, we take a closer look at the classing and explore to improve the RNNLM performance by enhancing word classing. More specially, we employed bi-gram mutual information clustering, a classical word clustering method which is more accurate, to obtain word classes. Finally, experiments on the standard test set Penn Tree Bank showed that 5%∼7% relative reduction in perplexity (PPL) could be obtained by bigram mutual information clustering method compared to the frequency based word clustering method.