Optimization of topic estimation for the domain adapted neural network language model

Abstract

We present a neural network language model adapted for topics fluctuating in broadcast programs. Topic adapted n-gram language models constructed by using latent Dirichlet allocation for topic estimation are widely used. The conventional method estimates topics by separating the corpora into chunks that have few sentences. While the n-gram model uses several preceding words, the recurrent neural network and long short-term memory can learn to store huge amounts of past information in the hidden layers. Consequently, chunks for language models trained by using neural networks may have a longer optimal length than the chunks for language models trained by using the conventional methods. In this paper, the length of chunks and topic estimation process are optimized for the neural network language models. For the topic estimation, k-mean clustering, latent Dirichlet allocation, and word2vec were compared. On the basis of the results of comparison, we designed a neural network language model.