Language models and classification analysis for dream of the red chamber

Abstract

Dream of the Red Chamber is regarded as one of the four great classical novels in Chinese literature and the dispute over the authorship of the last 40 chapters has existed for years. In this paper the novel is divided into three parts: the first 40 chapters the middle 40 chapters and the last 40 chapters. They are analyzed separately based on language models and text classification. To be specific in this research on the one hand from the aspect of linguistic features we analyze N-gram models calculate Jaccard Index acquire the differences of words and describe the collocations with the grammatical information. On the other hand from the aspect of machine learning the method of Random Forest is used to extract characteristics in experiments for automatic classification and the result shows that most of the first 40 chapters and the middle 40 chapters are not identified as the group of the last 40 chapters. With all these experiments above finally we are able to reach the conclusion that differences between the last 40 chapters and the first 80 chapters do exist.