Statistical learning and analyses of Chinese ancient books for information retrieval

Abstract

The technique of full text retrieval for modern Chinese has been studied for a long time, but the same cannot be said for ancient Chinese books, especially in China. This paper tries to find the characteristics of Chinese ancient books which can be used for information retrieval. Statistical analysis was carried out on ancient Chinese books of over 35,000,000 words, including most of the works in common use. Based on these experiments some characteristics of ancient Chinese works are analyzed and compared with modern Chinese, including the basic unit of ancient works, the proportion of double character words, sentence length, and the field dependency of ancient Chinese works. We then give conclusions on ancient Chinese which is useful for information retrieval, especially when building inverted indexes and selecting the index unit. Depending on the conclusion, a full-text retrieval system for ancient Chinese books has been designed and realized. It shows that statistical learning and analyses are a great help in ancient Chinese information retrieval.