Automatic construction of classification dimensions by clustering texts based on common words

Abstract

A way to efficiently manage large sets of texts is to construct multiple dimensions on texts and manage the texts from the dimensions. The key problem is to use a suitable representation model of text to classify texts into class trees representing dimensions. Traditional vector-based distance definitions are unsuitable to restrict the semantic relations between texts and measure the similarity of a set of more than two texts. Frequently occurred common words within a set of texts represent the texts but traditional frequent-term-set based text clustering approaches and the topic model are unsuitable for generating class trees on texts. This paper proposes an approach to using common words to represent texts and measuring the similarity of a class of texts by calculating the sum of the weights of common words of the class that indicate the common semantics of these texts. The idea is in line with the characteristics of human classification of texts. A bottom-up text clustering approach is proposed to construct class trees of texts. The common words of each class on the class trees are used as the label of the class to indicate the common semantics of the class and manage the texts of the class. Therefore, the approach can be used to construct multiple classification trees on texts. The experiments for evaluating the classification accuracy and the structure of the constructed class trees show that our approach is better than other clustering algorithms. A document summarization approach based on our approach reaches a good performance.