Feature Extension for Chinese Short Text Based on Tongyici Cilin

Abstract

Since the short text has characteristics such as sparse features, calculating its similarity is a considerable challenge. However, there is less research on the method of Chinese short text feature extension in short text similarity calculation. Therefore, to have a deeper understanding of the method on using feature extension in the similarity of Chinese short texts, this paper adopts a feature extension algorithm based on an external thesaurus Tongyici Cilin (extended) for short texts. The purpose is to solve the feature sparseness problem of Chinese short text feature vectors. Firstly, segment words in the short text according to certain rules with high surface similarity and extract the main difference components in the text. Then, calculate the similarity of the major difference components between the two short texts based on Cilin. Finally, perform feature extension according to the similar results in the corresponding short text. In the large-scale Chinese Question Matching Corpus LCQMC, a variety of unsupervised models are used for testing. The experimental results show that the method in this paper has a certain improvement effect on various spatial vector similarity algorithms. It can achieve accuracy rates and F1-score of about 3% improvements.