Correlation analysis of short text based on network model

Abstract

Correlation of words in the text is of great importance in text analysis like text retrieval, keywords extraction, and text clustering. For short text, because of the limited information of text content, it is difficult to catch the correlation well among words. In this paper, we propose an algorithm based on the complex network to calculate the correlation of words in short texts. A new variable Edge-degree is proposed and used in studying the network model of texts. By using fluctuation analysis, we give the condition that Edge-degree correlation between words exists beyond nearest neighbors. Further analysis shows that numerical results of the fluctuation function of Edge-degree act a power law distribution and that the scaling exponent diverges at a long distance under the finite size effect and varies in different texts. The fluctuation function separates the words in a text into different clusters, and this property is used to measure inner-correlation of different words. Hub nodes act a significant influence on the long-range Edge-degree correlation through changing the linear trend of the fluctuation function in a log–log plot.