Change point detection in text data

Abstract

The analysis of text data using artificial intelligence and statistical methods has become increasingly important in recent years. One application is the automatic assignment of documents. For this purpose, a classification model is trained on the basis of historical data. If the structure of the texts to be classified changes over time, the quality of the classification will decrease. Change point detection algorithms can counteract this. Such algorithms automatically detect changes in the structure of the texts and indicate that the trained classification model has to be adapted. However, the undesired influence of the length of the document needs to be handled when modeling the text data. We present a multinomial change-point model detecting changes in text structures. The results are supported by simulation studies.