Improvement of Text Segmentation TextTiling Algorithm

Abstract

Text segmentation has a wide range of applications in the fields of information extraction and abstract generation. The method uses local information to determine a specific segmentation, and cannot fully consider the information of the remaining paragraphs. In response to this problem, this paper improves the segmentation point selection of the algorithm. After the similarity values of the respective interval points are obtained by the similarity calculation, the sliding window is used to slide on the interval points, and the similarity value of the interval points at the center of the window is replaced with the average of all the similarity values in the window. The experimental results show that the text segmentation based on the improved method can suppress small local changes on the curve, thus highlighting the large similarity value changes, and effectively improving the performance of text segmentation.