Improving paragraph segmentation using BERT with additional information from probability density function modeling of segmentation distances

Abstract

Paragraphs play a key role in writing and reading texts. Therefore, studies about dividing texts into appropriate paragraphs, or paragraph segmentation have gathered academic attention for a long time. Recent advancements in pre-trained language models have achieved state-of-the-art performances in various natural language processing fields, including paragraph segmentation. However, pre-trained language model based paragraph segmentation methods had a problem in that they could not consider statistical metadata such as how far each paragraph segmentation point should be apart from each other. Therefore we focused on combining paragraph segmentation distance and pre-trained language models so that both statistical metadata and state-of-the-art representation ability could be considered at the same time. We propose a novel model by modifying BERT, a state-of-the-art pre-trained language model, by adding segmentation distance information via probability density function modeling. Our model was trained and tested on the domain of the novel, and showed improved performance compared to baseline BERT and previous study, acquiring a mean of 0.8877 F1-score and 0.8708 MCC. Furthermore, our model showed robust performance regardless of the authors of the novels.