A Comparison of Thai Sentence Boundary Detection Approaches Using Online Product Review Data

Abstract

In Natural Language Processing (NLP), the goal of sentence boundary detection (SBD) is to identify sentence boundaries in a phrase, paragraph, or document, which can be used in current NLP applications, including sentimental analysis, contextual chatbot, and machine translation, etc. Previous studies and existing NLP libraries often provide a straightforward approach to the task; for instance, they assume that a sentence always ends with certain punctuation symbols such as a period, a semicolon, a exclamation mark, or a question mark. The mentioned approach is impractical for other languages, such as Thai, where there is no symbol to designate where a sentence ends. With regard to developing an effective sentimental analysis or machine translation for the Thai language, a solid effort in detecting sentence boundary is needed. There is also as a need validating the SBD model against a real-world dataset, by involving the use of an online textual corpus. This paper attempts to compare Condition Random Fields (CRF) and Bidirectional Long-Short Term Memory with CRF layer (BiLSTM-CRF) on the online textual dataset. We scraped our own corpus from the top Thai web forums through the use of a Scrapy web-crawling framework. In the paper, 2,496 comments related to beauty product reviews were manually segmented by a Thai linguistic expert. Our experimental results revealed that the CRF based on the word-based labelling approach with widow size outperformed the BiLSTM-CRF.