BoydCut: Bidirectional LSTM-CNN Model for Thai Sentence Segmenter

Abstract

Sentence is imperative in order to build the top level of NLP (Natural Language Processing) applications such as Information retrieval, News Summarization, Knowledge graph. In Thai language, neither each word token is not separated by using just only space like English language, nor the sentence is verified its boundary by using full stop symbols. This paper proposes BoydCut, an NLP framework for identifying sentence boundaries based on Bidirectional LSTM-CNN Model. We develop this framework by utilizing the combination of character, word, and part of speech features. With Bidirectional LSTM, it can learn sequent of word-level in sentences and learn extracted features from character level. With the benefit of the combination Bidirectional LSTM-CNN Model, we do not need feature engineering for feature extraction that can be saved a lot of cost and time in order to build a sentence segmentation model. We also simply design the experiments in a different bucket of features extracted from a deep sequential model. The result empirically shows well perform in internal and external data, as well as help a lot in order to build several useful on the top level of NLP applications.