Deep Neural Network Based Sentence Boundary Detection and End Marker Suggestion for Social Media Text

Abstract

For processing any natural language processing application, the knowledge of structure of sentence including its boundaries plays a vital role. Incorrect sentence boundary may lead to wrong outputs and hence decreasing the performance of NLP systems. Detecting sentence boundaries in code mixed social media text is not an easy task. People generally omits the boundary markers and use punctuation for other stylistic tasks. We propose a deep neural network approach for sentence boundary marking as well as suggesting appropriate punctuation mark in code mixed social media text. We experimented with single layer bidirectional and two layer bidirectional models. Both word sequence and character sequence are experimented. Bidirectional model using character sequence out performs all other models for sentence boundary detection as well as end marker suggestion.