Sentence boundary detection in chinese broadcast news using conditional random fields and prosodic features

Abstract

This paper studies the use of condition random fields (CRF) and prosodic features for sentence boundary detection in Chinese broadcast news. Previous approaches mostly use first-order CRF and ignore the important context and sequential information. In this paper, we explore high-order CRF models to fully make use of the contextual and sequential information. Moreover, we show the effectiveness of CRF in sentence boundary detection by comparing it with various competitive models. The prosodic feature set is usually designed to be as exhaustive as possible in previous approaches. As a result, features may be highly correlated and some of them may be not effective. In this paper, we use a correlation-based feature selection method to select a subset with the most useful features. Finally, the use of the prosodic features, e.g., pitch, in Chinese sentence segmentation deserves further investigation because the tonal aspect of Chinese may complicate the expressions of pitch features. In this paper, we study the effectiveness of the prosodic features and rank their importance by an analysis of feature usage.