Sparse Data and Smoothing in Statistical Part-of-Speech Tagging*

Abstract

This article reports on a series of experiments in statistical part-of-speech tagging of Swedish texts, with different probabilistic models and different smoothing schemes for both lexical and contextual probabilities. The most important conclusions are that lexical and contextual probabilities require different smoothing methods and that smoothing is only crucial for lexical probabilities. Of the particular smoothing methods tested in the experiments, Good-Turing estimation achieves the best results for the lexical model, while a simple additive smoothing scheme gives the best performance for the contextual model.