Groundhog Day is Not a Good Model for Corpus Dispersion

Abstract

ABSTRACT This study argues that corpus analysis has yet to incorporate probabilistic reasoning into the measurement of dispersion. This proposal continues a recent observation that some commonly used dispersion measures were impaired by an extreme sensitivity to the random noise characterizing any real-world data set. This article will show that this could have been avoided if (a) the underlying metaphor for ‘corpus dispersion’ had been more carefully considered and (b) the measures had been derived from an appropriate probability mass distribution. The text then proposes a computationally efficient dispersion measure developed from a Poisson model of an unbiased word dispersion. Unlike several current measures, this measure is shown to give a score that is meaningful at all word frequencies while being more sensitive to real dispersional bias and less sensitive to noise.