Study of Optimal Text Size Phenomenon in Zipf–Mandelbrot’s Distribution on the Bases of Full and Distorted Texts. Author’s Frequency Characteristics and derivation of Hapax Legomena

Abstract

ABSTRACTThis paper explores word-frequency patterns when considering text length, authorship, and random distortion of texts. Through a series of experiments, we determined an optimal text size, a phenomenon that was predicted by George Zipf, which sees a minimal discrepancy between calculated and observed frequencies. A graphic representation allowed a plausible explanation behind the existence of this phenomenon. Working on the assumption that distorted texts might disobey Zipf’s Law, we explored correlations among frequencies and text entirety compared with text distortions. Results reveal the crucial role of text length for maintaining Zipfian distribution: randomly chosen sets of words and fragmentary texts of optimal size still obey Zipf’s Law. Findings show that authorship manifests itself through the author constant, defined as the relative frequency of the most frequent words, which remains constant throughout the works of any given author, including randomly chosen text chunks and fragments of sentences of various sizes.