Word frequency and text type: Some observations based on the LOB corpus of British English texts

Abstract

The study of word frequencies and the printing of word frequency lists started before the advent of the computer (e.g., Thorndike and Lorge 1944), but what was once a long and laborious job is now a routine affair, given the availability of the computer and corpora of machine-readable texts. Some major computer-based English word frequency lists are those published by Kuiera and Francis (1967), Carroll et al. (1971), Engels et al. (1981), and Hofland and Johansson (1982). In several respects Thorndike and Lorge's work is more sophisticated than the more recent publications. It is based on a larger material, it gives the frequencies of lexemes rather than of word forms, and it reports the results of large-scale counts based on different types of material. The aim of the book is to provide information on how common a word is "in standard English reading matter" (p.x), primarily for the use of language teachers. The four separate counts reported were based on (1) texts used at school ("readers, textbooks, the Bible and the English classics," p.x), (2) popular magazines, (3) juvenile books ("books recommended for boys and girls in grades 3 to 8," p.x), and (4) "a miscellany of juvenile and adult readings-of old and recent, and of matter-of-fact and imaginative-but ... [omitting] school readers and textbooks" (p.x). As a guide to word frequency and text type in present-day English, the work of Thorndike and Lorge is, however, of limited value. For one thing, the lists are based on texts which are at least forty years old; in addition, three of the counts used a mixture of old and contemporary texts. Two of the counts only are based on well-defined categories of