Reading in the mist: high-quality optical character recognition based on freely available early modern digitized books

Abstract

Abstract In this paper, we present a workflow for reworking digitized versions of early modern books, freely available in the public domain, in such a way that they will be capable of yielding high-quality optical character recognition (OCR) results suitable for computational text mining. Testing our method, we observed that anything above 90% OCR accuracy is sufficient for semantic analysis. In addition,the overall homogeneity in the OCR accuracy across the corpus proved to be more important than having perhaps only a few works with higher accuracy and the rest available in a lower quality. In terms of the OCR process, this paper illustrates how it was possible to reduce the processing time at maximum quality of a single book of average length (ca. 500 pages) from a minimum of 20 hrs to an average of about 3 hrs (though theoretically nearly infinitely reducible). This was achieved by replacing a step-by-step OCR process with a fully automated pipeline system run on an arbitrary number of servers, breaking up the full process of OCRing one book into minimal tasks that can be handled simultaneously by multiple servers.