Bootstrapping Weakly Supervised Segmentation-free Word Spotting through HMM-based Alignment

Abstract

Recent work in word spotting in handwritten documents has yielded impressive results, largely using supervised learning systems, which are dependent on manually annotated data, making deployment to new collections a significant effort. In this paper, we propose an approach that utilises transcripts without bounding box annotations to train segmentation-free query-by-string word spotting models, given a partially trained model. This is done through a training-free alignment procedure based on hidden Markov models. This procedure creates a tentative mapping between word region proposals and the transcriptions to automatically create additional weakly annotated training data, without choosing any single alignment possibility as the correct one. When only using between 1% and 10% of the fully annotated training sets for partial convergence, we automatically annotate the remaining training data and achieve successful training using it. In terms of mean average precision, our final trained model then comes within a few percent of the performance of a model trained with the full training set on all our datasets. We believe that this will be a significant advance towards a more general use of word spotting, since digital transcription data will already exist for parts of many collections of interest.