Convolutional recurrent neural networks with hidden Markov model bootstrap for scene text recognition

Abstract

Text recognition in natural scene remains a challenging problem due to the highly variable appearance in unconstrained condition. The authors develop a system that directly transcribes scene text images to text without character segmentation. They formulate the problem as sequence labelling. They build a convolutional recurrent neural network (RNN) by using deep convolutional neural networks (CNN) for modelling text appearance and RNNs for sequence dynamics. The two models are complementary in modelling capabilities and so integrated together to form the segmentation free system. They train a Gaussian mixture model–hidden Markov model to supervise the training of the CNN model. The system is data driven and needs no hand labelled training data. Their method has several appealing properties: (i) It can recognise arbitrary length text images. (ii) The recognition process does not involve sophisticated character segmentation. (iii) It is trained on scene text images with only word‐level transcriptions. (iv) It can recognise both the lexicon‐based or lexicon‐free text. The proposed system achieves competitive performance comparison with the state of the art on several public scene text datasets, including both lexicon‐based and non‐lexicon ones.