Personalized Hand Writing Recognition Using Continued LSTM Training

Abstract

On-line handwriting recognition has seen major improvements in the last few years with the successful application of deep learning techniques. While the training data is massive and training times are large, the current accuracies are very good even with reasonable small model sizes (when compared to models in image recognition tasks). However, the important task of personalizing such deep models (after a new user has downloaded the model onto his mobile device) to the writing style of the new user, resulting in an improved model with significantly higher accuracy, has not been investigated so far. It is clearly not feasible to download the massive training data for each new user, and re-train the model on the device itself (after mixing original data with the limited quantity of training samples a new user can supply). To our knowledge, it is not known so far with what adaptation technique, with what old training data (in addition to the user's new samples), how flexibly the retraining can be done on the device, how much accuracy gain can be expected, and whether the model can be simplified as a result of the retraining. In this paper we address all these important questions regarding deep network models on device: we demonstrate simple but robust continued-training techniques for adapting a pre-trained model to a specific user's writing style. The pre-training is done using Long Short Term Memory (LSTM) network and a Connectionist Temporal Classification (CTC) loss function. It is seen that only a small set of the old training data can be used along with the user's samples to successfully retrain the model. The retrained (adapted) model achieves around 2.5% more accuracy than the roughly 92% accuracy of the non-personalized model for on-line handwritten Hindi text recognition, also surpassing all existing techniques. We also derive some additional insights into the questions posed above.