A Pipeline Approach to Context-Aware Handwritten Text Recognition

Yee Fan Tan,Tee Connie,Michael Kah Ong Goh,Andrew Beng Jin Teoh

doi:10.3390/app12041870

Abstract

Despite concerted efforts towards handwritten text recognition, the automatic location and transcription of handwritten text remain a challenging task. Text detection and segmentation methods are often prone to errors, affecting the accuracy of the subsequent recognition procedure. In this paper, a pipeline that locates texts on a page and recognizes the text types, as well as the context of the texts within the detected region, is proposed. Clinical receipts are used as the subject of study. The proposed model is comprised of an object detection neural network that extracts text sequences present on the page regardless of size, orientation, and type (handwritten text, printed text, or non-text). After that, the text sequences are fed to a Residual Network with a Transformer (ResNet-101T) model to perform transcription. Next, the transcribed text sequences are analyzed using a Named Entity Recognition (NER) model to classify the text sequences into their corresponding contexts (e.g., name, address, prescription, and bill amount). In the proposed pipeline, all the processes are implicitly learned from data. Experiments performed on 500 self-collected clinical receipts containing 15,297 text segments reported a character error rate (CER) and word error rate (WER) of 7.77% and 10.77%, respectively.

Highlights

Handwritten text recognition (HTR) has gained enormous research interest due to the potential benefits that can be derived from accurate text transcription that eases attempts to digitize handwritten content [1,2]
A system that can recognize human handwritten text is significantly essential in automatic information storage and management
This paper presents a pipeline approach towards HTR

Summary

Introduction

Handwritten text recognition (HTR) has gained enormous research interest due to the potential benefits that can be derived from accurate text transcription that eases attempts to digitize handwritten content [1,2]. An HTR system is applicable to a myriad of scenarios, ranging from reading bank cheque amounts to transcribing medical records and notes [3]. Highly desirable in practical applications, HTR is faced with a number of challenges. The current HTR systems generally apply a Convolutional Neural Network (CNN). Long Short-Term Memory (LSTM) model for text transcription [3]. Using a single text recognition model is insufficient for the text transcription task. Ingle et al proposed a text style classification approach using an LSTM-based and fully feed-forward network model for line-level segmentation

Methods

Results

Discussion

Conclusion