Leveraging various Transformers Based Architectures for Truecasing

Abstract

Truecasing is the process of finding the proper capitalization of words in a text where the correct case information of the word is not given. The case of letters is often lost while performing language translation from non-capitalized languages (eg: Japanese, Arabic, Hindi) to capitalized languages (eg: Greek, Latin, English). In Optical Character Recognition (OCR) where the different cases for letters like ‘p’ and ‘P’ are similar, the case as well as the context and proper meaning of the text is lost. A similar situation is observed for text generated from Automatic Speech Recognition. This paper proposes to solve the problem of truecasing while focusing on three different architectures, all based on transformers. Transformers have generally proved to outperform recurrent and convolutional models. A proposal for a novel light-weight transformer architecture has been made which performs competitively with existing models that are computationally expensive. To provide a basis for comparison, two existing transformer architectures were fine-tuned for truecasing - BERT and GPT-2. These architectures are analyzed and compared on various metrics. The proposed architecture shows an accuracy of 98.25%, which is also an improvement of 0.41 against the current state-of-art.