UBERT22: Unsupervised Pre-training of BERT for Low Resource Urdu Language

Abstract

Natural Language Understanding (NLU) tools have enabled the development of sophisticated and powerful Natural Language Processing (NLP) models. However, this progress is limited to English and European languages and low resource languages lack such tools due paucity of resources. In this paper, we develop UBERT22- an unsupervised pre-trained BERT model for Urdu language. For this purpose, first, we develop a dataset of ‘Zakheera’ containing high-quality content of 1.16 million Urdu language news and blog articles posted on top 37 Urdu websites. Next, we pre-process the text and tokenize content in 21.8 million Urdu sentences. Finally, we extract the word-piece vocabulary of 30,000 tokens and pre-train the BERT model for Masked Language Modelling (MLM) and Next Sentence Prediction (NSP) tasks. We compare the performance of UBERT22 with existing multilingual and small-Urdu BERT for various downstream tasks. We notice that UBERT22 outperforms multilingual and small-Urdu BERT for fake news identification, propaganda classification, topic categorization, and sentiment analysis tasks. Overall, UBERT22 achieves 2-19% higher accu-racy compared to baseline results and competitive BERT models. We believe that the public availability of our pre-trained model and upstream dataset will enable the development of state-of-the-art NLP models of Urdu language such as chatbots, question answering systems, sentiment analyzers, virtual assistants, speech recognizers, and machine translators.