N-Gram based language processing using Twitter dataset to identify COVID-19 patients

Abstract

Due to the rapid growth of electronic documents, e.g., tweets, blogs, Facebook posts, snaps in different languages that use the same writing script, language categorization, and processing have great importance. For instance, to identify COVID-19 positive patients or people’s emotions on COVID-19 pandemic from tweets written in 35 different languages faster and accurate, language categorization and processing of tweets is significantly essential. Among many language categorization and processing techniques, character and word n-gram based techniques are very popular and simple but very efficient for categorizing and processing both short and large documents. One of the fundamental problems of language processing is the efficient use of memory space in implementing a technique so that a vast collection of documents can be easily categorized and processed. In this paper, we introduce a framework that categorizes the language of tweets using n-gram based language categorization technique and further processes the tweets using the machine-learning approach, Linear Support Vector Machine (LSVM), that may be able to identify COVID-19 positive patients. We evaluate and compare the performance of the proposed framework in terms of language categorization accuracy, precession, recall, and F-measure over n-gram length. The proposed framework is scalable as many other applications that involve extracting features and classifying languages collected from social media, and different types of networks may use this framework. This proposed framework, also being a part of health monitoring and improvement, tends to achieve the goal of having a sustainable society.