Language Identifier

Abstract

This project presents a language identifier system focused on achieving high accuracy in identifying a specific set of 22 languages: Estonian, Swedish, English, Russian, Romanian, Persian, Pashto, Spanish, Hindi, Korean, Chinese, French, Portuguese, Indonesian, Urdu, Latin, Turkish, Japanese, Dutch, Tamil, Thai, and Arabic. Existing LI systems might struggle with the nuances of these languages, often prioritizing identification of more common languages. Our targeted approach allows for tailored optimization to achieve superior accuracy. We employ the Multinomial Naive Bayes (MNB) algorithm due to its effectiveness in text classification tasks and its ability to handle the high-dimensional, sparse features characteristic of LI based on character and word frequencies. The system achieves a promising accuracy of 95% using an 80/20 split for training and testing data. Keywords: Natural Language Processing (NLP), Language Identification (LI), Multinomial Naive Bayes (MNB), Text Classification, Language Barriers, Accuracy, Text Analysis.