Identification of the Similarity of Bangla Words Using Different Word Embedding Techniques

Abstract

From the browser search bar to the voice command in a computer system, everywhere Natural Language Processing (NLP) is broadly used nowadays. NLP is mainly used for the purpose of converting user data into compatible search materials. Some specific applications of NLP are sentiment analysis, text mining, news classification and many more. Those concepts are also very important in case of Bangla language processing. A number of researches have been conducted to process Bangla language to produce potential output from the user data of Bangla-speaking people. Word embedding is a major part of NLP and bears a vast importance in processing languages. In this work the word embedding for Bangla language is mainly focused. In doing the embedding words in Bangla language we have used Word2Vec model and FastText model with Gensim library. The Word2Vec model produces vector of words and similarly does the FastText model while FastText breaks the words into small blocks to train into machine. There are a very countable number of researches regarding the word embedding in Bangla language. In the proposed word FastText produces promising result over the Word2Vec model although no numerical conclusion was possible to derive at this phase of proposed implementation.