STOPWORDS REMOVAL AND ITS ALGORITHMS BASED ON DIFFERENT METHODS

Abstract

Abstract : The research performed in this thesis is focused on the analysis of different methods to remove stopwords in Punjabi language. For organizing unstructured text in order to implement stopwords removal techniques, text preprocessing has to be applied. Text processing describes a variety of processing that is performed on raw data to prepare it for one more processing procedure which will be more helpful for performing some further, more purposeful analytic tasks. The words are called stopwords that occur most frequently in a document and contain very little information which is not essential in a document such as ਦ , ਹ , ਦੀ , ਤ , ਦਾ , ਨ , ਅਤ , ਤ etc. A list of such words is known as ‘stopwords list’ or ‘stopwords corpus’. These words are removed in the preprocessing phase of the text classification process. The process of removing stopwords help to save time and reduces the size of those document. It also helps to increase the accuracy as well as performance of IR tasks. Most of the researchers worked on languages such as English, Arabic, Sanskrit etc. in the informational retrieval (IR) field. Therefore a lot of work and efforts need to be done in languages other than the languages in which the research has been already done to a great extent. The main goal of this thesis is to remove the stopwords in Punjabi language by using different techniques. Punjabi language is 11 th most-spoken language of India which is written in Gurmukhi script. Punjabi language is also used in mass media such as news, advertisements, movies, music etc. There is no standard stopword list created for Punjabi language as most of the stopwords lists are created for English and other languages. In this work, four different algorithms viz classical method using a pre-compiled stoplist, method based on frequency, method based on removing singletons and method based on Punjabi word corpus to remove Punjabi stopwords have been proposed, implemented and analyzed. Thus the size of the document is reduced by 30-35% by eliminating the set of such stopwords.