A TfidfVectorizer and SVM based sentiment analysis framework for text data corpus

Abstract

E-commerce and social networking sites are very much dependent on the available data which can be analyzed in real time to predict their future business strategies. However, analyzing huge amount of data manually is not possible in time context of business. Therefore, automated sentimental analysis, which can automatically determine the sentiments from the text data corpus plays an important role in today's world. Many sentimental analysis frameworks with state of the art results have been proposed in the literature. However, many of these frameworks have low accuracy on the textual data corpus contains emoticons and special texts. In addition, many of these frameworks are also energy and computation intensive with which puts limitation in their real time deployment. In this paper, we aim to address these issues by proposing a novel sentimental analysis framework based on Support Vector Machine (SVM). The proposed framework uses a novel technique to tokenize the text documents, wherein stop words, special characters, emoticons present in the text documents are eliminated. In addition, words with similar meanings and annotations are clubbed together into one type, using the concept of stemming. The pre-processed tokenized documents are then vectorized into n-gram integers vectors using the ‘TfidfVectorizer’ for use as input to the SVM based machine learning classifier model. Experimental results on the Amazon's electronics item review dataset and IMDB's movie review data corpus show that the proposed sentimental analysis framework achieves high performance compared to other similar frameworks proposed in the literature.