Improving text classification with word embedding

Abstract

One challenge in text classification is that it is difficult to make feature reductions based on the definition of the features. An ineffective feature reduction may even worsen the classification accuracy. Word2Vec, a word embedding method, has recently been gaining popularity due to its high precision rate of analyzing the semantic similarity between words at relatively low computational cost. However, there is limited research about feature reduction using Word2Vec. In this project, we developed a method using Word2Vec to reduce the feature size while increasing the classification accuracy. We achieved feature reduction by loosely clustering similar features using graph search techniques. Similarity thresholds above 0.5 were used in our method to pair and cluster the features. Finally, we utilized Multinomial Naïve Bayes classifier, Support Vector Machine, K Nearest Neighbor and Random Forest classifier to evaluate the effect of our method. Four datasets with dimensions up to 100,000 feature size and 400,000 document size were used to evaluate the result of our method. The result showed that around 4-10% feature reduction was achieved with up to 1-4% improvement of classification accuracy in terms of different datasets and classifiers. Meanwhile, we also succeeded in improving feature reduction and classification accuracy by combining our method with other classic feature reduction techniques such as chi-square and mutual information.