A novel negative sampling based on TFIDF for learning word representation

Abstract

In recent years, continuous-valued word embedding learned by neural network attaches extensive attentions. Especially, Mikolov׳s Continuous Bag-of-Words (CBOW) model and Skip-gram model optimized by Negative Sampling (NEG) achieves impressive result in capturing word semantic and syntactic information. However, the property of NEG that samples over word frequency has the drawbacks of unbalanced training and tends to decrease the training times of medium-frequency words which have greater amount of information; furthermore, words with higher frequency are less probability of being the negative samples. Focus on these issues, this paper proposed a novel NEG strategy that samples negatives based on the notion of Term Frequency-Inverse Document Frequency (NEG-TFIDF). Two ideas are included in this method. First, Negative Sampling is considers as the more effective optimization approach for learning word representation; second, the concept of TFIDF is combined to optimize Negative Sampling, which is a classic and popular method to assess the discrimination of one word for a corpus. Statistical results prove that NEG-TFIDF feasibly reduces the interference of high-frequency words that have very less information and increases the training times of medium-frequency words keeping the total training times being constant and without more calculations. The experiment results show that NEG-TFIDF outperforms Mikolov׳s NEG on both word analogy and word similarity test tasks, particularly in terms of the performance of medium-frequency words. Meanwhile, it also achieves better effects on NLP tasks. Moreover, NEG-TFIDF is a reasonable solution to the problem that NEG׳s performance decreases when the number of negative samples increases beyond a boundary value.