Calibrating GloVe model on the principle of Zipf’s law

Abstract

GloVe is a popular method for learning word representation by fitting log-bilinear model. However, many researchers have observed that the number of co-occurrences of word pairs in Glove generally follow power law distribution with the value of power law exponent larger than 1; and the value becomes increasingly larger with the increase of corpus size, which is inconsistent with the principle of Zipf’s law. Zipf’s law is a power law distribution with the value of power law exponent equal to 1 and is a basic law of corpus linguistics. On the basis of this principle, this study aims to verify whether the performance of GloVe can be improved after processing certain calibrations on the GloVe model. The estimated value of power law exponent is generally obtained via least square estimation in the linear regression for the logarithm of the number of co-occurrences with the logarithm of their ranks. Inspired by processing a suitable power transformation of response variable in linear regression model to improve the model fitting, we implement power transformation to the logarithm of the number of co-occurrences of word pairs [i.e. (logXij)β] and appropriately select β, such that the estimated value of power law exponent approximates to 1. An estimation method of the hyperparameter β is presented for the different sizes of corpus. Then, we use (logXij)β instead of logXij in GloVe to learn the distributed representations of words. We evaluate this new approach on word analogy and word similarity tasks. Experimental results show that our method outperforms GloVe on the word analogy task, and the improvement of performance is significant under McNemar’s test. For word similarity task, the proposed method also outperforms GloVe on five benchmark datasets. Furthermore, our method has a higher convergence speed than GloVe.