Comparing Neural Lexical Models of a Classic National Corpus and a Web Corpus: The Case for Russian

Abstract

AbstractIn this paper we compare the Russian National Corpus to a larger Russian web corpus composed in 2014; the assumption behind our work is that the National corpus, being limited by the texts it contains and their proportions, presents lexical contexts (and thus meanings) which are different from those found ‘in the wild’ or in a language in use.To do such a comparison, we used both corpora as training sets to learn vector word representations and found the nearest neighbors or associates for all top-frequency nominal lexical units. Then the difference between these two neighbor sets for each word was calculated using the Jaccard similarity coefficient. The resulting value is the measure of how much the meaning of a given word is different in the language of web pages from the Russian language in the National corpus. About 15% of words were found to acquire completely new neighbors in the web corpus.In this paper, the methodology of research is described and implications for Russian National Corpus are proposed. All experimental data are available online.Keywordscorpora comparisondeep learningsemantic similarityvector representations of lexical unitslexical co-occurrence networksRussian National CorpusWeb as corpusword2vec