Constructing Sentiment Lexicons in Norwegian from a Large Text Corpus

Abstract

Sentiment lexicons are useful to automatically extract sentiment from text. In this paper, we generate several Norwegian sentiment lexicons by extracting sentiment information from two different types of Norwegian text corpus, namely, news corpus and discussions forums. The methodology is based on the Point wise Mutual Information (PMI). We introduce a modification of the PMI that considers small blocks of the text instead of the text as a whole. The rational of this modification is to counter the detrimental effect of the length of the text on the PMI and on the quality of the lexicon. The high computational cost due to the huge amount of textual information to be processed in addition to our modified PMI formula is tackled efficiently by relying heavily on parallelization using Map-Reduce and Mongo DB shards. Movie and product reviews are used to evaluate the generated sentiment lexicons to correctly classify review ratings. The lexicon exhibits a satisfactory performance when evaluated, in particular when considering the context change in the corpuses. In fact, surprisingly enough, sentiment lexicon generated from large News corpus exhibits a satisfactory performance when tested on annotated product and movie reviews despite that the later two have different contexts, bearing similarity to the notion of Transfer Learning [1] reported in the literature. Some suggestions on how to increase the performance are proposed. All the sentiment lexicons are publicly available for those that are interested.