HieNN-DWE: A hierarchical neural network with dynamic word embeddings for document level sentiment classification

Abstract

The recent rapid developments of neural networks have stimulated significant performance progress on document level sentiment classification. The existing researches focus on extracting long-distance features from documents, which fail to exploit the deep semantic information of documents. Besides, few of them have taken the representations of polysemous words into consideration, such that the semantic information of documents are not precisely exploited. To tackle these issues, we propose a novel hierarchical neural network model based on dynamic word embeddings (HieNN-DWE) for document level sentiment classification. Specifically, the dynamic word embeddings are produced by ELMo (Embedding from Language Models), which can deeply exploit the semantic information for polysemous words. The embeddings are then fed into HieNN-DWE which is a two-layer hierarchical neural network model. Note that the first layer utilizes bidirectional gated recurrent unit (BiGRU) as well as attention mechanism to encode sentences. The second layer uses both BiGRU and convolutional neural network (CNN) to capture global and local features from sentence representations. Thus, the final document representations are constructed by concatenating the outputs from BiGRU and CNN. Moreover, experimental results have shown that HieNN-DWE outperforms existing competitive methods on four public datasets for document level sentiment classification.