Learning Local and Global Multi-context Representations for Document Classification

Abstract

Context-based attention models achieve state-of-the-art results on constructing document representations. Current methods build document representations based on a trainable global context vector for all documents, while local contexts capturing document-specific information are ignored. Representing document context as a vector also restricts the model's capacity to capture complete semantic components of documents. In this work, we address these limitations by proposing novel attention-based approaches. We first propose a local and global context attention (LGCA) model, which constructs the context representations of a document as a combination of global and local contexts. The local context is used to capture local document-specific information. Furthermore, a multi-context attention (MCA) model is proposed to capture multiple local contexts within a document, thereby fusing the overall semantic information of documents. We conduct experiments on document-level sentiment classification tasks where extra user and product information is shown to be important. Experimental results indicate that our methods significantly outperform previous models with and without user and product information. It is also demonstrated that our model has great potential of handling extra information to improve document classification performance.