Knowledge attention sandwich neural network for text classification

Abstract

Text Classification is a fundamental and crucial issue in many Natural Language Processing (NLP) tasks. An effective and efficient representation model is the key to text classification. However, most existing representation models either learn insufficient structural information or just rely on pre-defined structures, resulting in degradation of performance and generalization capability. We propose a novel Sandwich Neural Network (SNN), which is able to learn local semantic and global structural representations automatically without parsers. To combine semantic and structural representations sensibly, we propose four fusion methods incorporated with SNN: Static Fusion, Adaptive Learning, Self-Attention, and Knowledge Attention methods. Static Fusion weights semantic and structural representations equally, Adaptive Learning learns the weights at corpus level, and Self-Attention learns the weights at instance level. More importantly, within Knowledge Attention fusion method, external semantic and structural knowledge are incorporated into SNN to improve the attention procedure and further boost the performance of SNN. Evaluated with four mainstream datasets: Text REtrieval Conference (TREC), SUBJectivity (SUBJ), Movie Reviews (MR) and Stanford Sentiment Treebank with Five Labels (SST-5), the proposed Knowledge Attention Sandwich Neural Network(KA-SNN) model achieves very competitive performance. Specifically, the accuracies achieve 96.2% (TREC), 94.1% (SUBJ), 82.3% (MR) and 51.5% (SST-5). Moreover, the proposed Knowledge Attention reduces the structural complexity of attention module by 77.66-81.43% with a computing time reduction of 21.47-34.05%, compared with Self-Attention fusion method.