A Window-Based Self-Attention approach for sentence encoding

Abstract

Recently, much progress has been made in the representation of sentences for the task of natural language processing (NLP). Most of existing methods utilize deep learning methods with RNN/CNN to capture contextual information. These models typically treat each word in a sentence equally, which ignores the fact that keywords always play a leading role in expressing the sentence semantic. Especially, on the tasks of sentence classification and semantic relatedness prediction, we can judge through the information of several keywords, no need for the whole sentence. To this end, we propose a window-based intra-weighing approach to weigh words in the sentence. In calculating attentive weights, we take as input multi-window n-grams and use max-pooling for feature extraction. To evaluate our model, we conduct experiments on 6 benchmark data sets. The experimental results demonstrate the superiority of our model in calculating word importance. Although our model has fewer parameters and much lower computational complexity than state-of-the-art models, it achieves comparable results with them.