Learning Sentence Embeddings Based on Weighted Contexts from Unlabelled Data

Abstract

Supervised learning and unsupervised learning are mainstream methods to solve semantic textual similarity tasks. However, it is obvious that supervised learning needs substantial labeled data which is hard to obtain in reality. Therefore, we turn our attention to construct sentence embeddings using unlabelled data due to lack of annotated data and success of unsupervised word embeddings in multiple tasks. We present a simple but efficient unsupervised learning method of sentence embeddings inspired by attention mechanism, in which weighted contexts are added to models to train distributed sentence representations inspired by word2vec. Our method outperforms state-of-the-art unsupervised models on semantic textual similarity tasks.