A label-oriented loss function for learning sentence representations

Abstract

Neural network methods which leverage word-embedding obtained from unsupervised learning models have been widely adopted in many natural language processing (NLP) tasks, including sentiment analysis and sentence classification. Existing sentence representation generation approaches which serve for classification tasks generally rely on complex deep neural networks but relatively simple loss functions, such as cross entropy loss function. These approaches cannot produce satisfactory separable sentence representations because the usage of cross entropy may ignore the sentiment and semantic information of the labels. To extract useful information from labels for improving the distinguishability of the obtained sentence representations, this paper proposes a label-oriented loss function. The proposed loss function takes advantage of the word-embeddings of labels to guide the production of meaningful sentence representations which serve for downstream classification tasks. Compared with existing end-to-end approaches, the evaluation experiments on several datasets illustrate that using the proposed loss function can achieve competitive and even better classification results.