A Siamese Neural Network for Learning Semantically-Informed Sentence Embeddings

Abstract

Semantic representation is a way of expressing the meaning of a text that can be processed by a machine to serve a particular natural language processing (NLP) task that usually requires meaning comprehension such as text summarisation, question answering or machine translation. In this paper, we present a semantic parsing model based on neural networks to obtain semantic representation of a given sentence. We utilise semantic representation of each sentence to generate semantically informed sentence embeddings for extrinsic evaluation of the proposed semantic parser, in particular for the semantic textual similarity task. Our neural parser utilises self-attention mechanism to learn semantic relations between words in a sentence to generate semantic representation of a sentence in UCCA (Universal Conceptual Cognitive Annotation) semantic annotation framework (Abend and Rappoport, 2013), which is a cross-linguistically applicable graph-based semantic representation. The UCCA representations are conveyed into a Siamese Neural Network built on top of two Recursive Neural Networks (Siamese-RvNN) to derive semantically informed sentence embeddings which are evaluated on semantic textual similarity task. We conduct both single-lingual and cross-lingual experiments with zero-shot and few-shot learning, which have shown superior performance even in low-resource scenario. The experimental results show that the proposed self-attentive neural parser outperforms the other parsers in the literature on English and German, and shows significant improvement in the cross-lingual setting for French which has comparatively low sources. Moreover, the results obtained from other downstream tasks such as sentiment analysis confirm that semantically informed sentence embeddings provide higher-quality embeddings compared to other pre-trained models such as SBERT (Reimers et al., 2019) or SimCSE (Gao et al., 2021), which do not utilise such structured information.