Graph transformer networks based text representation

Abstract

Graph Neural Networks (GNN) has been used to exploit global features in text representation learning for natural language processing (NLP) tasks, including text classification, sequence tagging, neural machine translation and relational reasoning. However, GNN based models usually build a graph for the entire corpus, they have high memory consumption, ignoring the order of words and containing test documents in the training graph. Thus, these models are inherently transductive and have difficulties in inductive learning. In order to solve the above problems, we propose a Graph Transformer Networks based Text representation (GTNT) model. It first constructs a degree-centric text graph, which generates a text graph for each document in the corpus. Then it adopts a graph transformer network to model the graph to obtain node embeddings. When we apply our proposed GTNT model to citation recommendation and text classification tasks, the experimental results show that our model outperforms other state-of-the-art models.