Abstract

There is an increasing interest in developing text-based relational reasoning systems, which are capable of systematically reasoning about the relationships between entities mentioned in a text. However, there remains a substantial performance gap between NLP models for relational reasoning and models based on graph neural networks (GNNs), which have access to an underlying symbolic representation of the text. In this work, we investigate how the structured knowledge of a GNN can be distilled into various NLP models in order to improve their performance. We first pre-train a GNN on a reasoning task using structured inputs and then incorporate its knowledge into an NLP model (e.g., an LSTM) via knowledge distillation. To overcome the difficulty of cross-modal knowledge transfer, we also employ a contrastive learning based module to align the latent representations of NLP models and the GNN. We test our approach with two state-of-the-art NLP models on 13 different inductive reasoning datasets from the CLUTRR benchmark and obtain significant improvements.

Highlights

  • The task of text-based relational reasoning—where an agent must infer and compose relations between entities based on a passage of text—has received increasing attention in natural language processing (NLP) (Andreas, 2019)

  • We describe our approach for structured distillation, which involves improving the performance of an NLP model by distilling structured knowledge from a graph neural networks (GNNs) (Fig. 1)

  • Our key experimental question is whether an NLP model can be improved by distilling structured knowledge from a GNN. We investigate this question using the GNN and NLP models defined in the previous section, and we follow the experimental protocol from Sinha et al (2019)

Read more

Summary

Introduction

The task of text-based relational reasoning—where an agent must infer and compose relations between entities based on a passage of text—has received increasing attention in natural language processing (NLP) (Andreas, 2019) This task has been especially prominent in the context of systematic generalization in NLP, with synthetic datasets, such as CLUTTR and SCAN, being used to probe the ability of NLP models to reason in a systematic and logical way (Lake and Baroni, 2018; Sinha et al, 2019). Perhaps one of the biggest challenges is the persistent gap between the performance that can be achieved using NLP models and the performance of structured models—such as graph neural networks (GNNs)—which perform relational reasoning based on structured or symbolic inputs. CLUTRR includes relational reasoning problems that can be posed both in textual or symbolic form, and preliminary investigations using CLUTRR show that GNN-based models—which leverage the structured symbolic input—are able to achieve higher accuracy, better generalization, and are more robust than purely text-based systems (Sinha et al, 2019)

Objectives
Methods
Findings
Conclusion
Full Text
Published version (Free)

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call