Enhancing pretrained language models with structured commonsense knowledge for textual inference

Abstract

Transformer-based pretrained language models have shown promising performances in various textual inference tasks. However, additional relational knowledge between the semantic units within the input text may also play a critical role in the inference process. To equip the pretrained language models with the relational knowledge, previous methods employ a retrieval-based strategy, which obtain the relational features from prebuilt knowledge bases by a lookup operation. However, the inherent sparsity of part of the knowledge bases would prevent the direct retrieval of the relational features. To address this issue, we propose a MIX-strategy based Structural commonsense integration framework (Mix-Sci). In addition to the traditional retrieval strategy which only adapts to the knowledge bases with high coverage, Mix-Sci introduces an additional generative strategy to incorporate the sparse knowledge bases with the pretrained language models. In specific, in the training process, Mix-Sci learns to generate the structural information of the knowledge base, including the embedding of nodes and the connection relationship between the nodes. So that in the test process, the structural information can be generated to enhance the inference process. Experimental results on two textual inference tasks: machine reading comprehension and event prediction show that Mix-Sci can effectively utilize both the dense and the sparse knowledge bases, to consistently improve the performance of pretrained language models on textual inference tasks.