Pingan Smart Health and SJTU at COIN - Shared Task: utilizing Pre-trained Language Models and Common-sense Knowledge in Machine Reading Tasks

Xiepeng Li,Guotong Xie,Wei Zhu,Peng Gao,Zheng Li,Zhexi Zhang,Junchi Yan,Yuan Ni

doi:10.18653/v1/d19-6011

Abstract

To solve the shared tasks of COIN: COmmonsense INference in Natural Language Processing) Workshop in , we need explore the impact of knowledge representation in modeling commonsense knowledge to boost performance of machine reading comprehension beyond simple text matching. There are two approaches to represent knowledge in the low-dimensional space. The first is to leverage large-scale unsupervised text corpus to train fixed or contextual language representations. The second approach is to explicitly express knowledge into a knowledge graph (KG), and then fit a model to represent the facts in the KG. We have experimented both (a) improving the fine-tuning of pre-trained language models on a task with a small dataset size, by leveraging datasets of similar tasks; and (b) incorporating the distributional representations of a KG onto the representations of pre-trained language models, via simply concatenation or multi-head attention. We find out that: (a) for task 1, first fine-tuning on larger datasets like RACE (Lai et al., 2017) and SWAG (Zellersetal.,2018), and then fine-tuning on the target task improve the performance significantly; (b) for task 2, we find out the incorporating a KG of commonsense knowledge, WordNet (Miller, 1995) into the Bert model (Devlin et al., 2018) is helpful, however, it will hurts the performace of XLNET (Yangetal.,2019), a more powerful pre-trained model. Our approaches achieve the state-of-the-art results on both shared task’s official test data, outperforming all the other submissions.

Highlights

Machine reading comprehension (MRC) tasks have always been the most studied tasks in the field of natural language understanding
Kaushik and Lipton (2018) demonstrate that most questions in previous MRC tasks can be answered by matching the patterns in the textual level even with passage or question only, but existing models perform badly on questions that require incorporating knowledge in more sophisticated ways
Human beings can reason with knowledge from contexts or commonsense knowledge when doing MRC task

Summary

Introduction

Machine reading comprehension (MRC) tasks have always been the most studied tasks in the. (Zhang et al, 2019; Ran et al, 2019) compute the contextual representation of passages, questions and options separately with BERT and match the representation in downstream networks They achieved the best results on RACE dataset at their submission time. Shared task 2 uses the ReCoRD dataset (Zhang et al, 2018a), a machine reading comprehension dataset in news articles It annotates named entities in the news articles and provides some brief bullet points that summarize the news. It asks for cloze-style answers, filling in a blank in a sentence related to the news article Accomplishing these tasks requires both the capability of reading comprehension and commonsense knowledge inference. Our system achieves state-of-the-art performance on the both shared tasks’ official test data, even though we only train on the train sets and only submit single models

Pretrained language model Fine-tuning

Multi-funetuning

Knowledge fusing

Answer Verification

Dataset

Experimental setting

Results

Conclusions