Adapting Open Domain Fact Extraction and Verification to COVID-FACT through In-Domain Language Modeling

Zhenghao Liu,Zhuyun Dai,Zhiyuan Liu,Chenyan Xiong,Maosong Sun,Si Sun

doi:10.18653/v1/2020.findings-emnlp.216

Abstract

With the epidemic of COVID-19, verifying the scientifically false online information, such as fake news and maliciously fabricated statements, has become crucial. However, the lack of training data in the scientific domain limits the performance of fact verification models. This paper proposes an in-domain language modeling method for fact extraction and verification systems. We come up with SciKGAT to combine the advantages of open-domain literature search, state-of-the-art fact verification systems and in-domain medical knowledge through language modeling. Our experiments on SCIFACT, a dataset of expert-written scientific fact verification, show that SciKGAT achieves 30% absolute improvement on precision. Our analyses show that such improvement thrives from our in-domain language model by picking up more related evidence pieces and accurate fact verification. Our codes and data are released via Github.

Highlights

This paper proposes an in-domain language modeling method for fact extraction and verification systems
Some work (Beltagy et al, 2019; Lee et al, 2020) transfers medical domain knowledge into pre-trained language models for better medical semantic understanding, which provides a potential way to deal with COVID-FACT checking problem
We evaluate the impacts of the in-domain language model on individual fact extraction and verification components of Scientific KGAT (SciKGAT)

Summary

Introduction

This paper proposes an in-domain language modeling method for fact extraction and verification systems. Our indomain language modelings improve the fact verification performance with more than 10% absolute F1 score and 30% absolute precision (from 46.6% to 76%) than previous state-of-the-art on SCIFACT. Such improvement shows that our model provides a set of solutions for low-resource fact verification tasks, such as COVID-19. The small-scale training data of SCIFACT may Existing fact extraction and verification models usually employ a three-step pipeline system (Chen et al, 2017): document retrieval (abstract retrieval), sentence selection (rationale selection) and fact verification (Thorne et al, 2018; Wadden et al, 2020)

Objectives

Methods

Results