Abstract

In recent years, distantly-supervised relation extraction has achieved a certain success by using deep neural networks. Distant Supervision (DS) can automatically generate large-scale annotated data by aligning entity pairs from Knowledge Bases (KB) to sentences. However, these DS-generated datasets inevitably have wrong labels that result in incorrect evaluation scores during testing, which may mislead the researchers. To solve this problem, we build a new dataset NYTH, where we use the DS-generated data as training data and hire annotators to label test data. Compared with the previous datasets, NYT-H has a much larger test set and then we can perform more accurate and consistent evaluation. Finally, we present the experimental results of several widely used systems on NYT-H. The experimental results show that the ranking lists of the comparison systems on the DS-labelled test data and human-annotated test data are different. This indicates that our human-annotated data is necessary for evaluation of distantly-supervised relation extraction.

Highlights

  • There has been significant progress on Relation Extraction (RE) in recent years using models based on machine learning algorithms (Mintz et al, 2009; Hoffmann et al, 2011; Zeng et al, 2015; Zhou et al, 2016; Ji et al, 2017; Su et al, 2018; Qin et al, 2018b; Zhang et al, 2019)

  • The first one is Distant Supervision (DS) labels as Ground Truths (DSGT), which assumes that all the DS labels are ground truths

  • The second one is the Manual Annotation as Ground Truth (MAGT), where the metrics are calculated on the same evaluation set, but uses the human-annotated labels as truths

Read more

Summary

Introduction

There has been significant progress on Relation Extraction (RE) in recent years using models based on machine learning algorithms (Mintz et al, 2009; Hoffmann et al, 2011; Zeng et al, 2015; Zhou et al, 2016; Ji et al, 2017; Su et al, 2018; Qin et al, 2018b; Zhang et al, 2019). The task of RE is to identify semantic relationships among entities from texts. Traditional supervised methods require a massive amount of annotated data, which are often labelled by human annotators. It is hard to annotate data within strict time limits and hiring the annotators is non-scalable and costly.

Methods
Results
Conclusion

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call

Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.