Abstract

Open Information Extraction systems extract (“subject text”, “relation text”, “object text”) triples from raw text. Some triples are textual versions of facts, i.e., non-canonicalized mentions of entities and relations. In this paper, we investigate whether it is possible to infer new facts directly from the open knowledge graph without any canonicalization or any supervision from curated knowledge. For this purpose, we propose the open link prediction task,i.e., predicting test facts by completing (“subject text”, “relation text”, ?) questions. An evaluation in such a setup raises the question if a correct prediction is actually a new fact that was induced by reasoning over the open knowledge graph or if it can be trivially explained. For example, facts can appear in different paraphrased textual variants, which can lead to test leakage. To this end, we propose an evaluation protocol and a methodology for creating the open link prediction benchmark OlpBench. We performed experiments with a prototypical knowledge graph embedding model for openlink prediction. While the task is very challenging, our results suggests that it is possible to predict genuinely new facts, which can not be trivially explained.

Highlights

  • A knowledge graph (KG) (Hayes-Roth, 1983) is a set of-triples, where the subject and object correspond to vertices, and relations to labeled edges

  • We can view Open information extraction systems (OIE) data as an open knowledge graph (OKG) (Galarraga et al, 2014), in which vertices correspond to mentions of entities and edges to open relations

  • To experimentally explore whether it is possible to predict new facts, we focus on knowledge graph embedding (KGE) models (Nickel et al, 2016), which have been applied successfully to LP in KGs

Read more

Summary

Introduction

A knowledge graph (KG) (Hayes-Roth, 1983) is a set of (subject, relation, object)-triples, where the subject and object correspond to vertices, and relations to labeled edges. We can view OIE data as an open knowledge graph (OKG) (Galarraga et al, 2014), in which vertices correspond to mentions of entities and edges to open relations (see Fig. 1). Given the question (“NBC-TV”, “has office in”, ?), correct answers include “NYC” and “New York”; see Fig. 2b). A simple but problematic way to transfer this approach to OKGs is to sample a set of evaluation triples from the OKG and to use the remaining part of the OKG for training To see why this approach is problematic, consider the test triple (“NBC-TV”, “has office in”, “New York”) and suppose that the triple (“NBC”, “has headquarter in”, “NYC”) is part of the OKG. We show that paraphrasing and non-relational information can dilute performance evaluation, but can be remedied by appropriate dataset construction and experimental settings

Open Knowledge Graphs
Evaluation protocol
Creating the Open Link Prediction Benchmark OLPBENCH
Source Dataset
Evaluation Data
Training Data
Open Knowledge Graph Embeddings
Models and Training
Results
Conclusion
A Related Work
B Dataset creation
Multi-Label Binary Classification Batch-Negative Example Loss
Training settings
D Performance Metrics
E Additional Results
Full Text
Published version (Free)

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call