TAC KBP Research Articles

Event detection with multi-order edge-aware graph convolution networks

Event detection, specifically, identifying the type of events in a piece of text, is a crucial task in information extraction. Previous event detection task research proved that a syntactic graph based on a dependency tree can be integrated into a graph convolutional neural network to better capture the context of a sentence. However, most of the existing studies rely only on first-order syntactic relations and usually ignore dependence label information. In this paper, we propose a multi-order edge-aware graph convolution network (MEA-GCN) based on multi-order grammar and typed dependent label information. We use the BERT representation and the multi-head attention update mechanism to generate a variety of multi-order syntactic graph representations; consequently, our model can automatically learn dependent information and improve the representation ability of the syntactic relations. The experimental results on the widely used ACE 2005 and TAC KBP 2015 show that our model achieves significant improvement over the competitive baseline methods.

HEFT: A History-Enhanced Feature Transfer framework for incremental event detection

Incremental event detection is a challenging subfield of event detection, which could be applied in various complex situations, especially for the real-time streaming data scenario where new event classes are continuously emerging. To handle the catastrophic forgetting problem, knowledge distillation has been proved to be a promising mechanism since it requires no extra tools or data storage. However, conventional knowledge distillation-based methods are not effective due to two challenging issues: (a) there is an inherent conflict between reserving history information from old tasks and adjusting to new tasks with shared model output features. (b) the historical category information is not fully utilized, making the model prone to forgetting knowledge on the observed tasks. In this work, we present a History-Enhanced Feature Transfer (HEFT) framework to solve the two challenges mentioned above. For the first challenge, we employ a feature transfer module to separate reserving and adjusting sub-functions by reconstructing the old task features. In this manner, the reconstructed features are responsible for preserving historical knowledge, and the feature extractor with followed classifier only requires to focus on new task classification. For the second issue, a history-enhanced question answering model is introduced to reinforce the ability to memorize historical information, which leverages the trained categories as clues to guide the reasoning process. The experimental results show HEFT outperforms the state-of-the-art model by 6.2% and 9.1% of the whole F1 score on ACE 2005 and TAC KBP 2017 benchmarks, respectively. The ablation study and case study also demonstrate that HEFT could overcome catastrophic forgetting of old class event triggers without heavily relying on historically preserved samples.

DeepType 2: Superhuman Entity Linking, All You Need Is Type Interactions

Across multiple domains from computer vision to speech recognition, machine learning models have been shown to match or outperform human experts at recognition tasks. We lack such a comparison point for Entity Linking. We construct a human benchmark on two standard datasets (TAC KBP 2010 and AIDA (YAGO)) to measure human accuracy. We find that current systems still fall short of human performance. We present DeepType 2, a novel entity linking system that closes the gap. Our proposed approach overcomes shortcomings of previous type-based entity linking systems, and does not use pre-trained language models to reach this level. Three key innovations are responsible for DeepType 2's performance: 1) an abstracted representation of entities that favors shared learning and greater sample efficiency, 2) autoregressive entity features indicating type interactions (e.g. list type homogeneity, shared employers, geographical co-occurrence) with previous predictions that enable globally coherent document-wide predictions, 3) the entire model is trained end to end using a single entity-level maximum likelihood objective function. This is made possible by associating a context-specific score to each of the entity's abstract representation's sub-components (types), and summing these scores to form a candidate entity logit. In this paper, we explain how this factorization focuses the learning on the salient types of the candidate entities. Furthermore, we show how the scores can serve as a rationale for predictions. The key contributions of this work are twofold: 1) we create the first human performance benchmark on standard benchmarks in entity linking (TAC KBP 2010 and AIDA (YAGO)) which will be made publicly available to support further analyses, 2) we obtain a new state of the art on these datasets and are the first to outperform humans on our benchmark. We perform model ablations to measure the contribution of the different facets of our system. We also include an analysis of human and algorithmic errors to provide insights into the causes, notably originating from journalistic style and historical context.

HGEED: Hierarchical graph enhanced event detection

Existing methods that use document-level information for event detection ignore the dependencies between sentences and also have shortcomings in modeling the dependencies among words. In this paper, we propose a novel Hierarchical Graph Enhanced Event Detection (HGEED) framework to make full use of syntax and document information for the task of event detection. First, a sentence graph is used to model word-to-word dependencies, enriching the local information of words by incorporating syntactic features. Then, a document graph is built to model sentence-to-sentence dependencies, obtaining global semantic representations for word-level prediction. The experiment results on the widely used ACE 2005 and TAC KBP 2015 corpora show that our model can capture local and global information with dependencies and achieve significant improvements as compared to all baselines.

Joint event extraction along shortest dependency paths using graph convolutional networks

Event extraction (EE) is one of the core information extraction tasks, whose purpose is to automatically identify and extract information about incidents and their actors from texts. This may be beneficial to several domains such as knowledge base construction, question answering and summarization tasks, to name a few. The problem of extracting event information from texts is longstanding and usually relies on elaborately designed lexical and syntactic features, which, however, take a large amount of human effort and lack generalization. More recently, deep neural network approaches have been adopted as a means to learn underlying features automatically. However, existing networks do not make full use of syntactic features, which play a fundamental role in capturing very long-range dependencies. Also, most approaches extract each argument of an event separately without considering associations between arguments which ultimately leads to low efficiency, especially in sentences with multiple events. To address the above-referred problems, we propose a novel joint event extraction framework that aims to extract multiple event triggers and arguments simultaneously by introducing shortest dependency path in the dependency graph. We do this by eliminating irrelevant words in the sentence, thus capturing long-range dependencies. Also, an attention-based graph convolutional network is proposed, to carry syntactically related information along the shortest paths between argument candidates that captures and aggregates the latent associations between arguments; a problem that has been overlooked by most of the literature. Our results show a substantial improvement over state-of-the-art methods on two datasets, namely ACE 2005 and TAC KBP 2015.

A Machine Learning Filter for the Slot Filling Task

Slot Filling, a subtask of Relation Extraction, represents a key aspect for building structured knowledge bases usable for semantic-based information retrieval. In this work, we present a machine learning filter whose aim is to enhance the precision of relation extractors while minimizing the impact on the recall. Our approach consists in the filtering of relation extractors’ output using a binary classifier. This classifier is based on a wide array of features including syntactic, semantic and statistical features such as the most frequent part-of-speech patterns or the syntactic dependencies between entities. We experimented the classifier on the 18 participating systems in the TAC KBP 2013 English Slot Filling track. The TAC KBP English Slot Filling track is an evaluation campaign that targets the extraction of 41 pre-identified relations (e.g., title, date of birth, countries of residence, etc.) related to specific named entities (persons and organizations). Our results show that the classifier is able to improve the global precision of the best 2013 system by 20.5% and improve the F1-score for 20 relations out of 33 considered.

DeepType: Multilingual Entity Linking by Neural Type System Evolution

The wealth of structured (e.g. Wikidata) and unstructured data about the world available today presents an incredible opportunity for tomorrow's Artificial Intelligence. So far, integration of these two different modalities is a difficult process, involving many decisions concerning how best to represent the information so that it will be captured or useful, and hand-labeling large amounts of data.DeepType overcomes this challenge by explicitly integrating symbolic information into the reasoning process of a neural network with a type system.First we construct a type system, and second, we use it to constrain the outputs of a neural network to respect the symbolic structure. We achieve this by reformulating the design problem into a mixed integer problem: create a type system and subsequently train a neural network with it. In this reformulation discrete variables select which parent-child relations from an ontology are types within the type system, while continuous variables control a classifier fit to the type system. The original problem cannot be solved exactly, so we propose a 2-step algorithm: 1) heuristic search or stochastic optimization over discrete variables that define a type system informed by an Oracle and a Learnability heuristic, 2) gradient descent to fit classifier parameters.We apply DeepType to the problem of Entity Linking on three standard datasets (i.e. WikiDisamb30, CoNLL (YAGO), TAC KBP 2010) and find that it outperforms all existing solutions by a wide margin, including approaches that rely on a human-designed type system or recent deep learning-based entity embeddings, while explicitly using symbolic information lets it integrate new entities without retraining.

On the Importance of Drill-Down Analysis for Assessing Gold Standards and Named Entity Linking Performance

Rigorous evaluations and analyses of evaluation results are key towards improving Named Entity Linking systems. Nevertheless, most current evaluation tools are focused on benchmarking and comparative evaluations. Therefore, they only provide aggregated statistics such as precision, recall and F1-measure to assess system performance and no means for conducting detailed analyses up to the level of individual annotations.This paper addresses the need for transparent benchmarking and fine-grained error analysis by introducing Orbis, an extensible framework that supports drill-down analysis, multiple annotation tasks and resource versioning. Orbis complements approaches like those deployed through the GERBIL and TAC KBP tools and helps developers to better understand and address shortcomings in their Named Entity Linking tools.We present three uses cases in order to demonstrate the usefulness of Orbis for both research and production systems: (i) improving Named Entity Linking tools; (ii) detecting gold standard errors; and (iii) performing Named Entity Linking evaluations with multiple versions of the included resources.

Plato: A Selective Context Model for Entity Resolution

We present Plato, a probabilistic model for entity resolution that includes a novel approach for handling noisy or uninformative features, and supplements labeled training data derived from Wikipedia with a very large unlabeled text corpus. Training and inference in the proposed model can easily be distributed across many servers, allowing it to scale to over 107 entities. We evaluate Plato on three standard datasets for entity resolution. Our approach achieves the best results to-date on TAC KBP 2011 and is highly competitive on both the CoNLL 2003 and TAC KBP 2012 datasets.

HLTCOE Participation in TAC KBP 2015: Cold Start and TEDL

The JHU HLTCOE participated in the Cold Start and the Entity Discovery and Linking tasks of the 2016 Text Analysis Conference Knowledge Base Population evaluation. For our fifth year of participation in Cold Start we continued our research with the KELVIN system. We submitted experimental variants that explore use of linking to Freebase across three languages and add relations beyond those required by Cold Start. This is our second year of participation in EDL. We used KELVIN in all runs.