Event Extraction System Research Articles

Is a Large Language Model a Good Annotator for Event Extraction?

Event extraction is an important task in natural language processing that focuses on mining event-related information from unstructured text. Despite considerable advancements, it is still challenging to achieve satisfactory performance in this task, and issues like data scarcity and imbalance obstruct progress. In this paper, we introduce an innovative approach where we employ Large Language Models (LLMs) as expert annotators for event extraction. We strategically include sample data from the training dataset in the prompt as a reference, ensuring alignment between the data distribution of LLM-generated samples and that of the benchmark dataset. This enables us to craft an augmented dataset that complements existing benchmarks, alleviating the challenges of data imbalance and scarcity and thereby enhancing the performance of fine-tuned models. We conducted extensive experiments to validate the efficacy of our proposed method, and we believe that this approach holds great potential for propelling the development and application of more advanced and reliable event extraction systems in real-world scenarios.

An End-to-End Natural Language Processing System for Automatically Extracting Radiation Therapy Events From Clinical Texts

Real-world evidence for radiation therapy (RT) is limited because it is often documented only in the clinical narrative. We developed a natural language processing system for automated extraction of detailed RT events from text to support clinical phenotyping. A multi-institutional data set of 96 clinician notes, 129 North American Association of Central Cancer Registries cancer abstracts, and 270 RT prescriptions from HemOnc.org was used and divided into train, development, and test sets. Documents were annotated for RT events and associated properties: dose, fraction frequency, fraction number, date, treatment site, and boost. Named entity recognition models for properties were developed by fine-tuning BioClinicalBERT and RoBERTa transformer models. A multiclass RoBERTa-based relation extraction model was developed to link each dose mention with each property in the same event. Models were combined with symbolic rules to create a hybrid end-to-end pipeline for comprehensive RT event extraction. Named entity recognition models were evaluated on the held-out test set with F1 results of 0.96, 0.88, 0.94, 0.88, 0.67, and 0.94 for dose, fraction frequency, fraction number, date, treatment site, and boost, respectively. The relation model achieved an average F1 of 0.86 when the input was gold-labeled entities. The end-to-end system F1 result was 0.81. The end-to-end system performed best on North American Association of Central Cancer Registries abstracts (average F1 0.90), which are mostly copy-paste content from clinician notes. We developed methods and a hybrid end-to-end system for RT event extraction, which is the first natural language processing system for this task. This system provides proof-of-concept for real-world RT data collection for research and is promising for the potential of natural language processing methods to support clinical care.

DEES: a real-time system for event extraction from disaster-related web text

DEES: a real-time system for event extraction from disaster-related web text

Comparing neural models for nested and overlapping biomedical event detection

BackgroundNested and overlapping events are particularly frequent and informative structures in biomedical event extraction. However, state-of-the-art neural models either neglect those structures during learning or use syntactic features and external tools to detect them. To overcome these limitations, this paper presents and compares two neural models: a novel EXhaustive Neural Network (EXNN) and a Search-Based Neural Network (SBNN) for detection of nested and overlapping events.ResultsWe evaluate the proposed models as an event detection component in isolation and within a pipeline setting. Evaluation in several annotated biomedical event extraction datasets shows that both EXNN and SBNN achieve higher performance in detecting nested and overlapping events, compared to the state-of-the-art model Turku Event Extraction System (TEES).ConclusionsThe experimental results reveal that both EXNN and SBNN are effective for biomedical event extraction. Furthermore, results on a pipeline setting indicate that our models improve detection of events compared to models that use either gold or predicted named entities.

Event-Argument Linking in Disaster Domain

Linking event triggers with their respective arguments is an essential component for building an event extraction system. It is challenging to link event triggers with the corresponding arguments triggers when the sentence contains multiple events and arguments triggers. The task becomes even more challenging in a low-resource setup due to the unavailability of natural language processing resource and tools. In this paper, we study the event-argument linking task based on disaster event ontology in a low resource setup. We use BERT and non-BERT-based deep learning models in both monolingual and cross-lingual event-argument linking task.We also perform an ablation study of various features like position embeddings (PE), position indicator (PI), and segment ID (SI) to understand their contribution to performance improvement in non-BERT-based models. Using three different languages viz. Hindi, Bengali, and Marathi, we compare the results with multilingual BERT-based deep neural models in both monolingual and cross-lingual scenarios. We observe that the multilingual BERT-based model outperforms the best performing non-BERT-based model in cross-lingual settings. But in monolingual settings, the performance is similar in Hindi and Bengali datasets and slightly better in Marathi dataset. We choose the disaster domain due to its social implications. Our current experiments can be helpful in mining important information related to disaster events from news articles and building event knowledge graphs in low-resource languages.

SENTiVENT: enabling supervised information extraction of company-specific events in economic and financial news

We present SENTiVENT, a corpus of fine-grained company-specific events in English economic news articles. The domain of event processing is highly productive and various general domain, fine-grained event extraction corpora are freely available but economically-focused resources are lacking. This work fills a large need for a manually annotated dataset for economic and financial text mining applications. A representative corpus of business news is crawled and an annotation scheme developed with an iteratively refined economic event typology. The annotations are compatible with benchmark datasets (ACE/ERE) so state-of-the-art event extraction systems can be readily applied. This results in a gold-standard dataset annotated with event triggers, participant arguments, event co-reference, and event attributes such as type, subtype, negation, and modality. An adjudicated reference test set is created for use in annotator and system evaluation. Agreement scores are substantial and annotator performance adequate, indicating that the annotation scheme produces consistent event annotations of high quality. In an event detection pilot study, satisfactory results were obtained with a macro-averaged F_1-score of 59% validating the dataset for machine learning purposes. This dataset thus provides a rich resource on events as training data for supervised machine learning for economic and financial applications. The dataset and related source code is made available at https://osf.io/8jec2/.

Modeling Textual Cohesion for Event Extraction

Event extraction systems typically locate the role fillers for an event by analyzing sentences in isolation and identifying each role filler independently of the others. We argue that more accurate event extraction requires a view of the larger context to decide whether an entity is related to a relevant event. We propose a bottom-up approach to event extraction that initially identifies candidate role fillers independently and then uses that information as well as discourse properties to model textual cohesion. The novel component of the architecture is a sequentially structured sentence classifier that identifies event-related story contexts. The sentence classifier uses lexical associations and discourse relations across sentences, as well as domain-specific distributions of candidate role fillers within and across sentences. This approach yields state-of-the-art performance on the MUC-4 data set, achieving substantially higher precision than previous systems.

Enhancing Event Descriptions through Twitter Mining

We describe a simple IR approach for linking news about events, detected by an event extraction system, to messages from Twitter (tweets). In particular, we explore several methods for creating event-specific queries for Twitter and provide a quantitative and qualitative evaluation of the relevance and usefulness of the information obtained from the tweets. We showed that methods based on utilization of word co-occurrence clustering, domain-specific keywords and named entity recognition improve the performance with respect to a basic approach.

SEEFT: Planned Social Event Discovery and Attribute Extraction by Fusing Twitter and Web Content

Social events comprise some of the most popular topics in social media. Automatically identifying planned social events and extracting structured information, such as event title, date, and location, would enable more effective index, display and search for social events. However, the informal and noisy nature of language used in social media can degrade the quality of event extraction, resulting in broken titles, incorrect or absent attributes - making the resulting event databases not suitable for realistic applications. Previous work mostly focused on event identification and categorization in Twitter. Yet, event title extraction, arguably one of the most useful and difficult tasks in this domain, has never been investigated. In this paper, we address the task of identifying and extracting structured information (titles, dates, locations) for planned social events, and introduce SEEFT, a social event extraction system, which uses social media content to discover events. To extract the event title and other attributes, SEEFT fuses the original social media content and the content of other Tweets and webpages. Experiments over multiple popular event types and more than a thousand of event instances show that SEEFT significantly outperforms the previous state-of-the-art system in event identification. Moreover, by fusing information from multiple sources, SEEFT is able to extract event titles with high accuracy, providing the foundation for practical applications such as event discovery, search, and recommendation.

Cross-Context News Corpus for Protest Event-Related Knowledge Base Construction

Abstract We describe a gold standard corpus of protest events that comprise various local and international English language sources from various countries. The corpus contains document-, sentence-, and token-level annotations. This corpus facilitates creating machine learning models that automatically classify news articles and extract protest event-related information, constructing knowledge bases that enable comparative social and political science studies. For each news source, the annotation starts with random samples of news articles and continues with samples drawn using active learning. Each batch of samples is annotated by two social and political scientists, adjudicated by an annotation supervisor, and improved by identifying annotation errors semi-automatically. We found that the corpus possesses the variety and quality that are necessary to develop and benchmark text classification and event extraction systems in a cross-context setting, contributing to the generalizability and robustness of automated text processing systems. This corpus and the reported results will establish a common foundation in automated protest event collection studies, which is currently lacking in the literature.

Construction Model and Evaluation of Dynamic Knowledge Map for Deep Learning

The related technologies of knowledge map have been widely applied in many fields, such as search engine, intelligent question and answer, language understanding, recommendation calculation, big data decision analysis and so on. Deep learning originates from the research of artificial neural network, which uses deep network to learn knowledge from massive data. Its advantage is that it can obtain features without manual work. It is one of the current research hotspots to integrate the method of deep learning into the application of knowledge map. In this paper, a dynamic knowledge map reasoning method based on multi-relationship cyclic events is proposed, and an improved multi-relationship proximity aggregator is used to fuse the neighborhood information of target entities to obtain a more accurate representation of entity neighborhood vectors. It improves the speed of manual annotation, reduces the time and labor cost of manual annotation, and improves the migration ability of event extraction system.

Shallow and deep learning for event relatedness classification

In the two recent decades various security authorities around the world acknowledged the importance of exploiting the ever-growing amount of information published on the web on various types of events for early detection of certain threats, situation monitoring and risk analysis. Since the information related to a particular real-world event might be scattered across various sources and mentioned on different dates, an important task is to link together all event mentions that are interrelated. This article studies the application of various statistical and machine learning techniques to solve a new application-oriented variation of the task of event pair relatedness classification, which merges different fine-grained event relation types reported elsewhere into one concept. The task focuses on linking event templates automatically extracted from online news by an existing event extraction system, which contain only short text snippets, and potentially erroneous and incomplete information. Results of exploring the performance of shallow learning methods such as decision tree-based random forest and gradient boosted tree ensembles (XGBoost) along with kernel-based support vector machines (SVM) are presented in comparison to both simpler shallow learners as well as a deep learning approach based on long short-term memory (LSTM) recurrent neural network. Our experiments focus on using linguistically lightweight features (some of which not reported elsewhere) which are easily portable across languages. We obtained F1 scores ranging from 92% (simplest shallow learner) to 96.4% (LSTM-based recurrent neural network) evaluated on a newly created event linking corpus.

Interactive learning for joint event and relation extraction

We tackle the problems of both event and entity relation extraction, and come up with a novel method to implement joint extraction: iteratively interactive learning. This method is motivated by the empirical findings as below: the extracted event attributes (e.g., trigger and event type) can be used as the reliable features for the recognition of entity relation types, and vice versa. Accordingly, on one hand, we utilize the predicted event attributes (by a certain event extraction system) to remodel the distributed representations of features for entity relation extraction, and on the other hand, we use entity relations (recognized by a certain relation extraction system) to remodel the features for event extraction. This enables a double-channel task-independent joint model with an interactive learning: learning events for relation extraction, and meanwhile learning relations for event extraction. In practice, we perform the interactive learning in an iterative manner, so as to boost the joint model progressively. Methodologically, we take the neural network of bidirectional long short-term memory (Bi-LSTM) for learning event and relation respectively. And as usual, the attention mechanism is used. In our experiments, the automatic content extraction corpus is used for the evaluation of the proposed method. Such a corpus consists of event, entity and relation samples with gold-standard attribute tags. Experimental results show that our method outperforms the baselines (Bi-LSTMs with attention without interactive learning) in both event and relation extraction tasks, yielding performance gains of about 1.6% and 1.8% F-scores respectively, at the condition of low-resource setting.

Exploring Multiple Embedded Features on Event Extraction

In recent years, the neural network method can automatically learn effectively features. Unlike traditional discrete features, neural network features are mostly continuous features and can be automatically combined to build higher-level features. The efficiency of the features has been proven in numerous tasks in natural language processing and has led to breakthroughs. In this paper, we propose a event extraction system based on combination of multiple embedded features. Our work is mainly based on the three aspects: (1) traditional pipeline systems have serious error propagation problems; (2) there are several different event descriptions in the text; (3) representation learning can provide rich semantic and syntactic representation. As a result, we achieve competitive performance, specifically, F1-measure of 60.25 in event extraction. Meanwhile, evaluation results point out some shortcomings that need to be addressed in future work.

An automatic multi-camera-based event extraction system for real soccer videos

In this article, we propose a novel and effective system based on multiple cameras to extract the events for soccer matches. A precise ontological definition of the soccer events is still an open point. According to our definition, the events include the free kick, corner kick, penalty kick and the goal, because they are the representative shots for the audience to watch. The events are very important for highlights selection and sport data analysis. At present, the events including the ball and players information are selected and labeled manually from the images, which is a big workload for the staffs. Addressing this problem, our system provides an automatic extraction of the events. For soccer videos, our system first uses the local-based deep neural network for the ball and player detection from the input images. Then, we handle with the ball and player bounding boxes separately. For players, a player can be labeled as one of the three types: two teams or the referee, and a novel unsupervised U-encoder is designed for the player labeling. For soccer ball, the application of multiple cameras allows us to refine the ball detection results. We can get the world coordinate of ball according to the camera parameters and then rebuild the ball trajectory and the court in a top view. Based on the reconstructed map, we get the soccer events by motion analysis of ball trajectory and then apply the ball location and player classification results to display the events for each camera. The test results on real videos of European soccer league show the good detection and labeling performance of our system. We find all the events in the test videos. Our proposed system can deal with many complex cases such as occlusion and pose variation that happen frequently in real applications.

Domain transformation on biological event extraction by learning methods.

Event extraction and annotation has become a significant focus of recent efforts in biological text mining and information extraction (IE). However, event extraction, event annotation methods, and resources have so far focused almost exclusively on a single domain. State-of-the-art studies on biological event extraction and annotation are typically domain-dependent and domain-restricted. In this paper, we adopt an approach aimed at extracting events and relations for two different tasks by generating a common dataset using transfer learning and structural correspondence learning (SCL). A deep learning event extraction system was developed to evaluate our results. Our approach comprises two stages: (1) generating a dataset from two independent event extraction tasks or domains, and (2) using a classifier model to learn feature patterns from the generated dataset for event and relation extraction. The classifier in the proposed model can extract events and relations irrespective of the domain of the test input. Our study shows that this approach performs competitively compared to domain specific or dependent tasks.

Enhancing metabolic event extraction performance with multitask learning concept.

To extract and generate a valid metabolic pathway from research articles, biologists need substantial amounts of time to digest unstructured text. Text mining currently plays a central role in this research area, because it provides the ability to automatically discover useful information in a reasonable time. A text mining model can be built using a training data or a corpus in supervised manner. Unfortunately, a corpus of the domain of interest may not be always available or insufficient in practice, because a corpus construction is a labor-intensive task and needs specialist annotation. In this paper, we developed an event extraction system, a text-mining task, to extract metabolic interactions from research literature and then reconstruct metabolic pathways. The proposed system consists of the pipeline of four supervised-learning steps: named entity recognition, trigger detection, edge detection, and event reconstruction. We also introduced a multitask-learning algorithm, a transfer-learning paradigm, that can leverage additional resources of an existing source domain to facilitate a classification of the metabolic event extraction in the target domain. To demonstrate a proof of concept, edge detection, a core step in our event extraction system, was used as a case study in multitask-learning classification. The experimental results showed that the proposed event extraction system provided competitive performance against those of state-of-the-art related system. In particular, the proposed multitask-learning can improve the performance of edge detection, therefore the overall performance of the event extraction system was also improved accordingly.

Bio-molecular event extraction by integrating multiple event-extraction systems

Event extraction from biomedical text is a very important task in text mining and natural language processing. The overall task involves finding event-related expressions, classifying these into predefined categories and attaching arguments to these events. We perform event detection and event classification in one step using an ensemble of classifiers. For event argument extraction, we also use an ensemble of classification models. Our base models are developed using supervised machine learning that makes use of statistical, contextual and syntactic features. Our experimental result on the benchmark datasets of BioNLP-2011 shared task shows the recall, precision and F-measure values of 51.20%, 65.78% and 57.58%, respectively.

Leveraging event-based semantics for automated text simplification

Automated Text Simplification (ATS) aims to transform complex texts into their simpler variants which are easier to understand to wider audiences and easier to process with natural language processing (NLP) tools. While simplification can be applied on lexical, syntactic, and discourse level, all previously proposed ATS systems only operated on the first two levels, thus failing at simplifying texts on the discourse level. We present a semantically-motivated ATS system which is the first system that is applied on the discourse level. By exploiting the state-of-the-art event extraction system, it is the first ATS system able to eliminate large portions of irrelevant information from texts, by maintaining only those parts of the original text that belong to factual event mentions. A few handcrafted rules ensure that the output of the system is syntactically simple, by placing each factual event mention in a separate short sentence, while the state-of-the-art unsupervised lexical simplification module, based on using word embeddings, replaces complex and infrequent words with their simpler variants. We perform a thorough evaluation, both automatic and manual, showing that our system produces more readable and simpler texts than the state-of-the-art ATS systems. Our newly proposed post-editing evaluation further reveals that our system requires less human effort for correcting grammaticality and meaning preservation on news articles than the state-of-the-art ATS system.

One, no one and one hundred thousand events: Defining and processing events in an inter-disciplinary perspective

AbstractWe present an overview of event definition and processing spanning 25 years of research in NLP. We first provide linguistic background to the notion of event, and then present past attempts to formalize this concept in annotation standards to foster the development of benchmarks for event extraction systems. This ranges from MUC-3 in 1991 to the Time and Space Track challenge at SemEval 2015. Besides, we shed light on other disciplines in which the notion of event plays a crucial role, with a focus on the historical domain. Our goal is to provide a comprehensive study on event definitions and investigate which potential past efforts in the NLP community may have in a different research domain. We present the results of a questionnaire, where the notion of event for historians is put in relation to the NLP perspective.