Approach For Relation Extraction Research Articles

Unsupervised literature mining approaches for extracting relationships pertaining to habitats and reproductive conditions of plant species

IntroductionFine-grained, descriptive information on habitats and reproductive conditions of plant species are crucial in forest restoration and rehabilitation efforts. Precise timing of fruit collection and knowledge of species' habitat preferences and reproductive status are necessary especially for tropical plant species that have short-lived recalcitrant seeds, and those that exhibit complex reproductive patterns, e.g., species with supra-annual mass flowering events that may occur in irregular intervals. Understanding plant regeneration in the way of planning for effective reforestation can be aided by providing access to structured information, e.g., in knowledge bases, that spans years if not decades as well as covering a wide range of geographic locations. The content of such a resource can be enriched with literature-derived information on species' time-sensitive reproductive conditions and location-specific habitats.MethodsWe sought to develop unsupervised approaches to extract relationships pertaining to habitats and their locations, and reproductive conditions of plant species and corresponding temporal information. Firstly, we handcrafted rules for a traditional rule-based pattern matching approach. We then developed a relation extraction approach building upon transformer models, i.e., the Text-to-Text Transfer Transformer (T5), casting the relation extraction problem as a question answering and natural language inference task. We then propose a novel unsupervised hybrid approach that combines our rule-based and transformer-based approaches.ResultsEvaluation of our hybrid approach on an annotated corpus of biodiversity-focused documents demonstrated an improvement of up to 15 percentage points in recall and best performance over solely rule-based and transformer-based methods with F1-scores ranging from 89.61 to 96.75% for reproductive condition - temporal expression relations, and ranging from 85.39% to 89.90% for habitat - geographic location relations. Our work shows that even without training models on any domain-specific labeled dataset, we are able to extract relationships between biodiversity concepts from literature with satisfactory performance.

Extracting Drug-Protein Relation from Literature Using Ensembles of Biomedical Transformers.

Automatic extraction of relations between drugs/chemicals and proteins from ever-growing biomedical literature is required to build up-to-date knowledge bases in biomedicine. To promote the development of automated methods, BioCreative-VII organized a shared task - the DrugProt track, to recognize drug-protein entity relations from PubMed abstracts. We participated in the shared task and leveraged deep learning-based transformer models pre-trained on biomedical data to build ensemble approaches to automatically extract drug-protein relation from biomedical literature. On the main corpora of 10,750 abstracts, our best system obtained an F1-score of 77.60% (ranked 4th among 30 participating teams), and on the large-scale corpus of 2.4M documents, our system achieved micro-averaged F1-score of 77.32% (ranked 2nd among 9 system submissions). This demonstrates the effectiveness of domain-specific transformer models and ensemble approaches for automatic relation extraction from biomedical literature.

Few-Shot Relation Extraction With Dual Graph Neural Network Interaction.

Recent advances in relation extraction with deep neural architectures have achieved excellent performance. However, current models still suffer from two main drawbacks: 1) they require enormous volumes of training data to avoid model overfitting and 2) there is a sharp decrease in performance when the data distribution during training and testing shift from one domain to the other. It is thus vital to reduce the data requirement in training and explicitly model the distribution difference when transferring knowledge from one domain to another. In this work, we concentrate on few-shot relation extraction under domain adaptation settings. Specifically, we propose, a novel graph neural network (GNN) based approach for few-shot relation extraction. leverages an edge-labeling dual graph (i.e. an instance graph and a distribution graph) to explicitly model the intraclass similarity and interclass dissimilarity in each individual graph, as well as the instance-level and distribution-level relations across graphs. A dual graph interaction mechanism is proposed to adequately fuse the information between the two graphs in a cyclic flow manner. We extensively evaluate on FewRel1.0 and FewRel2.0 benchmarks under four few-shot configurations. The experimental results demonstrate that can match or outperform previously published approaches. We also perform experiments to further investigate the parameter settings and architectural choices, and we offer a qualitative analysis.

Relation-driven Query of Multiple Time Series.

Querying time series based on their relations is a crucial part of multiple time series analysis. By retrieving and understanding time series relations, analysts can easily detect anomalies and validate hypotheses in complex time series datasets. However, current relation extraction approaches, including knowledge- and data-driven ones, tend to be laborious and do not support heterogeneous relations. By conducting a formative study with 11 experts, we concluded six time series relations, including correlation, causality, similarity, lag, arithmetic, and meta, and summarized three pain points in querying time series involving these relations. We proposed RelaQ, an interactive system that supports the time series query via relation specifications. RelaQ allows users to intuitively specify heterogeneous relations when querying multiple time series, understand the query results based on a scalable, multi-level visualization, and explore possible relations beyond the existing queries. RelaQ is evaluated with two cases and a user study with 12 participants, showing promising effectiveness and usability.

Mathematical formula detection in document images: A new dataset and a new approach

Most of existing mathematical formula detectors focus on detecting formula entities through object detection or instance segmentation techniques. However, these methods often fail to convey complete messages due to the absence of the contextual and layout information of mathematical formulas. For a more comprehensive understanding of mathematical formulas in document images, it is preferable to detect logical formula blocks that include one or multiple formula entities arranged in their natural reading order. These logical formula blocks enable the transmission of complete contextual messages of mathematical formulas and aid in the reconstruction of layout information of the document images, resulting in a more accurate mathematical formula detection. In this paper, we present a novel perspective on mathematical formula detection by framing it as a joint task of formula entity detection and formula relation extraction for identifying logical formula blocks. To this end, we introduce a new, large-scale dataset, called ArxivFormula, that includes well-annotated formula entity bounding boxes and formula relationships. We also propose a new approach, called FormulaDet, to address these two sub-tasks simultaneously. FormulaDet first employs a dynamic convolution-based formula entity detector, named DynFormula, to detect formula entities. It then uses a multi-modal transformer-based relation extraction method, named RelFormer, to group these detected formula entities into logical formula blocks. Extensive experiments on standard benchmarks in this field and the proposed dataset demonstrate that our FormulaDet can achieve significantly improved performance on formula entity detection and formula relation extraction compared to previous state-of-the-art methods. The joint detection and relation extraction approach provides a more thorough understanding of mathematical formulas in document images and effectively supports downstream tasks such as document layout analysis and scientific document digitization. The ArxivFormula dataset is publicly available at https://github.com/microsoft/ArxivFormula.

Transformer-based approach to variable typing

The upsurge of multifarious endeavors across scientific fields propelled Big Data in the scientific domain. Despite the advancements in management systems, researchers find that mathematical knowledge remains one of the most challenging to manage due to the latter's inherent heterogeneity. One novel recourse being explored is variable typing where current works remain preliminary and, thus, provide a wide room for contribution. In this study, a primordial attempt to implement the end-to-end Entity Recognition (ER) and Relation Extraction (RE) approach to variable typing was made using the BERT (Bidirectional Encoder Representations from Transformers) model. A micro-dataset was developed for this process. According to our findings, the ER model and RE model, respectively, have Precision of 0.8142 and 0.4919, Recall of 0.7816 and 0.6030, and F1-Scores of 0.7975 and 0.5418. Despite the limited dataset, the models performed at par with values in the literature. This work also discusses the factors affecting this BERT-based approach, giving rise to suggestions for future implementations.

Extracting Reproductive Condition and Habitat Information from Text Using a Transformer-based Information Extraction Pipeline

Understanding the biology underpinning the natural regeneration of plant species in order to make plans for effective reforestation is a complex task. This can be aided by providing access to databases that contain long-term and wide-scale geographical information on species distribution, habitat, and reproduction. Although there exists widely-used biodiversity databases that contain structured information on species and their occurrences, such as the Global Biodiversity Information Facility (GBIF) and the Atlas of Living Australia (ALA), the bulk of knowledge about biodiversity still remains embedded in textual documents. Unstructured information can be made more accessible and useful for large-scale studies if there are tools and services that automatically extract meaningful information from text and store it in structured formats, e.g., open biodiversity databases, ready to be consumed for analysis (Thessen et al. 2022). We aim to enrich biodiversity occurrence databases with information on species reproductive condition and habitat, derived from text. In previous work, we developed unsupervised approaches to extract related habitats and their locations, and related reproductive condition and temporal expressions (Gabud and Batista-Navarro 2018). We built a new unsupervised hybrid approach for relation extraction (RE), which is a combination of classical rule-based pattern-matching methods and transformer-based language models that framed our RE task as a natural language inference (NLI) task. Using our hybrid approach for RE, we were able to extract related biodiversity entities from text even without a large training dataset. In this work, we implement an information extraction (IE) pipeline comprised of a named entity recognition (NER) tool and our hybrid relation extraction (RE) tool. The NER tool is a transformer-based language model that was pretrained on scientific text and then fine-tuned using COPIOUS (Conserving Philippine Biodiversity by Understanding big data; Nguyen et al. 2019), a gold standard corpus containing named entities relevant to species occurrence. We applied the NER tool to automatically annotate geographical location, temporal expression and habitat information contained within sentences. A dictionary-based approach is then used to identify mentions of reproductive conditions in text (e.g., phrases such as "fruited heavily" and "mass flowering"). We then use our hybrid RE tool to extract reproductive condition - temporal expression and habitat - geographical location entity pairs. We test our IE pipeline on the forestry compendium available in the CABI Digital Library (Centre for Agricultural and Biosciences International), and show that our work enables the enrichment of descriptive information on reproductive and habitat conditions of species. This work is a step towards enhancing a biodiversity database with the inclusion of habitat and reproductive condition information extracted from text.

Automated tabulation of clinical trial results: A joint entity and relation extraction approach with transformer-based language representations

Evidence-based medicine, the practice in which healthcare professionals refer to the best available evidence when making decisions, forms the foundation of modern healthcare. However, it relies on labour-intensive systematic reviews, where domain specialists must aggregate and extract information from thousands of publications, primarily of randomised controlled trial (RCT) results, into evidence tables. This paper investigates automating evidence table generation by decomposing the problem across two language processing tasks: named entity recognition, which identifies key entities within text, such as drug names, and relation extraction, which maps their relationships for separating them into ordered tuples. We focus on the automatic tabulation of sentences from published RCT abstracts that report the results of the study outcomes. Two deep neural net models were developed as part of a joint extraction pipeline, using the principles of transfer learning and transformer-based language representations. To train and test these models, a new gold-standard corpus was developed, comprising over 550 result sentences from six disease areas. This approach demonstrated significant advantages, with our system performing well across multiple natural language processing tasks and disease areas, as well as in generalising to disease domains unseen during training. Furthermore, we show these results were achievable through training our models on as few as 170 example sentences. The final system is a proof of concept that the generation of evidence tables can be semi-automated, representing a step towards fully automating systematic reviews.

Exploiting Functional Discourse Grammar to Enhance Complex Arabic Relation Extraction using a Hybrid Semantic Knowledge Base - Machine Learning Approach

Relation extraction from unstructured Arabic text is especially challenging due to the Arabic language complex morphology and the variation in word semantics and lexical categories. The research documented in this paper presents a hybrid Semantic Knowledge base - Machine Learning (SKML) approach for extracting complex Arabic relations from unstructured Arabic documents; the proposed approach exploits the principles of Functional Discourse Grammar (FDG) to emphasise the semantic and pragmatic properties of the language and facilitate the identification of relation elements. At the initial phase, the novel FDG-SKML relation extraction approach deploys a lexical-based mechanism that utilises a purposely built domain-specific Semantic Knowledge to encode the semantic association between the identified relations’ elements. The evaluation of the initial stage evidenced improved accuracy for extracting most complex Arabic relations. The initial relation extraction mechanism was further extended by integrating its output into a Machine Learning classifier that facilitated extracting especially complex relations with significant disparity in the relation elements’ presence, order, and correlation. Using Economics as the problem domain, experimental evaluation evidenced the high accuracy of our FDG-SKML approach in complex Arabic relation extraction task and demonstrated its further improvement upon integration with machine learning classifiers.

Few-Shot Relation Extraction via the Entity Feature Enhancement and Attention-Based Prototypical Network

In order to overcome the problems that the feature representation and classification effect of the existing methods need to be improved in complex contexts, this paper presents a novel few-shot relation extraction approach via the entity feature enhancement and attention-based prototypical network. The proposed model uses the pretrained RoBERTa model as the encoder while using the BiLSTM module for directional feature extraction. We further incorporate the entity feature enhancement module to improve the feature representation ability of the model. At last, the attention-based prototypical network is used to predict relations. The experimental results show that the proposed method not only outperforms the baseline models on the datasets from the bridge inspection and health domains but also achieves competitive results on the FewRel dataset in the general domain.

An Easy Partition Approach for Joint Entity and Relation Extraction

The triplet extraction (TE) task aims to identify the entities and relations mentioned in a given text. TE consists of two tasks: named entity recognition (NER) and relation classification (RC). Previous work has either treated TE as two separate tasks with independent encoders, or as a single task with a unified encoder. However, both approaches have limitations in capturing the interaction and independence of the features for different subtasks. In this paper, we propose a simple and direct feature selection and interaction scheme. Specifically, we use a pretraining language model (e.g., BERT) to extract various features, including entity recognition, shared, and relation classification features. To capture the interaction, shared features consist of the common semantic information used by the two tasks simultaneously. We use a gate module to obtain the task-specific features. Experimental results on various public benchmarks show that our proposed method can achieve competitive performance, and the calculation speed of our model is seven times faster than CasRel, and two times faster than PFN.

Multimodal learning on graphs for disease relation extraction

Disease knowledge graphs have emerged as a powerful tool for artificial intelligence to connect, organize, and access diverse information about diseases. Relations between disease concepts are often distributed across multiple datasets, including unstructured plain text datasets and incomplete disease knowledge graphs. Extracting disease relations from multimodal data sources is thus crucial for constructing accurate and comprehensive disease knowledge graphs. We introduce REMAP, a multimodal approach for disease relation extraction. The REMAP machine learning approach jointly embeds a partial, incomplete knowledge graph and a medical language dataset into a compact latent vector space, aligning the multimodal embeddings for optimal disease relation extraction. Additionally, REMAP utilizes a decoupled model structure to enable inference in single-modal data, which can be applied under missing modality scenarios. We apply the REMAP approach to a disease knowledge graph with 96,913 relations and a text dataset of 1.24 million sentences. On a dataset annotated by human experts, REMAP improves language-based disease relation extraction by 10.0% (accuracy) and 17.2% (F1-score) by fusing disease knowledge graphs with language information. Furthermore, REMAP leverages text information to recommend new relationships in the knowledge graph, outperforming graph-based methods by 8.4% (accuracy) and 10.4% (F1-score). REMAP is a flexible multimodal approach for extracting disease relations by fusing structured knowledge and language information. This approach provides a powerful model to easily find, access, and evaluate relations between disease concepts.

REEGAT: RoBERTa Entity Embedding and Graph Attention Networks Enhanced Sentence Representation for Relation Extraction

Relation extraction is one of the most important intelligent information extraction technologies, which can be used to construct and optimize services in intelligent communication systems (ICS). One issue with the existing relation extraction approaches is that they use one-sided sentence embedding as their final prediction vector, which degrades relation extraction performance. The innovative relation extraction model REEGAT (RoBERTa Entity Embedding and Graph Attention networks enhanced sentence representation) that we present in this paper, incorporates the concept of enhanced word embedding from graph neural networks. The model first uses RoBERTa to obtain word embedding and PyTorch embedding to obtain relation embedding. Then, the multi-headed attention mechanism in GAT (graph attention network) is introduced to weight the word embedding and relation embedding to enrich further the meaning conveyed by the word embedding. Finally, the entity embedding component is used to obtain sentence representation by pooling the word embedding from GAT and the entity embedding from named entity recognition. The weighted and pooled word embedding contains more relational information to alleviate the one-sided problem of sentence representation. The experimental findings demonstrate that our model outperforms other standard methods.

A Concise Relation Extraction Method Based on the Fusion of Sequential and Structural Features Using ERNIE

Relation extraction, a fundamental task in natural language processing, aims to extract entity triples from unstructured data. These triples can then be used to build a knowledge graph. Recently, pre-training models that have learned prior semantic and syntactic knowledge, such as BERT and ERNIE, have enhanced the performance of relation extraction tasks. However, previous research has mainly focused on sequential or structural data alone, such as the shortest dependency path, ignoring the fact that fusing sequential and structural features may improve the classification performance. This study proposes a concise approach using the fused features for the relation extraction task. Firstly, for the sequential data, we verify in detail which of the generated representations can effectively improve the performance. Secondly, inspired by the pre-training task of next-sentence prediction, we propose a concise relation extraction approach based on the fusion of sequential and structural features using the pre-training model ERNIE. The experiments were conducted on the SemEval 2010 Task 8 dataset and the results show that the proposed method can improve the F1 value to 0.902.

It Takes Two Flints to Make a Fire: Multitask Learning of Neural Relation and Explanation Classifiers

AbstractWe propose an explainable approach for relation extraction that mitigates the tension between generalization and explainability by jointly training for the two goals. Our approach uses a multi-task learning architecture, which jointly trains a classifier for relation extraction, and a sequence model that labels words in the context of the relations that explain the decisions of the relation classifier. We also convert the model outputs to rules to bring global explanations to this approach. This sequence model is trained using a hybrid strategy: supervised, when supervision from pre-existing patterns is available, and semi-supervised otherwise. In the latter situation, we treat the sequence model’s labels as latent variables, and learn the best assignment that maximizes the performance of the relation classifier. We evaluate the proposed approach on the two datasets and show that the sequence model provides labels that serve as accurate explanations for the relation classifier’s decisions, and, importantly, that the joint training generally improves the performance of the relation classifier. We also evaluate the performance of the generated rules and show that the new rules are a great add-on to the manual rules and bring the rule-based system much closer to the neural models.

Span-based single-stage joint entity-relation extraction model.

Extracting entities and relations from the unstructured text has attracted increasing attention in recent years. The existing work has achieved considerable results, yet it is difficult to solve entity overlap and exposure bias. To address cascading errors, exposure bias, and entity overlap in existing entity relation extraction approaches, we propose a joint entity relation extraction model (SMHS) based on a span-level multi-head selection mechanism, transforming entity relation extraction into a span-level multi-head selection problem. Our model uses span-tagger and span-embedding to construct span semantic vectors, utilizes LSTM and multi-head self-attention mechanism for span feature extraction, multi-head selection mechanism for span-level relation decoding, and introduces span classification task for multi-task learning to decode out the relation triad in a single-stage. Experiments on the classic English dataset NYT and the publicly available Chinese relationship extraction dataset DuIE 2.0 show that this method achieves better results than the baseline method, which verifies the effectiveness of this method. Source code and data are published here(https://github.com/Beno-waxgourd/NLP.git).

Road-Related Information Mining from Social Media Data: A Joint Relation Extraction and Entity Recognition Approach

Social media data have been gradually regarded as a prospective social sensor in the transportation domain for capturing road conditions. Most existing social media data-based sensors (SMDbSs) of road conditions, however, rely heavily on lexicon-based methods for information extraction and provide coarse-grained location information. Hence, this work newly devises an SMDbS based on joint relation extraction and entity recognition for sensing road conditions from social media data, which eliminates the reliance on lexicon-based methods and offers finer-grained location information in comparison with existing SMDbSs. This SMDbS development consists of four major steps, including data collection and annotation, data cleansing, two-stage information extraction, and model verification. A tweet dataset in Lexington city is exploited to demonstrate this SMDbS, which shows satisfactory information extraction performance. This study would help facilitate social media data to be an extra information source in the transportation domain.

Exploiting global context and external knowledge for distantly supervised relation extraction

Distantly supervised relation extraction aims to obtain relational facts from unstructured texts. Although distant supervision can automatically generate labeled training instances, it inevitably suffers from the wrong-label problem. Most of the current work is based on the bag-level for solving the noise problem, where a bag is composed of multiple sentences containing mentions of the same entity pair. However, previous studies mostly represent sentences from a single perspective, wherein insufficient modeling of global information restricts the effectiveness of denoising. In this study, we propose a novel distantly supervised relation extraction approach that incorporates the global contextual information of sentences to guide the denoising process and generate an effective bag-level representation. Simultaneously, knowledge-aware word embeddings were generated to enrich sentence-level representations by introducing both structured knowledge from external knowledge graphs and semantic knowledge from the corpus. The experimental results demonstrate that our proposed approach outperforms state-of-the-art methods on both versions of the large-scale benchmark New York Times dataset. In addition, the differences between the two versions of the dataset were investigated through further comparative experiments.

Translational relation embeddings for multi-hop knowledge base question answering

Multi-hop Knowledge Base Question Answering (KBQA) aims to predict answers that require multi-hop reasoning from the topic entity in the question over the Knowledge Base (KB). Relation extraction is a core step in KBQA, which extracts the relation path from the topic entity to the answer entity. Compared with single-hop questions, multi-hop ones have more complex syntactic structures to understand, and multi-hop relation paths lead to a larger search space, which makes it much more challenging to extract the correct relation paths. To tackle the above challenges, most existing relation extraction approaches focus on the semantic similarity between questions and relation paths. However, those approaches only consider the word semantics of the relation names but ignore the graph semantics inside the knowledge base. As a result, their generalization ability relying on the naming rules of the relations, making it more difficult to generalize over large knowledge bases.To address the current limitations and take advantage of the graph semantics of relations, we propose a novel translational embedding-based relation extractor that utilizes pretrained embeddings from TransE. In particular, we treat the multi-hop relation path as a translation from the first relation to the last one in the semantic space of TransE. Then we map the question into this space under the supervision of the path embeddings. To take full advantage of the pretrained graph semantics in TransE, we propose a KBQA framework that leverages pretrained relation semantics in relation extraction and pretrained entity semantics in answer selection. Our approach achieves state-of-the-art performance on two benchmark datasets, WebQuestionSP and MetaQA, demonstrating its effectiveness on the multi-hop KBQA task.

A pattern-first pipeline approach for entity and relation extraction

Entity-relation extraction is the task of extracting entities and their semantic relations from a piece of unstructured text. In recent studies, Machine Reading Comprehension (MRC) based methods have been applied to this task and achieved significant results. As a pipelined approach, these methods always extract head entities first, and then identify related tail entities by enumerating each relationship. These entity-first methods will lead to the entity redundancy problem. They also suffer from the error propagation issue, which is an inherent issue of the multi-step inference process. Moreover, most existing MRC-based models, which use tagging-based methods for entity recognition, could not deal with overlapping entities. To address these, we propose Patti, a Pattern-First Pipeline Approach for Entity and Relation Extraction. Firstly, Patti leverages a novel MRC-based pattern classifier to identify relation patterns. Next, a span-based method was introduced to extract entities under the guidance of questions parameterized by the patterns yield in the first step. Finally, to alleviate the error propagation issue, Patti employs an additional MRC-based classifier to remove falsely extracted candidate entity-relation triples. Experiment results show that our approach significantly outperforms the entity-first baseline models on CoNLL04 and ACE05 datasets.