Distant Supervised Relation Extraction Research Articles

Read, Eliminate, and Focus: A reading comprehension paradigm for distant supervised relation extraction

Read, Eliminate, and Focus: A reading comprehension paradigm for distant supervised relation extraction

Hierarchical symmetric cross entropy for distant supervised relation extraction

Hierarchical symmetric cross entropy for distant supervised relation extraction

Interaction-and-Response Network for Distantly Supervised Relation Extraction.

Distantly supervised relation extraction (DSRE) aims to identify semantic relations from massive plain texts. A broad range of the prior research has leveraged a series of selective attention mechanisms over sentences in a bag to extract relation features without considering dependencies among the relation features. As a result, potential discriminative information existed in the dependencies is ignored, causing a decline in the performance of extracting entity relations. In this article, we focus on going beyond the selective attention mechanisms and propose a new framework termed interaction-and-response network (IR-Net) that adaptively recalibrates the features of sentence, bag, and group levels by explicitly modeling interdependencies among the features on each level. The IR-Net consists of a series of interactive and responsive modules throughout feature hierarchy, seeking to strengthen its power of learning salient discriminative features for distinguishing entity relations. We conduct extensive experiments on three benchmark DSRE datasets, including NYT-10, NYT-16, and Wiki-20m. The experimental results demonstrate that the IR-Net brings obvious improvements in performance when comparing ten state-of-the-art DSRE methods for entity relation extraction.

MKDAT: Multi-Level Knowledge Distillation with Adaptive Temperature for Distantly Supervised Relation Extraction

Distantly supervised relation extraction (DSRE), first used to address the limitations of manually annotated data via automatically annotating the data with triplet facts, is prone to issues such as mislabeled annotations due to the interference of noisy annotations. To address the interference of noisy annotations, we leveraged a novel knowledge distillation (KD) method which was different from the conventional models on DSRE. More specifically, we proposed a model-agnostic KD method, Multi-Level Knowledge Distillation with Adaptive Temperature (MKDAT), which mainly involves two modules: Adaptive Temperature Regulation (ATR) and Multi-Level Knowledge Distilling (MKD). ATR allocates adaptive entropy-based distillation temperatures to different training instances for providing a moderate softening supervision to the student, in which label hardening is possible for instances with great entropy. MKD combines the bag-level and instance-level knowledge of the teacher as supervisions of the student, and trains the teacher and student at the bag and instance levels, respectively, which aims at mitigating the effects of noisy annotation and improving the sentence-level prediction performance. In addition, we implemented three MKDAT models based on the CNN, PCNN, and ATT-BiLSTM neural networks, respectively, and the experimental results show that our distillation models outperform the baseline models on bag-level and instance-level evaluations.

Distantly Supervised Relation Extraction via Contextual Information Interaction and Relation Embeddings

Distantly supervised relation extraction (DSRE) utilizes an external knowledge base to automatically label a corpus, which inevitably leads to the problem of mislabeling. Existing approaches utilize BERT to provide instances and relation embeddings to capture a wide set of relations and address the noise problem. However, the method suffers from a single method of textual information processing, underutilizing the feature information of entity pairs in the relation embeddings part and being interfered with by noisy labels when classifying multiple labels. For this reason, we propose the contextual information interaction and relation embeddings (CIRE) method. First, we utilize BERT and Bi-LSTM to construct a neural network model to enhance contextual information interaction by filtering and supplementing sequence information through the error repair capability of the Bi-LSTM gating mechanism. At the same time, we combine the vector difference between entity pairs and entity pairs in the relation embeddings layer to improve the relation embeddings accuracy. Finally, we choose sparse softmax as the classifier, which improves the ability to control the noise categories by controlling the number of output categories. The experimental results show that our method significantly outperforms the baseline method and improves the AUC metric by 2.6% on the NYT2010 dataset.

Semantic piecewise convolutional neural network with adaptive negative training for distantly supervised relation extraction

Distantly Supervised Relation Extraction (DSRE) aligns existing knowledge bases with unstructured text to extract relation facts, and its automatically generated training data is inevitably noisy. Most existing works identify and reduce the impact of noise by enhancing semantic features. However, they only consider the semantic information in a single instance and ignore the semantic information between different instances. In this work, we propose a Semantic Piecewise Convolutional Neural Network (SPCNN), which uses the similarity between different entity pairs as semantic information to improve relation extraction. Specifically, to learn better semantic vector representations, we combine position features with entity pair features and entity similarity features in a high-dimensional space respectively, and generating two different semantic-aware representations. Then we unify these two representations to form a high-quality bag representation for training. Moreover, we design an Adaptive Negative Training (ANT) strategy, which facilitates the network to further exploit the rich semantic features to reduce the interference of noisy labels. Extensive experimental results on a large-scale benchmark dataset show that our method significantly outperforms other baselines.

Learning Relation Ties with a Force-Directed Graph in Distant Supervised Relation Extraction

Relation ties, defined as the correlation and mutual exclusion between different relations, are critical for distant supervised relation extraction. Previous studies usually obtain this property by greedily learning the local connections between relations. However, they are essentially limited because of failing to capture the global topology structure of relation ties and may easily fall into a locally optimal solution. To address this issue, we propose a novel force-directed graph to comprehensively learn relation ties. Specifically, we first construct a graph according to the global co-occurrence of all relations. Then, we borrow the idea of Coulomb’s law from physics and introduce the concept of attractive force and repulsive force into this graph to learn correlation and mutual exclusion between relations. Finally, the obtained relation representations are applied as an inter-dependent relation classifier. Extensive experimental results demonstrate that our method is capable of modeling global correlation and mutual exclusion between relations, and outperforms the state-of-the-art baselines. In addition, the proposed force-directed graph can be used as a module to augment existing relation extraction systems and improve their performance.

Retracted: Utilizing Entity-Based Gated Convolution and Multilevel Sentence Attention to Improve Distantly Supervised Relation Extraction.

[This retracts the article DOI: 10.1155/2021/6110885.].

Distant supervised relation extraction based on residual attention

Distant supervised relation extraction based on residual attention

Distantly Supervised Relation Extraction using Global Hierarchy Embeddings and Local Probability Constraints

To find relational facts of interest from plain texts, distantly supervised relation extraction (DSRE) has drawn significant attention. Recent works exploit relation hierarchies to mine more clues for long-tail relations and achieve good performance. However, they ignore or underutilize the correlation of relations in the hierarchical structure. Empirically, the correlation facilitates knowledge transfer between different relations to further handle long-tail relations and improves inter-relational discrimination. In this paper, we devise an end-to-end network to model the correlation of relations from two perspectives. Globally, we construct an undirected connected graph according to the relation hierarchies, and employ Graph Attention Networks (GATs) to aggregate node information and generate correlation-aware Global Hierarchy Embeddings (GHE). Locally, we assume that along the relation hierarchies, the classification results of adjacent levels should be highly interdependent, and introduce a constraint called Local Probability Constraints (LPC) to take it into account. LPC is then combined with a branch network for both sentence-level and bag-level classification. Experimental results on the popular New York Times (NYT) dataset show that, our model GHE-LPC outperforms other state-of-the-art baselines in terms of AUC, Top-N precision, accuracy of Hits@K, etc.

A pattern-aware self-attention network for distant supervised relation extraction

Distant supervised relation extraction is an efficient strategy of finding relational facts from unstructured text without labeled training data. A recent paradigm to develop relation extractors is using pre-trained Transformer language models to produce high-quality sentence representations. However, due to the original Transformer is weak at capturing local dependencies and phrasal structures, existing Transformer-based methods cannot identify various relational patterns in sentences. To address this issue, we propose a novel distant supervised relation extraction model, which employs a specific-designed pattern-aware self-attention network to automatically discover relational patterns for pre-trained Transformers in an end-to-end manner. Specifically, the proposed method assumes that the correlation between two adjacent tokens reflects the probability that they belong to the same pattern. Based on this assumption, a novel self-attention network is designed to generate the probability distribution of all patterns in a sentence. Then, the probability distribution is applied as a constraint in the first Transformer layer to encourage its attention heads to follow the relational pattern structures. As a result, fine-grained pattern information is enhanced in the pre-trained Transformer without losing global dependencies. Extensive experimental results on two popular benchmark datasets demonstrate that our model performs better than the state-of-the-art baselines.

Multi-Graph Cooperative Learning Towards Distant Supervised Relation Extraction

The Graph Convolutional Network (GCN) is a universal relation extraction method that can predict relations of entity pairs by capturing sentences’ syntactic features. However, existing GCN methods often use dependency parsing to generate graph matrices and learn syntactic features. The quality of the dependency parsing will directly affect the accuracy of the graph matrix and change the whole GCN’s performance. Because of the influence of noisy words and sentence length in the distant supervised dataset, using dependency parsing on sentences causes errors and leads to unreliable information. Therefore, it is difficult to obtain credible graph matrices and relational features for some special sentences. In this article, we present a Multi-Graph Cooperative Learning model (MGCL), which focuses on extracting the reliable syntactic features of relations by different graphs and harnessing them to improve the representations of sentences. We conduct experiments on a widely used real-world dataset, and the experimental results show that our model achieves the state-of-the-art performance of relation extraction.

Improving Distant Supervised Relation Extraction using shortest dependency path

Distant supervised relation extraction is the problem of classifying the semantic relation between a pair of entities in a given sentence, where data is created by aligning unlabelled text with knowledge base automatically. Prior researches indicated that most of the sentences in the distant supervised relation extraction setting are very long and benefit from the attention over words. We note that the shortest dependency path between the entity pair in the syntax analysis tree of a sentence can help word based attention. In this paper, we propose a noval distant supervised neural relation extraction method, which makes use of the shortest dependency path to supervise the learning of word attention. Through extensive experiments on benchmark datasets, we demonstrate effectiveness of our model.

Distant supervised relation extraction with position feature attention and selective bag attention

Distant supervision greatly reduces manual consumption by automatically labeling data. The relation extraction methods under distant supervision divide sentences with the same entity pair into a bag, and perform training and testing on these sentence bags. The existing distant supervised relation extraction methods ignore two facts. First, there are many sentences where the target entity pairs appear multiple times. Second, the noise between sentence bags is different, and the sentences of some bags are even all mislabeled. To solve these two problems, we propose a novel relation extraction method with position feature attention and selective bag attention. The position feature attention is employed to obtain the weighted sentence representation with different position features by calculating all position combinations of the target entity pair. A bag with large noise and a bag with small noise are selected through the selective bag attention mechanism to form a bag pair, and training is performed at the level of the bag pair, which denoises at the bag level and at the same time balances the noise between different bag pairs. The experimental results show that our method is effective and outperforms several competitive baseline methods.

Empower Distantly Supervised Relation Extraction with Collaborative Adversarial Training

With recent advances in distantly supervised (DS) relation extraction (RE), considerable attention is attracted to leverage multi-instance learning (MIL) to distill high-quality supervision from the noisy DS. Here, we go beyond label noise and identify the key bottleneck of DS-MIL to be its low data utilization: as high-quality supervision being refined by MIL, MIL abandons a large amount of training instances, which leads to a low data utilization and hinders model training from having abundant supervision. In this paper, we propose collaborative adversarial training to improve the data utilization, which coordinates virtual adversarial training (VAT) and adversarial training (AT) at different levels. Specifically, since VAT is label-free, we employ the instance-level VAT to recycle instances abandoned by MIL. Besides, we deploy AT at the bag-level to unleash the full potential of the high-quality supervision got by MIL. Our proposed method brings consistent improvements (∼ 5 absolute AUC score) to the previous state of the art, which verifies the importance of the data utilization issue and the effectiveness of our method.

A noisy label and negative sample robust loss function for DNN-based distant supervised relation extraction

As a major method for relation extraction, distantly supervised relation extraction (DSRE) suffered from the noisy label problem and class imbalance problem (these two problems are also common for many other NLP tasks, e.g., text classification). However, there seems no existing research in DSRE or other NLP tasks that can simultaneously solve both problems, which is a significant insufficiency in related researches. In this paper, we propose a loss function which is robust to noisy label and efficient for the imbalanced class dataset. More specific, first we quantify the negative impacts of the noisy label and class imbalance problems. And then we construct a loss function that can minimize these negative impacts through a linear programming method. As far as we know, this seems to be the first attempt to address the noisy label problem and class imbalance problem simultaneously. We evaluated the constructed loss function on the distantly labeled dataset, our artificially noised dataset, human-annotated dataset of Docred, as well as the artificially noised dataset of CoNLL 2003. Experimental results indicate that a DNN model adopting the constructed loss function can outperform other models that adopt the state-of-the-art noisy label robust or negative sample robust loss functions.

Improving Distant Supervised Relation Extraction with Noise Detection Strategy

Distant supervised relation extraction (DSRE) is widely used to extract novel relational facts from plain text, so as to improve the knowledge graph. However, distant supervision inevitably suffers from the noisy labeling problem that will severely damage the performance of relation extraction. Currently, most DSRE methods are mainly focused on reducing the weights of noisy sentences, ignoring the bag-level noise where all sentences in a bag are wrongly labeled. In this paper, we present a novel noise detection-based relation extraction approach (NDRE) to automatically detect noisy labels with entity information and dynamically correct them, which can alleviate both instance-level and bag-level noisy problems. By this means, we can extend the dataset from the Web tables without introducing more noise. In this approach, to embed the semantics of sentences from corpus and web tables, we firstly propose a powerful sentence coder that employs an internal multi-head self-attention mechanism between the piecewise max-pooling convolutional neural network. Second, we adopt a noise detection strategy, which is expected to dynamically detect and correct the original noisy label according to the similarity between sentence representation and entity-aware embeddings. Then, we aggregate the information from corpus and web tables to make the final relation prediction. Experimental results on a public benchmark dataset demonstrate that our proposed approach achieves significant improvements over the state-of-the-art baselines and can effectively reduce the noisy labeling problem.

ESRE: handling repeated entities in distant supervised relation extraction

Distant supervised relation extraction has been widely used to find novel relational facts from unstructured text. As far as we know, nearly all existing relation extraction models assume that each sentence contains precisely one entity pair, i.e., two entities. However, in reality, the datasets constructed by distant supervision have lots of sentences which contain repeated entities. In other words, there may be more than two entities in a sentence. This phenomenon breaks the assumption of existing models and makes them inevitably encounter the attention bias problem; that is, the model focuses on the wrong entities during relation extraction. To alleviate this problem, in this paper, we utilize the idea of ensemble learning and propose a novel distant supervised relation extraction model. The proposed model follows the multi-instance multi-label learning mechanism and conducts relation extraction based on the sentence-bag representations. Specifically, it first tries to identify the most critical entity and keywords, and then it uses voting mechanism to determine the sentence-level and bag-level features. Experimental results show that our proposed model outperforms the state-of-the-art baselines in relation extraction on a popular benchmark dataset, and it also indicates that the proposed model can indeed alleviate the problem caused by repeated-entity phenomenon.

Improving Distantly-Supervised Relation Extraction Through BERT-Based Label and Instance Embeddings

Distantly-supervised relation extraction (RE) is an effective method to scale RE to large corpora but suffers from noisy labels. Existing approaches try to alleviate noise through multi-instance learning and by providing additional information but manage to recognize mainly the top frequent relations, neglecting those in the long-tail. We propose REDSandT (Relation Extraction with Distant Supervision and Transformers), a novel distantly-supervised transformer-based RE method that manages to capture a wider set of relations through highly informative instance and label embeddings for RE by exploiting BERT’s pre-trained model, and the relationship between labels and entities, respectively. We guide REDSandT to focus solely on relational tokens by fine-tuning BERT on a structured input, including the sub-tree connecting an entity pair and the entities’ types. Using the extracted informative vectors, we shape label embeddings, which we also use as an attention mechanism over instances to further reduce noise. Finally, we represent sentences by concatenating relation and instance embeddings. Experiments in the two benchmark datasets for distantly-supervised RE, NYT-10 and GDS, show that REDSandT captures a broader set of relations with higher confidence, achieving a state-of-the-art AUC (0.424) in NYT-10 and an excellent AUC (0.862) in GDS.

Utilizing Entity-Based Gated Convolution and Multilevel Sentence Attention to Improve Distantly Supervised Relation Extraction.

Distant supervision is an effective method to automatically collect large-scale datasets for relation extraction (RE). Automatically constructed datasets usually comprise two types of noise: the intrasentence noise and the wrongly labeled noisy sentence. To address issues caused by the above two types of noise and improve distantly supervised relation extraction, this paper proposes a novel distantly supervised relation extraction model, which consists of an entity-based gated convolution sentence encoder and a multilevel sentence selective attention (Matt) module. Specifically, we first apply an entity-based gated convolution operation to force the sentence encoder to extract entity-pair-related features and filter out useless intrasentence noise information. Furthermore, the multilevel attention schema fuses the bag information to obtain a fine-grained bag-specific query vector, which can better identify valid sentences and reduce the influence of wrongly labeled sentences. Experimental results on a large-scale benchmark dataset show that our model can effectively reduce the influence of the above two types of noise and achieves state-of-the-art performance in relation extraction.