Neural Relation Extraction Research Articles

Piecewise convolutional neural network relation extraction with self-attention mechanism

The task of relation extraction in natural language processing is to identify the relation between two specified entities in a sentence. However, the existing model methods do not fully utilize the word feature information and pay little attention to the influence degree of the relative relation extraction results of each word. In order to address the aforementioned issues, we propose a relation extraction method based on self-attention mechanism (SPCNN-VAE) to solve the above problems. First, we use a multi-head self-attention mechanism to process word vectors and generate sentence feature vector representations, which can be used to extract semantic dependencies between words in sentences. Then, we introduce the word position to combine the sentence feature representation with the position feature representation of words to form the input representation of piecewise convolutional neural network (PCNN). Furthermore, to identify the word feature information that is most useful for relation extraction, an attention-based pooling operation is employed to capture key convolutional features and classify the feature vectors. Finally, regularization is performed by a variational autoencoder (VAE) to enhance the encoding ability of model word information features. The performance analysis is performed on SemEval 2010 task 8, and the experimental results show that the proposed relation extraction model is effective and outperforms some competitive baselines.

Relation extraction in Chinese using attention-based bidirectional long short-term memory networks.

Relation extraction is an important topic in information extraction, as it is used to create large-scale knowledge graphs for a variety of downstream applications. Its goal is to find and extract semantic links between entity pairs in natural language sentences. Deep learning has substantially advanced neural relation extraction, allowing for the autonomous learning of semantic features. We offer an effective Chinese relation extraction model that uses bidirectional LSTM (Bi-LSTM) and an attention mechanism to extract crucial semantic information from phrases without relying on domain knowledge from lexical resources or language systems in this study. The attention mechanism included into the Bi-LSTM network allows for automatic focus on key words. Two benchmark datasets were used to create and test our models: Chinese SanWen and FinRE. The experimental results show that the SanWen dataset model outperforms the FinRE dataset model, with area under the receiver operating characteristic curve values of 0.70 and 0.50, respectively. The models trained on the SanWen and FinRE datasets achieve values of 0.44 and 0.19, respectively, for the area under the precision-recall curve. In addition, the results of repeated modeling experiments indicated that our proposed method was robust and reproducible.

Neural Attentional Relation Extraction with Dual Dependency Trees

Neural Attentional Relation Extraction with Dual Dependency Trees

Knowledge graph attention mechanism for distant supervision neural relation extraction

Distant supervision neural relation extraction is used to classify the relation labels of instance bags with the same head–tail entity. Since there are usually multirelation labels in an entity pair, intensive noise occurs in the instance bag. Recent works mostly focused on developing an attention mechanism to degrade the weights of some sentences whose ground-truth relation labels are different from the instance bag. However, the general weakness is the failure to explore the semantic correlations between entity pair and its context. Additionally, the number of relation categories follows a long-tail distribution, and it is still a challenge to extract long-tail relations. Therefore, the Knowledge Graph ATTention (KGATT) mechanism is proposed to deal with the noises and long-tail problem, and it contains two modules: a fine-alignment mechanism and an inductive mechanism. In particular, the fine-alignment mechanism is used to learn the ground-truth relation of the sentence itself by aligning with all predefined relations. The inductive mechanism is employed to learn the enhanced relations from neighbors of the knowledge graph (KG) to compensate for data scarcity. With the mutual reinforcement of the two modules, our model can enrich the representation of the instance bag, which not only improves the generalization ability but also eases the long-tail phenomenon. Extensive experiments and ablation studies are conducted on the NYT-FB60K and GIDS-FB8K datasets, and the results show the KGATT is effective in improving performance. Based on Piecewise Convolution Neural Network (PCNN), our model achieves superior performances in various indicators as well as the long-tail relations.

Deep neural network-based relation extraction: an overview

Deep neural network-based relation extraction: an overview

Improving Distant Supervised Relation Extraction using shortest dependency path

Distant supervised relation extraction is the problem of classifying the semantic relation between a pair of entities in a given sentence, where data is created by aligning unlabelled text with knowledge base automatically. Prior researches indicated that most of the sentences in the distant supervised relation extraction setting are very long and benefit from the attention over words. We note that the shortest dependency path between the entity pair in the syntax analysis tree of a sentence can help word based attention. In this paper, we propose a noval distant supervised neural relation extraction method, which makes use of the shortest dependency path to supervise the learning of word attention. Through extensive experiments on benchmark datasets, we demonstrate effectiveness of our model.

Evaluating and Enhancing the Robustness of Sustainable Neural Relationship Classifiers Using Query-Efficient Black-Box Adversarial Attacks

Neural relation extraction (NRE) models are the backbone of various machine learning tasks, including knowledge base enrichment, information extraction, and document summarization. Despite the vast popularity of these models, their vulnerabilities remain unknown; this is of high concern given their growing use in security-sensitive applications such as question answering and machine translation in the aspects of sustainability. In this study, we demonstrate that NRE models are inherently vulnerable to adversarially crafted text that contains imperceptible modifications of the original but can mislead the target NRE model. Specifically, we propose a novel sustainable term frequency-inverse document frequency (TFIDF) based black-box adversarial attack to evaluate the robustness of state-of-the-art CNN, CGN, LSTM, and BERT-based models on two benchmark RE datasets. Compared with white-box adversarial attacks, black-box attacks impose further constraints on the query budget; thus, efficient black-box attacks remain an open problem. By applying TFIDF to the correctly classified sentences of each class label in the test set, the proposed query-efficient method achieves a reduction of up to 70% in the number of queries to the target model for identifying important text items. Based on these items, we design both character- and word-level perturbations to generate adversarial examples. The proposed attack successfully reduces the accuracy of six representative models from an average F1 score of 80% to below 20%. The generated adversarial examples were evaluated by humans and are considered semantically similar. Moreover, we discuss defense strategies that mitigate such attacks, and the potential countermeasures that could be deployed in order to improve sustainability of the proposed scheme.

Neural relation extraction: a review

Neural relation extraction discovers semantic relations between entities from unstructured text using deeplearning methods. In this study, we make a clear categorization of the existing relation extraction methods in termsof data expressiveness and data supervision, and present a comprehensive and comparative review. We describe theevaluation methodologies and the datasets used for model assessment. We explicitly state the common challenges inrelation extraction task and point out the potential of the pretrained models to solve them. Accordingly, we investigateadditional research directions and improvement ideas in this field.

A Noise-aware Method with Type Constraint Pattern for Neural Relation Extraction

A Noise-aware Method with Type Constraint Pattern for Neural Relation Extraction

A survey on neural relation extraction

Relation extraction is a key task for knowledge graph construction and natural language processing, which aims to extract meaningful relational information between entities from plain texts. With the development of deep learning, many neural relation extraction models were proposed recently. This paper introduces a survey on the task of neural relation extraction, including task description, widely used evaluation datasets, metrics, typical methods, challenges and recent research progresses. We mainly focus on four recent research problems: (1) how to learn the semantic representations from the given sentences for the target relation, (2) how to train a neural relation extraction model based on insufficient labeled instances, (3) how to extract relations across sentences or in a document and (4) how to jointly extract relations and corresponding entities? Finally, we give out our conclusion and future research issues.

Integrating Relation Constraints with Neural Relation Extractors

Recent years have seen rapid progress in identifying predefined relationship between entity pairs using neural networks (NNs). However, such models often make predictions for each entity pair individually, thus often fail to solve the inconsistency among different predictions, which can be characterized by discrete relation constraints. These constraints are often defined over combinations of entity-relation-entity triples, since there often lack of explicitly well-defined type and cardinality requirements for the relations. In this paper, we propose a unified framework to integrate relation constraints with NNs by introducing a new loss term, Constraint Loss. Particularly, we develop two efficient methods to capture how well the local predictions from multiple instance pairs satisfy the relation constraints. Experiments on both English and Chinese datasets show that our approach can help NNs learn from discrete relation constraints to reduce inconsistency among local predictions, and outperform popular neural relation extraction (NRE) models even enhanced with extra post-processing. Our source code and datasets will be released at https://github.com/PKUYeYuan/Constraint-Loss-AAAI-2020.

Improving Neural Relation Extraction with Positive and Unlabeled Learning

We present a novel approach to improve the performance of distant supervision relation extraction with Positive and Unlabeled (PU) Learning. This approach first applies reinforcement learning to decide whether a sentence is positive to a given relation, and then positive and unlabeled bags are constructed. In contrast to most previous studies, which mainly use selected positive instances only, we make full use of unlabeled instances and propose two new representations for positive and unlabeled bags. These two representations are then combined in an appropriate way to make bag-level prediction. Experimental results on a widely used real-world dataset demonstrate that this new approach indeed achieves significant and consistent improvements as compared to several competitive baselines.

Distilling Knowledge from Well-Informed Soft Labels for Neural Relation Extraction

Extracting relations from plain text is an important task with wide application. Most existing methods formulate it as a supervised problem and utilize one-hot hard labels as the sole target in training, neglecting the rich semantic information among relations. In this paper, we aim to explore the supervision with soft labels in relation extraction, which makes it possible to integrate prior knowledge. Specifically, a bipartite graph is first devised to discover type constraints between entities and relations based on the entire corpus. Then, we combine such type constraints with neural networks to achieve a knowledgeable model. Furthermore, this model is regarded as teacher to generate well-informed soft labels and guide the optimization of a student network via knowledge distillation. Besides, a multi-aspect attention mechanism is introduced to help student mine latent information from text. In this way, the enhanced student inherits the dark knowledge (e.g., type constraints and relevance among relations) from teacher, and directly serves the testing scenarios without any extra constraints. We conduct extensive experiments on the TACRED and SemEval datasets, the experimental results justify the effectiveness of our approach.

A Neural Relation Extraction Model for Distant Supervision in Counter-Terrorism Scenario

Natural language processing (NLP) is the best solution to extensive, unstructured, complex, and diverse network big data for counter-terrorism. Through the text analysis, it is the basis and the most critical step to quickly extract the relationship between the relevant entities pairs in terrorism. Relation extraction lays a foundation for constructing a knowledge graph (KG) of terrorism and provides technical support for intelligence analysis and prediction. This paper takes the distant-supervised relation extraction as the starting point, breaks the limitation of artificial data annotation. Combining the Bidirectional Encoder Representation from Transformers (BERT) pre-training model and the sentence-level attention over multiple instances, we proposed the relation extraction model named BERT-att. Experiments show that our model is more efficient and better than the current leading baseline model over each evaluative metrics. Our model applied to the construction of anti-terrorism knowledge map, it used in regional security risk assessment, terrorist event prediction and other scenarios.

Syntax-aware entity representations for neural relation extraction

Distantly supervised relation extraction has been widely used to find novel relational facts between entities from text, and can be easily scaled to very large corpora. Previous studies on neural relation extraction treat this task as a multi-instance learning problem, and encode the sentences in low-dimensional spaces via neural networks. Although great progress has been made, they seldom consider the information represented by entities, which are of great significance to relation extraction. In this article, we propose several methods based on different tree-based models to learn syntax-aware entity representations for neural relation extraction. First, we encode the context of entities on dependency trees as sentence-level entity embedding based on tree-structured neural network models. Then, we utilize inter-sentence attention mechanism to obtain sentence bag level entity embedding over all sentences containing the specified entity pair. Finally, we combine both sentence embedding and entity embedding for relation classification. Experimental results on a widely used real-world dataset indicate that our system performs better than the state-of-the-art systems of relation extraction.

Multi-task and multi-view training for end-to-end relation extraction

Transfer learning has shown promising results for transferring knowledge ofsource tasks to target tasks in natural language processing (NLP). In this paper, we investigate a multi-task and multi-view learning (MTMVL) framework for end-to-end neural relation extraction, using large noisy data as one auxiliary task for improving manually labeled data, and a dependency parsing objective as a second auxiliary task for leveraging syntax. Two views are taken for relation extraction, consisting of a joint tagging view and a novel context relation view. To capture sentence multi-level information, we explore a weighted average approach to represent shared deep bi-directional recurrent neural networks and encourage auxiliary tasks to accommodate relation extraction task via a time-related scheduled sampling strategy. We evaluate MTMVL framework on manual-labeled ACE2005 dataset, and experimental results show that our model outperforms the state-of-the-art methods, which indicates the effectiveness of multi-auxiliary information for knowledge transferring.

Neural Relation Extraction within and across Sentence Boundaries

Past work in relation extraction mostly focuses on binary relation between entity pairs within single sentence. Recently, the NLP community has gained interest in relation extraction in entity pairs spanning multiple sentences. In this paper, we propose a novel architecture for this task: inter-sentential dependency-based neural networks (iDepNN). iDepNN models the shortest and augmented dependency paths via recurrent and recursive neural networks to extract relationships within (intra-) and across (inter-) sentence boundaries. Compared to SVM and neural network baselines, iDepNN is more robust to false positives in relationships spanning sentences. We evaluate our models on four datasets from newswire (MUC6) and medical (BioNLP shared task) domains that achieve state-of-the-art performance and show a better balance in precision and recall for inter-sentential relationships. We perform better than 11 teams participating in the BioNLP shared task 2016 and achieve a gain of 5.2% (0.587 vs 0.558) in F1 over the winning team. We also release the crosssentence annotations for MUC6.

SEE: Syntax-Aware Entity Embedding for Neural Relation Extraction

Distant supervised relation extraction is an efficient approach to scale relation extraction to very large corpora, and has been widely used to find novel relational facts from plain text. Recent studies on neural relation extraction have shown great progress on this task via modeling the sentences in low-dimensional spaces, but seldom considered syntax information to model the entities. In this paper, we propose to learn syntax-aware entity embedding for neural relation extraction. First, we encode the context of entities on a dependency tree as sentence-level entity embedding based on tree-GRU. Then, we utilize both intra-sentence and inter-sentence attentions to obtain sentence set-level entity embedding over all sentences containing the focus entity pair. Finally, we combine both sentence embedding and entity embedding for relation classification. We conduct experiments on a widely used real-world dataset and the experimental results show that our model can make full use of all informative instances and achieve state-of-the-art performance of relation extraction.

Distant supervision for neural relation extraction integrated with word attention and property features

Distant supervision for neural relation extraction is an efficient approach to extracting massive relations with reference to plain texts. However, the existing neural methods fail to capture the critical words in sentence encoding and meanwhile lack useful sentence information for some positive training instances. To address the above issues, we propose a novel neural relation extraction model. First, we develop a word-level attention mechanism to distinguish the importance of each individual word in a sentence, increasing the attention weights for those critical words. Second, we investigate the semantic information from word embeddings of target entities, which can be developed as a supplementary feature for the extractor. Experimental results show that our model outperforms previous state-of-the-art baselines.