Entity Relation Extraction Research Articles

Construction and application of knowledge graph for grid dispatch fault handling based on pre-trained model

With the construction of new power systems, the power grid has become extremely large, with an increasing proportion of new energy and AC/DC hybrid connections. The dynamic characteristics and fault patterns of the power grid are complex; additionally, power grid control is difficult, operation risks are high, and the task of fault handling is arduous. Traditional power-grid fault handling relies primarily on human experience. The difference in and lack of knowledge reserve of control personnel restrict the accuracy and timeliness of fault handling. Therefore, this mode of operation is no longer suitable for the requirements of new systems. Based on the multi-source heterogeneous data of power grid dispatch, this paper proposes a joint entity–relationship extraction method for power-grid dispatch fault processing based on a pre-trained model, constructs a knowledge graph of power-grid dispatch fault processing and designs, and develops a fault-processing auxiliary decision-making system based on the knowledge graph. It was applied to study a provincial dispatch control center, and it effectively improved the accident processing ability and intelligent level of accident management and control of the power grid.

QUARE: towards a question-answering model for requirements elicitation

Requirements elicitation is a stakeholder-centered approach; therefore, natural language remains an effective way of documenting and validating requirements. As the scope of the software domain grows, software analysts process a higher number of requirements documents, generating delays and errors while characterizing the software domain. Natural language processing is key in such a process, allowing software analysts for speeding up the requirements elicitation process and mitigating the impact of the ambiguity and misinterpretations coming from natural-language-based requirements documents. However, natural-language-processing-based proposals for requirements elicitation are mainly focused on specific domains and still fail for understanding several requirements writing styles. In this paper, we present QUARE, a question-answering model for requirements elicitation. The QUARE model comprises a meta-ontology for requirements elicitation, easing the generation of requirements-elicitation-related questions and the initial structuration of any software domain. In addition, the QUARE model includes a named entity recognition and relation extraction system focused on requirements elicitation, allowing software analysts for processing several requirements writing styles. Although software analysts address a software domain at a time, they use the same kind of questions for identifying and characterizing requirements abstractions such as actors, concepts, and actions from a software domain. Such a process may be framed into the QUARE model workflow. We validate our proposal by using an experimental process including real-world requirements documents coming from several software domains and requirements writing styles. The QUARE model is a novel proposal aimed at supporting software analysts in the requirements elicitation process.

Knowledge-Graph- and GCN-Based Domain Chinese Long Text Classification Method

In order to solve the current problems in domain long text classification tasks, namely, the long length of a document, which makes it difficult for the model to capture key information, and the lack of expert domain knowledge, which leads to insufficient classification accuracy, a domain long text classification model based on a knowledge graph and a graph convolutional neural network is proposed. BERT is used to encode the text, and each word’s corresponding vector is used as a node for the graph convolutional neural network so that the initialized vector contains rich semantic information. Using the trained entity–relationship extraction model, the entity-to-entity–relationships in the document are extracted and used as the edges of the graph convolutional neural network, together with syntactic dependency information. The graph structure mask is used to learn about edge relationships and edge types to further enhance the learning ability of the model for semantic dependencies between words. The method further improves the accuracy of domain long text classification by fusing knowledge features and data features. Experiments on three long text classification datasets—IFLYTEK, THUCNews, and the Chinese corpus of Fudan University—show accuracy improvements of 8.8%, 3.6%, and 2.6%, respectively, relative to the BERT model.

ASPER: Answer Set Programming Enhanced Neural Network Models for Joint Entity-Relation Extraction

AbstractA plethora of approaches have been proposed for joint entity-relation (ER) extraction. Most of these methods largely depend on a large amount of manually annotated training data. However, manual data annotation is time-consuming, labor-intensive, and error-prone. Human beings learn using both data (through induction) and knowledge (through deduction). Answer Set Programming (ASP) has been a widely utilized approach for knowledge representation and reasoning that is elaboration tolerant and adept at reasoning with incomplete information. This paper proposes a new approach, ASP-enhanced Entity-Relation extraction (ASPER), to jointly recognize entities and relations by learning from both data and domain knowledge. In particular, ASPER takes advantage of the factual knowledge (represented as facts in ASP) and derived knowledge (represented as rules in ASP) in the learning process of neural network models. We have conducted experiments on two real datasets and compare our method with three baselines. The results show that our ASPER model consistently outperforms the baselines.

Joint Multimodal Entity-Relation Extraction Based on Edge-Enhanced Graph Alignment Network and Word-Pair Relation Tagging

Multimodal named entity recognition (MNER) and multimodal relation extraction (MRE) are two fundamental subtasks in the multimodal knowledge graph construction task. However, the existing methods usually handle two tasks independently, which ignores the bidirectional interaction between them. This paper is the first to propose jointly performing MNER and MRE as a joint multimodal entity-relation extraction (JMERE) task . Besides, the current MNER and MRE models only consider aligning the visual objects with textual entities in visual and textual graphs but ignore the entity-entity relationships and object-object relationships. To address the above challenges, we propose an edge-enhanced graph alignment network and a word-pair relation tagging (EEGA) for the JMERE task. Specifically, we first design a word-pair relation tagging to exploit the bidirectional interaction between MNER and MRE and avoid error propagation. Then, we propose an edge-enhanced graph alignment network to enhance the JMERE task by aligning nodes and edges in the cross-graph. Compared with previous methods, the proposed method can leverage the edge information to auxiliary alignment between objects and entities and find the correlations between entity-entity relationships and object-object relationships. Experiments are conducted to show the effectiveness of our model.

Research on Chinese Semantic Named Entity Recognition in Marine Engine Room Systems Based on BERT

With the development of intelligentization in maritime vessels, the pursuit of an organized and scalable knowledge storage approach for marine engine room systems has become one of the current research hotspots. This study addressed the foundational named entity recognition (NER) task in constructing a knowledge graph for marine engine rooms. It proposed an entity recognition algorithm for Chinese semantics in marine engine rooms that integrates language models. Firstly, the bidirectional encoder representation from transformers (BERT) language model is used to extract text features and obtain word-level granularity vector matrices. Secondly, the trained word embeddings are fed into a bidirectional long short-term memory network (BiLSTM) to extract contextual information. It considers the surrounding words and their sequential relationships, enabling a better understanding of the context. Additionally, the conditional random field (CRF) model was used to extract the globally optimal sequence of named entities in the marine engine room semantic. The CRF model considered the dependencies between adjacent entities that ensured a coherent and consistent final result for entity recognition in marine engine room semantics. The experiment results demonstrate that the proposed algorithm achieves superior F1 scores for all three entity types. Compared with BERT, the overall precision, recall, and F1 score of the entity recognition are improved by 1.36%, 1.41%, and 1.38%, respectively. Future research will be carried out on named entity recognition of a small sample set to provide basic support for more efficient entity relationship extraction and construction of a marine engine room knowledge graph.

Entity Relationship Extraction Based on a Multi-Neural Network Cooperation Model

Entity relation extraction mainly extracts relations from text, which is one of the important tasks of natural language processing. At present, some special fields have insufficient data; for example, agriculture, the metallurgical industry, etc. There is a lack of an effective model for entity relationship recognition under the condition of insufficient data. Inspired by this, we constructed a suitable small balanced data set and proposed a multi-neural network collaborative model (RBF, Roberta–Bidirectional Gated Recurrent Unit–Fully Connected). In addition, we also optimized the proposed model. This model uses the Roberta model as the coding layer, which is used to extract the word-level features of the text. This model uses BiGRU (Bidirectional Gated Recurrent Unit)–FC (Fully Connected) as the decoding layer, which is used to obtain the optimal relationship of the text. To further improve the effect, the input layer is optimized by feature fusion, and the learning rate is optimized by the cosine annealing algorithm. The experimental results show that, using the small balanced data set, the F1 value of the RBF model proposed in the paper is 25.9% higher than the traditional Word2vec–BiGRU–FC model. It is 18.6% higher than the recent Bert–BiLSTM (Bidirectional Long Short Term Memory)–FC model. The experimental results show that our model is effective.

PipCKG-BS: A Method to Build Cybersecurity Knowledge Graph for Blockchain Systems via the Pipeline Approach

The increasing sophistication of cyberattacks on blockchain systems has significantly disrupted security experts from gaining immediate insight into the security situation. The Cybersecurity Knowledge Graph (CKG) currently provides a novel technical solution for blockchain system situational awareness by integrating massive fragmented Cyber Threat Intelligence (CTI) about blockchain technology. However, the existing literature does not provide a solution for building CKG appropriate for blockchain systems. Therefore, designing a method to construct a CKG for blockchain systems by efficiently extracting information from the CTI is mandatory. This paper proposes PipCKG-BS, a pipeline-based approach that builds CKG for blockchain systems. The PipCKG-BS incorporates contextual features and Pre-trained Language Models (PLMs) to improve the performance of the information extraction process. Precisely, we develop the Named Entity Recognition (NER) and Relation Extraction (RE) models for cybersecurity text in PipCKG-BS. In the NER model, we apply the prompt-based learning paradigm to cybersecurity text by constructing prompt templates. In the RE model, we employ external features and prior knowledge of sentences to improve entity relationship extraction accuracy. Several experimental results demonstrate that PipCKG-BS is better than advanced methods in extracting CTI information and is an appealing solution to build high-quality CKG for blockchain systems.

A Joint Extraction System Based on Conditional Layer Normalization for Health Monitoring

Natural language processing (NLP) technology has played a pivotal role in health monitoring as an important artificial intelligence method. As a key technology in NLP, relation triplet extraction is closely related to the performance of health monitoring. In this paper, a novel model is proposed for joint extraction of entities and relations, combining conditional layer normalization with the talking-head attention mechanism to strengthen the interaction between entity recognition and relation extraction. In addition, the proposed model utilizes position information to enhance the extraction accuracy of overlapping triplets. Experiments on the Baidu2019 and CHIP2020 datasets demonstrate that the proposed model can effectively extract overlapping triplets, which leads to significant performance improvements compared with baselines.

Chinese Clinical Named Entity Recognition From Electronic Medical Records Based on Multisemantic Features by Using Robustly Optimized Bidirectional Encoder Representation From Transformers Pretraining Approach Whole Word Masking and Convolutional Neural Networks: Model Development and Validation

Clinical electronic medical records (EMRs) contain important information on patients' anatomy, symptoms, examinations, diagnoses, and medications. Large-scale mining of rich medical information from EMRs will provide notable reference value for medical research. With the complexity of Chinese grammar and blurred boundaries of Chinese words, Chinese clinical named entity recognition (CNER) remains a notable challenge. Follow-up tasks such as medical entity structuring, medical entity standardization, medical entity relationship extraction, and medical knowledge graph construction largely depend on medical named entity recognition effects. A promising CNER result would provide reliable support for building domain knowledge graphs, knowledge bases, and knowledge retrieval systems. Furthermore, it would provide research ideas for scientists and medical decision-making references for doctors and even guide patients on disease and health management. Therefore, obtaining excellent CNER results is essential. We aimed to propose a Chinese CNER method to learn semantics-enriched representations for comprehensively enhancing machines to understand deep semantic information of EMRs by using multisemantic features, which makes medical information more readable and understandable. First, we used Robustly Optimized Bidirectional Encoder Representation from Transformers Pretraining Approach Whole Word Masking (RoBERTa-wwm) with dynamic fusion and Chinese character features, including 5-stroke code, Zheng code, phonological code, and stroke code, extracted by 1-dimensional convolutional neural networks (CNNs) to obtain fine-grained semantic features of Chinese characters. Subsequently, we converted Chinese characters into square images to obtain Chinese character image features from another modality by using a 2-dimensional CNN. Finally, we input multisemantic features into Bidirectional Long Short-Term Memory with Conditional Random Fields to achieve Chinese CNER. The effectiveness of our model was compared with that of the baseline and existing research models, and the features involved in the model were ablated and analyzed to verify the model's effectiveness. We collected 1379 Yidu-S4K EMRs containing 23,655 entities in 6 categories and 2007 self-annotated EMRs containing 118,643 entities in 7 categories. The experiments showed that our model outperformed the comparison experiments, with F1-scores of 89.28% and 84.61% on the Yidu-S4K and self-annotated data sets, respectively. The results of the ablation analysis demonstrated that each feature and method we used could improve the entity recognition ability. Our proposed CNER method would mine the richer deep semantic information in EMRs by multisemantic embedding using RoBERTa-wwm and CNNs, enhancing the semantic recognition of characters at different granularity levels and improving the generalization capability of the method by achieving information complementarity among different semantic features, thus making the machine semantically understand EMRs and improving the CNER task accuracy.

Review of entity relation extraction

In today’s big data era, there are a large number of unstructured information resources on the web. Natural language processing researchers have been working hard to figure out how to extract useful information from them. Entity Relation Extraction is a crucial step in Information Extraction and provides technical support for Knowledge Graphs, Intelligent Q&A systems and Intelligent Retrieval. In this paper, we present a comprehensive history of entity relation extraction and introduce the relation extraction methods based on Machine Mearning, the relation extraction methods based on Deep Learning and the relation extraction methods for open domains. Then we summarize the characteristics and representative results of each type of method and introduce the common datasets and evaluation systems for entity relation extraction. Finally, we summarize current entity relation extraction methods and look forward to future technologies.

THT: Transfer Head and Tail for Joint Entity Relation Extraction

Extracting entities and relations from unstructured text is the basic task of building knowledge graphs. Various methods based on sequence labeling have been proposed recently, but most of them use uniform token representations to identify entities. It is clear that the first and last words of the head and tail entities should have different representations. For this reason, we propose a method based on word interaction that provides different representations for the first and last words of entities. Specifically, initial token representation is obtained by BERT. Each token interacts with the next k tokens for the entity’s first token recognition, interacts with the previous k tokens for the entity’s last token recognition, and merges them to obtain the head entity. Then recognize the tail entities based on the head entities and form entity pairs. Finally, get the triples based on a simple multi-classification for relation determination. Extensive experimental results on four datasets show that THT performs better than state-of-the-art baselines.

AJE: Attention Mechanism for Entity-relation Joint Extraction

Joint extraction of entities and relations is an significant issue of information extraction, which is very helpful for many downstream tasks, including knowledge base construction, question answering, and biomedical text diagnosis[1], etc. The common approach of existing models is to extract the subject and the relation first, then compute the subject and the relation to obtain the object, and finally, the triplet is judged. However, such an approach cannot efficiently handle of information extraction, and the results are not very good for Subject-Object Overlap (SOO) case. In this paper, a joint entity-relation extraction method AJE is proposed based on dot-product attention mechanism. The method first maps subject, object and relationship into three matrices of Q, K and V. After that, the attention weighting is achieved on these three matrices and the output vector is used to determine whether the triple is correct or not. The F1-score is used in experiments to show that the proposed model is more efficient than the current existing ones. It also has better results in handling other cases such as SOO, multi-triple problem, etc.

Detecting contradictions from IoT protocol specification documents based on neural generated knowledge graph

Due to the boom of Internet of Things (IoT) in recent years, various IoT devices are connected to the Internet and communicate with each other through network protocols such as the Constrained Application Protocol (CoAP). These protocols are typically defined and described in specification documents, such as Request for Comments (RFC), which are written in natural or semi-formal languages. Since developers largely follow the specification documents when implementing web protocols, they have become the de facto protocol specifications. Therefore, it must be ensured that the descriptions in them are consistent to avoid technological issues, incompatibility, security risks, or even legal concerns. In this work, we propose Neural RFC Knowledge Graph (NRFCKG), a neural network-generated knowledge graph based contradictions detection tool for IoT protocol specification documents. Our approach can automatically parse the specification documents and construct knowledge graphs from them through entity extraction, relation extraction, and rule extraction with large language models. It then conducts an intra-entity and inter-entity contradiction detection over the generated knowledge graph. We implement NRFCKG and apply it to the most extensively used messaging protocols in IoT, including the main RFC (RFC7252) of CoAP, the specification document of MQTT, and the specification document of AMQP. Our evaluation shows that NRFCKG generalizes well to other specification documents and it manages to detect contradictions from these IoT protocol specification documents.

Information Extraction Network Based on Multi-Granularity Attention and Multi-Scale Self-Learning.

Transforming the task of information extraction into a machine reading comprehension (MRC) framework has shown promising results. The MRC model takes the context and query as the inputs to the encoder, and the decoder extracts one or more text spans as answers (entities and relationships) from the text. Existing approaches typically use multi-layer encoders, such as Transformers, to generate hidden features of the source sequence. However, increasing the number of encoder layers can lead to the granularity of the representation becoming coarser and the hidden features of different words becoming more similar, potentially leading to the model's misjudgment. To address this issue, a new method called the multi-granularity attention multi-scale self-learning network (MAML-NET) is proposed, which enhances the model's understanding ability by utilizing different granularity representations of the source sequence. Additionally, MAML-NET can independently learn task-related information from both global and local dimensions based on the learned multi-granularity features through the proposed multi-scale self-learning attention mechanism. The experimental results on two information extraction tasks, named entity recognition and entity relationship extraction, demonstrated that the method was superior to the method based on machine reading comprehension and achieved the best performance on the five benchmark tests.

A Survey on Multimodal Knowledge Graphs: Construction, Completion and Applications

As an essential part of artificial intelligence, a knowledge graph describes the real-world entities, concepts and their various semantic relationships in a structured way and has been gradually popularized in a variety practical scenarios. The majority of existing knowledge graphs mainly concentrate on organizing and managing textual knowledge in a structured representation, while paying little attention to the multimodal resources (e.g., pictures and videos), which can serve as the foundation for the machine perception of a real-world data scenario. To this end, in this survey, we comprehensively review the related advances of multimodal knowledge graphs, covering multimodal knowledge graph construction, completion and typical applications. For construction, we outline the methods of named entity recognition, relation extraction and event extraction. For completion, we discuss the multimodal knowledge graph representation learning and entity linking. Finally, the mainstream applications of multimodal knowledge graphs in miscellaneous domains are summarized.

Personalized Education Resource Recommendation Method Based on Deep Learning in Intelligent Educational Robot Environments

The goal of this article is to analyze the problem of low computational efficiency and propagation error rate in entity recognition and relation extraction. This paper proposes a personalized education resource recommendation algorithm framework XMAMBLSTM based on deep learning in an intelligent education robot environment. XMAMBLSTM uses XLNet to assign word vectors to text sequences, employs a Multi-Bi-LSTM layer to represent complex information of word vectors, and combines a multi-headed attention layer to realize weight distribution of each word vector. The experimental results show that compared with the traditional collaborative filtering algorithm, the comprehensive evaluation indexes of the proposed method, based on the intelligent education robot environment on the two platforms, are higher than 5.05% and 17.3%, respectively.

Abstract 5365: Extracting novel knowledge from scientific literature to build a web portal for cancer researchers to keep up with the latest scientific discoveries

Abstract Millions of new papers are published in biomedical sciences every year. In many disciplines, it has become impossible to read all the published new papers to learn what is happening in the frontier of a particular area. This gap is widening as the publishing speed has been accelerating in recent years. To address this challenge, one can convert unstructured text data into a structured form, which can then support highly accurate information retrieval, information integration, and automated knowledge discovery. A plausible approach for such a task is to use named entity recognition (NER) and relation extraction (RE) methods to identify important biological entities and extract their relations to construct knowledge graphs (KGs). The LitCoin Natural Language Processing (NLP) Challenge was recently organized by NCATS of NIH and NASA to spur innovation by rewarding the most creative and high-impact uses of biomedical text to create KGs. In addition to entities and relations, the manually annotated LitCoin dataset also contains the annotations of relations being new discoveries or background knowledge. Our team participated in the challenge and ranked first place. The novelty prediction model of our pipeline has achieved an F1 score of 0.90. We have applied our model to all the PubMed abstracts published previously and the newly published ones to extract the novel discoveries in each article. A web portal has been created to allow scientists to view the latest discoveries in cancer research. The web portal is updated daily with versatile visualization tools for cancer researchers to quickly grasp the latest discoveries in a particular area. It also offers powerful functions to explore the existing literature and make sophisticated inferences about causal and indirect relationships. Citation Format: Feng Pan, Yuan Zhang, Xin Sui, Donghu Sun, Menghan Chung, Jinfeng Zhang. Extracting novel knowledge from scientific literature to build a web portal for cancer researchers to keep up with the latest scientific discoveries. [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 5365.

Abstract 5366: Constructing the largest-scale biomedical knowledge graph using all PubMed articles and its application in automated knowledge discovery

Abstract The number of biomedical publications is growing at an accelerated speed. This ever-increasing amount of scientific literature has made reading all the published articles regularly impossible even for a very specific research area. A solid grasp of existing literature is essential for coming out with novel and plausible scientific ideas. To bridge the gap between the published scientific findings and our incapability of manually processing them, we need to convert the unstructured text into structured form to enable automated methods to use the structured, machine-readable information to generate novel hypotheses, which can then be manually validated. A plausible approach for converting unstructured text into structured form is to use named entity recognition (NER) and relation extraction (RE) methods to identify the biological entities and extract their relations to construct knowledge graphs (KGs). KGs can link concepts within existing research to allow researchers to find connections that may have been difficult to discover without them. The LitCoin Natural Language Processing (NLP) Challenge was recently organized by NCATS of NIH and NASA to spur innovation by rewarding the most creative and high-impact uses of biomedical, publication-free text to create KGs. Our team participated in the challenge and ranked first place. Using the pipelines developed for the LitCoin NLP challenge, we have constructed the largest-scale biomedical KG using all PubMed articles. We further develop advanced deep-learning methods to predict new links from the constructed KG. We demonstrate the power of this new framework using several examples important for drug discovery. Citation Format: Yuan Zhang, Feng Pan, Xin Sui, Donghu Sun, Menghan Chung, Jinfeng Zhang. Constructing the largest-scale biomedical knowledge graph using all PubMed articles and its application in automated knowledge discovery. [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 5366.

Abstract 5421: Constructing the largest-scale knowledge graph using all PubMed abstracts and its application for highly specific and accurate knowledge retrieval

Abstract Most of the biomedical knowledge the research community has acquired during the past few decades has been deposited in scientific literature as unstructured text. Converting the unstructured text into the structured form will enable novel methodologies and applications for scientific discovery that can fully harness the power of the existing knowledge. To this end, two fundamental questions need to be addressed: named entity recognition (NER) and relation extraction (RE). NER deals with identifying the concepts or entities in texts, such as diseases, genes/proteins, chemical compounds, etc. while RE aims to extract the relations among these entities. Together, the extracted information forms a knowledge graph (KG) where the nodes are entities in the texts and the edges represent their relationships. KGs can link concepts within existing research to allow researchers to find connections that may have been difficult to discover without them. The LitCoin Natural Language Processing (NLP) Challenge was recently organized by NCATS of NIH and NASA to spur innovation by rewarding the most creative and high-impact uses of biomedical text to create KGs. Our team participated in the challenge and ranked first place. We have applied the methods we developed for the LitCoin NLP challenge to all PubMed abstracts and constructed the largest-scale biomedical KG to date. We show that powerful and versatile query functions can be implemented on top of the KG to enable highly specific and accurate knowledge retrieval and inference of causal and indirect relationships. Citation Format: Xin Sui, Yuan Zhang, Feng Pan, Donghu Sun, Menghan Chung, Jinfeng Zhang. Constructing the largest-scale knowledge graph using all PubMed abstracts and its application for highly specific and accurate knowledge retrieval. [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 5421.