Knowledge Triples Research Articles

Construction of Legal Knowledge Graph Based on Knowledge-Enhanced Large Language Models

Legal knowledge involves multidimensional heterogeneous knowledge such as legal provisions, judicial interpretations, judicial cases, and defenses, which requires extremely high relevance and accuracy of knowledge. Meanwhile, the construction of a legal knowledge reasoning system also faces challenges in obtaining, processing, and sharing multisource heterogeneous knowledge. The knowledge graph technology, which is a knowledge organization form with triples as the basic unit, is able to efficiently transform multisource heterogeneous information into a knowledge representation form close to human cognition. Taking the automated construction of the Chinese legal knowledge graph (CLKG) as a case scenario, this paper presents a joint knowledge enhancement model (JKEM), where prior knowledge is embedded into a large language model (LLM), and the LLM is fine-tuned through the prefix of the prior knowledge data. Under the condition of freezing most parameters of the LLM, this fine-tuning scheme adds continuous deep prompts as prefix tokens to the input sequences of different layers, which can significantly improve the accuracy of knowledge extraction. The results show that the knowledge extraction accuracy of the JKEM in this paper reaches 90.92%. Based on the superior performance of this model, the CLKG is further constructed, which contains 3480 knowledge triples composed of 9 entities and 2 relationships, providing strong support for an in-depth understanding of the complex relationships in the legal field.

SF-GPT: A training-free method to enhance capabilities for knowledge graph construction in LLMs

Knowledge graphs (KGs) are constructed by extracting knowledge triples from text and fusing knowledge, enhancing information retrieval efficiency. Current methods for knowledge triple extraction include ”Pretrain and Fine-tuning” and Large Language Models (LLMs). The former shifts effort from manual extraction to dataset annotation and suffers from performance degradation with different test and training set distributions. LLMs-based methods face errors and incompleteness in extraction. We introduce SF-GPT, a training-free method to address these issues. Firstly, we propose the Entity Extraction Filter (EEF) module to filter triple generation results, addressing evaluation and cleansing challenges. Secondly, we introduce a training-free Entity Alignment Module based on Entity Alias Generation (EAG), tackling semantic richness and interpretability issues in LLM-based knowledge fusion. Finally, our Self-Fusion Subgraph strategy uses multi-response self-fusion and a common entity list to filter triple results, reducing noise from LLMs’ multi-responses. In experiments, SF-GPT showed a 55.5% increase in recall and a 32.6% increase in F1 score on the BDNC dataset compared to the UniRel model trained on the NYT dataset and achieved a 5% improvement in F1 score compared to GPT-4+EEF baseline on the WebNLG dataset in the case of a fusion round of three. SF-GPT offers a promising way to extract knowledge from unstructured information.

Knowledge Graph Guided Neural Machine Translation with Dynamic Reinforce-selected Triples

Previous methods incorporating knowledge graphs (KGs) into neural machine translation (NMT) adopt a static knowledge utilization strategy, that introduces many useless knowledge triples and makes the useful triples difficult be utilized by NMT. To address this problem, we propose a KG guided NMT model with dynamic reinforce-selected triples. The proposed methods could dynamically select the different useful knowledge triples for different source sentences. Specifically, the proposed model contains two components: 1) knowledge selector, that dynamically selects useful knowledge triples for a source sentence, and 2) knowledge guided NMT (KgNMT), that utilizes the selected triples to guide the translation of NMT. Meanwhile, to overcome the non-differentiable problem and guide the training procedure, we propose a policy gradient strategy to encourage the model to select useful triples and improve the generation probability of gold target sentence. Various experimental results show that the proposed method can significantly outperform the baseline models in both translation quality and handling the entities.

BVTED: A Specialized Bilingual (Chinese–English) Dataset for Vulnerability Triple Extraction Tasks

Extracting knowledge from cyber threat intelligence is essential for understanding cyber threats and implementing proactive defense measures. However, there is a lack of open datasets in the Chinese cybersecurity field that support both entity and relation extraction tasks. This paper addresses this gap by analyzing vulnerability description texts, which are standardized and knowledge-dense, to create a vulnerability knowledge ontology comprising 13 entities and 15 relations. We annotated 27,311 unique vulnerability description sentences from the China National Vulnerability Database, resulting in a dataset named BVTED for cybersecurity knowledge triple extraction tasks. BVTED contains 97,391 entities and 69,614 relations, with entities expressed in a mix of Chinese and English. To evaluate the dataset’s value, we trained five deep learning-based named entity recognition models, two relation extraction models, and two joint entity–relation extraction models on BVTED. Experimental results demonstrate that models trained on this dataset achieve excellent performance in vulnerability knowledge extraction tasks. This work enhances the extraction of cybersecurity knowledge triples from mixed Chinese and English threat intelligence corpora by providing a comprehensive ontology and a new dataset, significantly aiding in the mining, analysis and utilization of the knowledge embedded in cyber threat intelligence.

Knowledge Graph-Aware Deep Interest Extraction Network on Sequential Recommendation

Sequential recommendation is the mainstream approach in the field of click-through-rate (CTR) prediction for modeling users’ behavior. This behavior implies the change of the user’s interest, and the goal of sequential recommendation is to capture this dynamic change. However, existing studies have focused on designing complex dedicated networks to capture user interests from user behavior sequences, while neglecting the use of auxiliary information. Recently, knowledge graph (KG) has gradually attracted the attention of researchers as a structured auxiliary information. Items and their attributes in the recommendation, can be mapped to knowledge triples in the KG. Therefore, the introduction of KG to recommendation can help us obtain more expressive item representations. Since KG can be considered a special type of graph, it is possible to use the graph neural network (GNN) to propagate the rich information contained in the KG into the item representation. Based on this idea, this paper proposes a recommendation method that uses KG as auxiliary information. The method first propagates the knowledge information in the KG using GNN to obtain a knowledge-rich item representation. Then the temporal features in the item sequence are extracted using a transformer for CTR prediction, namely the Knowledge Graph-Aware Deep Interest Extraction network (KGDIE). To evaluate the performance of this model, we conducted extensive experiments on two real datasets with different scenarios. The results showed that the KGDIE method could outperform several state-of-the-art baselines. The source code of our model is available at https://github.com/gylgyl123/kgdie.

KnowCTI: Knowledge-based cyber threat intelligence entity and relation extraction

Structured cyber threat intelligence enables security researchers to know the occurrence of cyber threats in time, thereby improving the efficiency of security defense and analysis. Previous works usually use general deep learning and NLP techniques to extract intelligence. Such methods suffer from insufficient semantic understanding in the field of security. To address these issues, we propose a novel method called Knowledge-based Cyber Threat Intelligence Entity and Relation Extraction (KnowCTI), which incorporates cybersecurity knowledge into the model to enhance the understanding of the realm of cybersecurity and has a full picture of threats with the threat intelligence graph generation. Specifically, we first build a cybersecurity knowledge base and train cybersecurity-aware knowledge embeddings based on the base. Secondly, we refine the most related knowledge triples by attention mechanism and gate mechanism, and then construct a sentence tree through these triples. Next, we employ graph attention networks to incorporate knowledge information into the sentence by considering the sentence tree as a graph. Finally, we consider entity extraction as a sequence labeling problem and relation extraction as a classification problem to decode the entities and relation triples according to the threat intelligence ontology we designed. Experimental results demonstrate the superior performance with the F1 score exceeding 90.16 and 81.83 on entity and relation extraction separately.

CyberQ: Generating Questions and Answers for Cybersecurity Education Using Knowledge Graph-Augmented LLMs

Building a skilled cybersecurity workforce is paramount to building a safer digital world. However, the diverse skill set, constantly emerging vulnerabilities, and deployment of new cyber threats make learning cybersecurity challenging. Traditional education methods struggle to cope with cybersecurity's rapidly evolving landscape and keep students engaged and motivated. Different studies on students' behaviors show that an interactive mode of education by engaging through a question-answering system or dialoguing is one of the most effective learning methodologies. There is a strong need to create advanced AI-enabled education tools to promote interactive learning in cybersecurity. Unfortunately, there are no publicly available standard question-answer datasets to build such systems for students and novice learners to learn cybersecurity concepts, tools, and techniques. The education course material and online question banks are unstructured and need to be validated and updated by domain experts, which is tedious when done manually. In this paper, we propose CyberGen, a novel unification of large language models (LLMs) and knowledge graphs (KG) to generate the questions and answers for cybersecurity automatically. Augmenting the structured knowledge from knowledge graphs in prompts improves factual reasoning and reduces hallucinations in LLMs. We used the knowledge triples from cybersecurity knowledge graphs (AISecKG) to design prompts for ChatGPT and generate questions and answers using different prompting techniques. Our question-answer dataset, CyberQ, contains around 4k pairs of questions and answers. The domain expert manually evaluated the random samples for consistency and correctness. We train the generative model using the CyberQ dataset for question answering task.

Exploration of Innovative Teaching Methods in the Teaching Reform of Linear Algebra Course in the Internet Era

Abstract For the task of ternary extraction in unstructured text, an entity recognition model based on BiLSTM-CRF is proposed. By introducing graph convolutional neural network and attention mechanism respectively, a relation extraction model based on attention-guided graph convolutional neural network and a text classification model based on GCN are proposed to realize the automatic extraction of linear algebra knowledge triples. To solve the problem of ignoring semantic information of knowledge points and sparse data, the linear algebra knowledge level matrix of learners is obtained by using the DKVMN model. The knowledge graph’s relationships are assigned weights by experts through evaluation. The semantic information in the knowledge graph and the weights of relationships between nodes are utilized to complement the knowledge level matrix as a way to recommend appropriate linear algebra exercises for learners. Compared with the traditional teaching class, the experimental class of this paper’s method scores higher in algorithmic thinking, collaboration, critical thinking, and problem-solving by 0.67, 0.27, 0.41, and 0.37, respectively. Based on the learning data of each chapter’s knowledge points, it is understood that the standard deviation of the eigenvalues and eigenvectors in the eigenvalue and eigenvector and quadratic form, quadratic form, similarity matrices, positive qualitative and geometric questions, and proofs are respectively 4.076, 3.226, 6.185, 3.292, and 4.105, and the standard deviations are generally large compared to other chapters. This paper demonstrates that the method can be taught accurately depending on the students’ linear algebra level.

A hybrid methodology for knowledge organization and application of Chinese civil aviation regulations from mission safety support perspective

Safety is one of the most important issues in aviation. Aviation regulations provide significant information pertinent to the safety design and operation of aircraft; however, this information has not been effectively used. It is difficult to precisely identify and obtain the necessary information due to massive and unstructured provisions. In this study, a hybrid methodology is proposed to realize knowledge system construction using Chinese Civil Aviation Regulations as the object of study. To realize structured knowledge organization and intelligent knowledge application for aviation regulations, the hybrid methodology integrates a semantic cohesion model, a knowledge recognition model, a knowledge organization model, and a knowledge application model. A knowledge system of aviation regulations is built using the hybrid methodology comprising all knowledge necessary for aviation safety. The system provides intelligent knowledge support for the safety analysis of aircraft from the perspective of control system, including the accurate positioning of control elements and thorough acquisition of control conditions. Experiments were conducted to confirm the accuracy of the proposed method, and the 56,853 knowledge triples contained in the knowledge system supported its reliability. A few examples of knowledge retrieval are provided, focusing on the interaction processes of socio-technical elements during aircraft missions. It takes only a few seconds to acquire the knowledge required for safety analysis. The examples show how the hybrid methodology and knowledge system can be utilized to increase the efficiency of safety analysis for socio-technical systems while advancing intelligent knowledge applications in the aviation domain.

KerPrint: Local-Global Knowledge Graph Enhanced Diagnosis Prediction for Retrospective and Prospective Interpretations

While recent developments of deep learning models have led to record-breaking achievements in many areas, the lack of sufficient interpretation remains a problem for many specific applications, such as the diagnosis prediction task in healthcare. The previous knowledge graph(KG) enhanced approaches mainly focus on learning clinically meaningful representations, the importance of medical concepts, and even the knowledge paths from inputs to labels. However, it is infeasible to interpret the diagnosis prediction, which needs to consider different medical concepts, various medical relationships, and the time-effectiveness of knowledge triples in different patient contexts. More importantly, the retrospective and prospective interpretations of disease processes are valuable to clinicians for the patients' confounding diseases. We propose KerPrint, a novel KG enhanced approach for retrospective and prospective interpretations to tackle these problems. Specifically, we propose a time-aware KG attention method to solve the problem of knowledge decay over time for trustworthy retrospective interpretation. We also propose a novel element-wise attention method to select candidate global knowledge using comprehensive representations from the local KG for prospective interpretation. We validate the effectiveness of our KerPrint through an extensive experimental study on a real-world dataset and a public dataset. The results show that our proposed approach not only achieves significant improvement over knowledge-enhanced methods but also gives the interpretability of diagnosis prediction in both retrospective and prospective views.

BERT and hierarchical cross attention-based question answering over bridge inspection knowledge graph

Aiming at the problem of insufficient knowledge service in the field of bridge inspection, this paper proposes a knowledge graph question answering (KGQA) model by using BERT and a novel hierarchical cross-attention mechanism. First, the BERT and static domain dictionaries are used as embedding layer for bridge inspection QA pairs to extract multi-granularity features. Second, in order to extract the topic entities from the domain-specific questions, the bidirectional long short-term memory model is employed to further extract features of the contextual dependency. Finally, the proposed hierarchical cross-attention mechanism realizes information interaction among the questions and knowledge triples, and calculates the similarity from shallow vocabulary and deep semantics. The proposed model is evaluated by using a general KGQA benchmark and a Chinese bridge inspection KGQA dataset. The experimental results show that the proposed approach achieves outstanding performance on both datasets. In addition, a KGQA prototype system is employed as use case to illustrate the application effect in the practical scenario.

Revisiting Electrocatalyst Design by a Knowledge Graph of Cu-Based Catalysts for CO2 Reduction

Electrocatalysis takes a significant role in the production of sustainable fuels and chemicals. The combination of artificial intelligence and catalytic science is exhibiting great potential to extract, analyze, and predict electrocatalysts. However, the currently developed machine learning approach usually requires a mass of data from density functional theory calculations to train and optimize models. In contrast, a knowledge graph has the potential to extract useful information from a large amount of the literature without referring to density functional theory. Herein, a knowledge graph of Cu-based electrocatalysts for electrocatalytic CO2 reduction is constructed based on a linguistically enriched SciBERT-based framework. This framework retrieves multiple types of entities including material, regulation method, product, Faradaic efficiency, etc. from 757 scientific literature, generates representations with abundant domain-specific semantic information, and exhibits the capability to deal with electrocatalysts for CO2 reduction. The obtained graph shows the development history of related catalysts, builds relationships between the factors associated with catalysis, and provides intuitive charts for researchers to gain useful information. Furthermore, we propose a deep learning-based prediction model, which integrates the semantic information from the scientific literature (word embedding) with the correlation of knowledge triples (graph embedding) and realizes the prediction of the Faradaic efficiency for a targeted case. This work paves the way for catalyst design in the manner of merging artificial intelligence with catalytic science.

Ontological knowledge inferring approach: Introducing Directed Collocations (DC) and Joined Directed Collocations (JDC)

The growing need of utilizing unstructured knowledge embedded in open-domain natural language text into machine-processable forms requires the induction of hardly extracted structured knowledge into knowledge bases which makes the Semantic Web vision a reality. In this context, ontologies, and ontological knowledge (triples) plays a vital role. This research introduces two novel concepts named Directed Collocation (DC) and Joined Directed Collocation (JDC) along with a methodical application of them to infer new ontological knowledge. Introduced Quality-Threshold-Value (QTV) parameter improves the quality of the inferred ontological knowledge. Having set a moderate value (3) for QTV, this approach inferred 95,491 new ontological knowledge from 43,100 triples of open domain Sri Lankan English news corpus. Indeed, the outcome was approximately doubled in size as the source corpus. Some inferred ontological knowledge was identical with the original corpus content, which evidences the accuracy of this approach. The remaining were validated using inter-rater agreement method (high reliability) and out of which around 56% were estimated as effective. The inferred outcome which is in the triple format may use in any knowledge base. The proposed approach is domain independent. Thus, helps to construct/extend ontologies for any domain with the help of less or no human specialists.

Automatic Chinese knowledge-based question answering by the MGBA-LSTM-CNN model

The purpose of knowledge-based question answering (KBQA) is to accurately answer the questions raised by users through knowledge triples. Traditional Chinese KBQA methods rely heavily on artificial features, resulting in unsatisfactory QA results. To solve the above problems, this paper divides Chinese KBQA into two parts: entity extraction and attribute mapping. In the entity extraction stage, the improved Bi-LSTM-CNN-CRF model is used to identify the entity of questions and the Levenshtein distance method is used to resolve the entity link error. In the attribute mapping stage, according to the characteristics of questions and candidate attributes, the MGBA-LSTM-CNN model is proposed to encode questions and candidate attributes from the semantic level and word level, respectively, and splice them into new semantic vectors. Finally, the cosine distance is used to measure the similarity of the two vectors to find candidate attributes most similar to questions. The experimental results show that the system achieves good results in the Chinese question and answer data set.

Fine-grained biomedical knowledge negation detection via contrastive learning

Negation is a common and complex phenomenon in natural language, which will reverse the truth value of knowledge extracted from sentences and make knowledge validation indispensable. Negation detection at the granularity of knowledge triples is still a challenging and minimally explored task, especially in the biomedical domain where multiple knowledge triples frequently exist in one sentence and the same triple can express quite distinct meanings under different contexts. In this paper, we illustrate and define the label conflict problem in biomedical negation detection and propose CL-GSAN, a novel Contrastive Learning-based Gated Structured Attention Network for identifying the negated knowledge triples from the biomedical text. CL-GSAN jointly performs fine-grained knowledge-context modeling within a sentence via a gating mechanism and learns the target triple’s context representations by considering the semantic divergence of claims via contrastive learning. Experimental results demonstrate that CL-GSAN obtains promising performance on three biomedical corpora, including the Chinese Medical Knowledge Negation (CMKN) corpus constructed by us to facilitate fine-grained, i.e., triple-level, knowledge negation detection.

Towards Robust Knowledge Graph Embedding via Multi-Task Reinforcement Learning

Nowadays, Knowledge graphs (KGs) have been playing a pivotal role in AI-related applications. Despite the large sizes, existing KGs are far from complete and comprehensive. In order to continuously enrich KGs, automatic knowledge construction and update mechanisms are usually utilized, which inevitably bring in plenty of noise. However, most existing knowledge graph embedding (KGE) methods assume that all the triple facts in KGs are correct, and project both entities and relations into a low-dimensional space without considering noise and knowledge conflicts. This will lead to low-quality and unreliable representations of KGs. To this end, in this paper, we propose a general multi-task reinforcement learning framework, which can greatly alleviate the noisy data problem. In our framework, we exploit reinforcement learning for choosing high-quality knowledge triples while filtering out the noisy ones. Also, in order to take full advantage of the correlations among semantically similar relations, the triple selection processes of similar relations are trained in a collective way with multi-task learning. Moreover, we extend popular KGE models TransE, DistMult, ConvE and RotatE with the proposed framework. Finally, the experimental validation shows that our approach is able to enhance existing KGE models and can provide more robust representations of KGs in noisy scenarios.

Hyperbolic Directed Hypergraph-Based Reasoning for Multi-Hop KBQA

The target of the multi-hop knowledge base question-answering task is to find answers of some factoid questions by reasoning across multiple knowledge triples in the knowledge base. Most of the existing methods for multi-hop knowledge base question answering based on a general knowledge graph ignore the semantic relationship between each hop. However, modeling the knowledge base as a directed hypergraph has the problems of sparse incidence matrices and asymmetric Laplacian matrices. To make up for the deficiency, we propose a directed hypergraph convolutional network modeled on hyperbolic space, which can better deal with the sparse structure, and effectively adapt to the problem of an asymmetric incidence matrix of directed hypergraphs modeled on a knowledge base. We propose an interpretable KBQA model based on the hyperbolic directed hypergraph convolutional neural network named HDH-GCN which can update relation semantic information hop-by-hop and pays attention to different relations at different hops. The model can improve the accuracy of the multi-hop knowledge base question-answering task, and has application value in text question answering, human–computer interactions and other fields. Extensive experiments on benchmarks—PQL, MetaQA—demonstrate the effectiveness and universality of our HDH-GCN model, leading to state-of-the-art performance.

Knowledge Extraction and Quality Inspection of Chinese Petrographic Description Texts with Complex Entities and Relations Using Machine Reading and Knowledge Graph: A Preliminary Research Study

(1) Background: Geological surveying is undergoing a digital transformation process towards the adoption of intelligent methods in China. Cognitive intelligence methods, such as those based on knowledge graphs and machine reading, have made progress in many domains and also provide a technical basis for quality detection in unstructured lithographic description texts. (2) Methods: First, the named entities and the relations of the domain-specific knowledge graph of petrography were defined based on the petrographic theory. Second, research was carried out based on a manually annotated corpus of petrographic description. The extraction of N-ary and single-entity overlapping relations and the separation of complex entities are key steps in this process. Third, a petrographic knowledge graph was formulated based on prior knowledge. Finally, the consistency between knowledge triples extracted from the corpus and the petrographic knowledge graph was calculated. The 1:50,000 sheet of Fengxiangyi located in the Dabie orogenic belt was selected for the empirical research. (3) Results: Using machine reading and the knowledge graph, petrographic knowledge can be extracted and the knowledge consistency calculation can quickly detect description errors about textures, structures and mineral components in petrographic description. (4) Conclusions: The proposed framework can be used to realise the intelligent inspection of petrographic knowledge with complex entities and relations and to improve the quality of petrographic description texts effectively.

TSPNet: Translation supervised prototype network via residual learning for multimodal social relation extraction

Multimodal social relation extraction requires sufficient features fusion to identify the relation between different targets. Compared with traditional multimodal social relation extraction, there are many semantic gap issues for the few-shot scenario task, such as insufficient across-modality assistance, lacking explicit supervision, and unbalanced relations. To address the above problems, a novel Translation Supervised Prototype Network (TSPNet) is proposed, which extracts all the features of knowledge triples, not just relation features. First, the triple-level unimodal encoder learns textual and visual representation of knowledge triples from the entire information via two-stream encoding. Second, the triple-level multimodal extractor obtains multimodal knowledge triples by employing the residual learner to build the triple-level interaction across modalities. Finally, the intra-triple translation supervised decoder predicts the few-shot relations based on a prototype network supervised with the intra-triple translation as an explicit constraint. Our model achieves SOTA performance on three challenging benchmark datasets for few-shot multimodal social relation extraction, and further analysis shows that our model is effective and owns a strong generalization ability to avoid bias.

DKPLM: Decomposable Knowledge-Enhanced Pre-trained Language Model for Natural Language Understanding

Knowledge-Enhanced Pre-trained Language Models (KEPLMs) are pre-trained models with relation triples injecting from knowledge graphs to improve language understanding abilities.Experiments show that our model outperforms other KEPLMs significantly over zero-shot knowledge probing tasks and multiple knowledge-aware language understanding tasks. To guarantee effective knowledge injection, previous studies integrate models with knowledge encoders for representing knowledge retrieved from knowledge graphs. The operations for knowledge retrieval and encoding bring significant computational burdens, restricting the usage of such models in real-world applications that require high inference speed. In this paper, we propose a novel KEPLM named DKPLM that decomposes knowledge injection process of the pre-trained language models in pre-training, fine-tuning and inference stages, which facilitates the applications of KEPLMs in real-world scenarios. Specifically, we first detect knowledge-aware long-tail entities as the target for knowledge injection, enhancing the KEPLMs' semantic understanding abilities and avoiding injecting redundant information. The embeddings of long-tail entities are replaced by ``pseudo token representations'' formed by relevant knowledge triples. We further design the relational knowledge decoding task for pre-training to force the models to truly understand the injected knowledge by relation triple reconstruction. Experiments show that our model outperforms other KEPLMs significantly over zero-shot knowledge probing tasks and multiple knowledge-aware language understanding tasks. We further show that DKPLM has a higher inference speed than other competing models due to the decomposing mechanism.