Knowledge Graph Embedding Models Research Articles

Complex graph convolutional network for link prediction in knowledge graphs

Knowledge graph (KG) embedding models map nodes and edges to fixed-length vectors and obtain the similarity of nodes as the output of a scoring function to predict missing links between nodes. KG embedding methods based on graph convolutional networks (GCNs) have recently gained significant attention due to their ability to add information of neighboring nodes into the nodes’ embeddings. However, existing GCNs are primarily based on real-valued embeddings, which have high distortion, particularly when modeling graphs with varying geometric structures. In this paper, we propose complex graph convolutional network (ComplexGCN), a novel extension of the standard GCNs in complex space to combine the expressiveness of complex geometry with GCNs for improving the representation quality of KG components. The proposed ComplexGCN comprises a set of complex graph convolutional layers and a complex scoring function based on PARATUCK2 decomposition: the former includes information of neighboring nodes into the nodes’ embeddings, while the latter leverages these embeddings to predict new links between nodes. The proposed model demonstrates enhanced performance compared to existing methods on the two recent standard link prediction datasets.

Task-driven knowledge graph filtering improves prioritizing drugs for repurposing

BackgroundDrug repurposing aims at finding new targets for already developed drugs. It becomes more relevant as the cost of discovering new drugs steadily increases. To find new potential targets for a drug, an abundance of methods and existing biomedical knowledge from different domains can be leveraged. Recently, knowledge graphs have emerged in the biomedical domain that integrate information about genes, drugs, diseases and other biological domains. Knowledge graphs can be used to predict new connections between compounds and diseases, leveraging the interconnected biomedical data around them. While real world use cases such as drug repurposing are only interested in one specific relation type, widely used knowledge graph embedding models simultaneously optimize over all relation types in the graph. This can lead the models to underfit the data that is most relevant for the desired relation type. For example, if we want to learn embeddings to predict links between compounds and diseases but almost the entirety of relations in the graph is incident to other pairs of entity types, then the resulting embeddings are likely not optimised to predict links between compounds and diseases. We propose a method that leverages domain knowledge in the form of metapaths and use them to filter two biomedical knowledge graphs (Hetionet and DRKG) for the purpose of improving performance on the prediction task of drug repurposing while simultaneously increasing computational efficiency.ResultsWe find that our method reduces the number of entities by 60% on Hetionet and 26% on DRKG, while leading to an improvement in prediction performance of up to 40.8% on Hetionet and 14.2% on DRKG, with an average improvement of 20.6% on Hetionet and 8.9% on DRKG. Additionally, prioritization of antiviral compounds for SARS CoV-2 improves after task-driven filtering is applied.ConclusionKnowledge graphs contain facts that are counter productive for specific tasks, in our case drug repurposing. We also demonstrate that these facts can be removed, resulting in an improved performance in that task and a more efficient learning process.

STonKGs: a sophisticated transformer trained on biomedical text and knowledge graphs.

MotivationThe majority of biomedical knowledge is stored in structured databases or as unstructured text in scientific publications. This vast amount of information has led to numerous machine learning-based biological applications using either text through natural language processing (NLP) or structured data through knowledge graph embedding models. However, representations based on a single modality are inherently limited.ResultsTo generate better representations of biological knowledge, we propose STonKGs, a Sophisticated Transformer trained on biomedical text and Knowledge Graphs (KGs). This multimodal Transformer uses combined input sequences of structured information from KGs and unstructured text data from biomedical literature to learn joint representations in a shared embedding space. First, we pre-trained STonKGs on a knowledge base assembled by the Integrated Network and Dynamical Reasoning Assembler consisting of millions of text-triple pairs extracted from biomedical literature by multiple NLP systems. Then, we benchmarked STonKGs against three baseline models trained on either one of the modalities (i.e. text or KG) across eight different classification tasks, each corresponding to a different biological application. Our results demonstrate that STonKGs outperforms both baselines, especially on the more challenging tasks with respect to the number of classes, improving upon the F1-score of the best baseline by up to 0.084 (i.e. from 0.881 to 0.965). Finally, our pre-trained model as well as the model architecture can be adapted to various other transfer learning applications.Availability and implementationWe make the source code and the Python package of STonKGs available at GitHub (https://github.com/stonkgs/stonkgs) and PyPI (https://pypi.org/project/stonkgs/). The pre-trained STonKGs models and the task-specific classification models are respectively available at https://huggingface.co/stonkgs/stonkgs-150k and https://zenodo.org/communities/stonkgs.Supplementary information Supplementary data are available at Bioinformatics online.

Language Model Guided Knowledge Graph Embeddings

Knowledge graph embedding models have become a popular approach for knowledge graph completion through predicting the plausibility of (potential) triples. This is performed by transforming the entities and relations of the knowledge graph into an embedding space. However, knowledge graphs often include further textual information stored in literal, which is ignored by such embedding models. As a consequence, the learning process stays limited to the structure and the connections between the entities, which has the potential to negatively influence the performance. We bridge this gap by leveraging the capabilities of pre-trained language models to include textual knowledge in the learning process of embedding models. This is achieved by introducing a new loss function that guides embedding models in measuring the likelihood of triples by taking such complementary knowledge into consideration. The proposed solution is a model-independent loss function that can be plugged into any knowledge graph embedding model. In this paper, Sentence-BERT and fastText are used as pre-trained language models from which the embeddings of the textual knowledge are obtained and injected into the loss function. The loss function contains a trainable slack variable that determines the degree to which the language models influence the plausibility of triples. Our experimental evaluation on six benchmarks, namely Nations, UMLS, WordNet, and three versions of CodEx confirms the advantage of using pre-trained language models for boosting the accuracy of knowledge graph embedding models. We showcase this by performing evaluations on top of the five well-known knowledge graph embedding models such as TransE, RotatE, ComplEx, DistMult, and QuatE. The results show an improvement in accuracy up to 9% on UMLS dataset for the Distmult model and 4.2% on the Nations dataset for the ComplEx model when they are guided by pre-trained language models. We additionally studied the effect of multiple factors such as the structure of the knowledge graphs and training steps and presented them as ablation studies.

Knowledge graph embedding by reflection transformation

Due to the incompleteness of knowledge graph, knowledge graph embedding(KGE) has become a key technique for automatically predict missing facts in knowledge graph. KGE aims to learn low dimensional representations for both relations and entities. Despite existing KGE models achieved state-of-the-art(SOTA) performances, modeling and inferring relation connectivity(such as symmetry/antisymmetry, inversion, and composition), as well as complex relations prediction(such as M-to-1, 1-to-M and M-to-M) still have great challenges. In this paper, we propose a new KGE model called ReflectE, which regards each relation as a normal vector of a relation-specific reflection hyperplane. Specifically, ReflectE regards the tail entity(or head entity) in a triple as the reflection of the head entity(or tail entity) on a relation-specific hyperplane. Therefore it can model symmetric and inverse relations by reflection transformation naturally. Furthermore, ReflectE models complex relations by learning a relation-specific dynamic reflection hyperplane. In order to evaluate the effectiveness of our proposed model ReflectE, we choose previous SOTA models as baselines and conduct link prediction task on three popular datasets. Experimental results show that, compared with conventional distance-based KGE models, ReflectE achieves SOTA results for link prediction.

Knowledge graph embedding and reasoning for real-time analytics support of chemical diagnosis from exposure symptoms

Chemical exposure accidents pose a risk of serious injury and property damage if the diagnosis or response is not properly performed after the initial discovery. Due to lack of research on the dynamically changing environment and detection of chemical substances considering symptoms, real-time knowledge services are required, such as rapid diagnosis of chemicals exposed at the accident site and the following early response. In this study, we propose an AI-based analysis system, Symptom-based Expert for Advanced Response to Chemical Hazards (SEARCH), for chemical substance diagnosis from exposure symptoms actively collected for real-time response and mitigation to hazardous material accidents. Knowledge is collected from chemical database such as WISER, PubChem etc., and integrated for the analytics of chemical exposure accidents and contact symptoms. We design and construct ontology and knowledge graph (KG) for 1001 major chemical substances. The built KG is verified using KG embedding models and the performance of each model is compared. The proposed system identifies the substance candidates through KG query and reasoning considering the exposure conditions. Using the symptom KG, the system SEARCH can provide the means to analyze real-time data from the field and transform it into insights and actions related to emergency response.

Structural context-based knowledge graph embedding for link prediction

Knowledge graph embedding, which aims to address the limitation of symbolic representation of knowledge, has become an effective method for many AI downstream tasks, such as relation extraction, question answering. Existing knowledge graph embedding models mainly consider triples individually, and ignore the structural information connected with other entities. However, the connectivity between entities not only provides explicit structural information represented in triples, but also embodies a lot of implicit structure information. In this paper, a new knowledge graph embedding model is proposed, which can capture both the information of relational structure-context and edge structure-context by two-interaction. In addition, in order to model complex relations, we define different score function for different relation types. Moreover, the four relation connectivity types in knowledge graph (i.e. symmetry/antisymmetry, inversion, and composition) also can be modeled and inferred by StructurE. We evaluate our StructurE for knowledge graph link prediction task. Benefiting from the structural context and the relation-type-specific score function, compared with conventional geometric transformation-based knowledge graph embedding models StructurE achieves state-of-the-art results for link prediction. Moreover, compared with GCN-based models StructurE also achieves state-of-the-art results on more challenging dataset WN18RR which contains more symmetric relations.

Parallel training of knowledge graph embedding models

Knowledge graph embedding (KGE) models represent the entities and relations of a knowledge graph (KG) using dense continuous representations called embeddings. KGE methods have recently gained traction for tasks such as knowledge graph completion and reasoning as well as to provide suitable entity representations for downstream learning tasks. While a large part of the available literature focuses on small KGs, a number of frameworks that are able to train KGE models for large-scale KGs by parallelization across multiple GPUs or machines have recently been proposed. So far, the benefits and drawbacks of the various parallelization techniques have not been studied comprehensively. In this paper, we report on an experimental study in which we presented, re-implemented in a common computational framework, investigated, and improved the available techniques. We found that the evaluation methodologies used in prior work are often not comparable and can be misleading, and that most of currently implemented training methods tend to have a negative impact on embedding quality. We propose a simple but effective variation of the stratification technique used by PyTorch BigGraph for mitigation. Moreover, basic random partitioning can be an effective or even the best-performing choice when combined with suitable sampling techniques. Ultimately, we found that efficient and effective parallel training of large-scale KGE models is indeed achievable but requires a careful choice of techniques.

A Novel Negative Sampling Based on Frequency of Relational Association Entities for Knowledge Graph Embedding

Knowledge graph embedding improves the performance of relation extraction and knowledge reasoning by encoding entities and relationships in low-dimensional semantic space. During training, negative samples are usually constructed by replacing the head/tail entity. And the different replacing relationships lead to different accuracy of the prediction results. This paper develops a negative triplets construction framework according to the frequency of relational association entities. The proposed construction framework can fully consider the quantitative of relations and entities in the dataset to assign the proportion of relation and entity replacement and the frequency of the entities associated with each relationship to set reasonable proportions for different relations. To verify the validity of the proposed construction framework, it is integrated into the state-of-the-art knowledge graph embedding models, such as TransE, TransH, DistMult, ComplEx, and Analogy. And both the evaluation criteria of relation prediction and entity prediction are used to evaluate the performance of link prediction more comprehensively. The experimental results on two commonly used datasets, WN18 and FB15K, show that the proposed method improves entity link and triplet classification accuracy, especially the accuracy of relational link prediction.

Identifying disease-gene associations using a convolutional neural network-based model by embedding a biological knowledge graph with entity descriptions.

Understanding the role of genes in human disease is of high importance. However, identifying genes associated with human diseases requires laborious experiments that involve considerable effort and time. Therefore, a computational approach to predict candidate genes related to complex diseases including cancer has been extensively studied. In this study, we propose a convolutional neural network-based knowledge graph-embedding model (KGED), which is based on a biological knowledge graph with entity descriptions to infer relationships between biological entities. As an application demonstration, we generated gene-interaction networks for each cancer type using gene-gene relationships inferred by KGED. We then analyzed the constructed gene networks using network centrality measures, including betweenness, closeness, degree, and eigenvector centrality metrics, to rank the central genes of the network and identify highly correlated cancer genes. Furthermore, we evaluated our proposed approach for prostate, breast, and lung cancers by comparing the performance with that of existing approaches. The KGED model showed improved performance in predicting cancer-related genes using the inferred gene-gene interactions. Thus, we conclude that gene-gene interactions inferred by KGED can be helpful for future research, such as that aimed at future research on pathogenic mechanisms of human diseases, and contribute to the field of disease treatment discovery.

Text-Graph Enhanced Knowledge Graph Representation Learning.

Knowledge Graphs (KGs) such as Freebase and YAGO have been widely adopted in a variety of NLP tasks. Representation learning of Knowledge Graphs (KGs) aims to map entities and relationships into a continuous low-dimensional vector space. Conventional KG embedding methods (such as TransE and ConvE) utilize only KG triplets and thus suffer from structure sparsity. Some recent works address this issue by incorporating auxiliary texts of entities, typically entity descriptions. However, these methods usually focus only on local consecutive word sequences, but seldom explicitly use global word co-occurrence information in a corpus. In this paper, we propose to model the whole auxiliary text corpus with a graph and present an end-to-end text-graph enhanced KG embedding model, named Teger. Specifically, we model the auxiliary texts with a heterogeneous entity-word graph (called text-graph), which entails both local and global semantic relationships among entities and words. We then apply graph convolutional networks to learn informative entity embeddings that aggregate high-order neighborhood information. These embeddings are further integrated with the KG triplet embeddings via a gating mechanism, thus enriching the KG representations and alleviating the inherent structure sparsity. Experiments on benchmark datasets show that our method significantly outperforms several state-of-the-art methods.

Learning hyperbolic attention-based embeddings for link prediction in knowledge graphs

Knowledge graph (KG) embedding methods aim to learn low-dimensional representations of entities and relations to predict new valid triples for KG completion. Most of the existing KG embedding models learn embeddings in Euclidean space, which cannot accurately capture hierarchical structures and complex properties of relations found in KGs. To this effect, a recent model MuRP, as a first hyperbolic method, learns the KG embeddings in hyperbolic space and outperforms existing Euclidean embedding models. However, MuRP treats the KG triples individually, and hence fails to capture the complex structural information inherent in the local vicinity of a node, leading to low-quality node embeddings. On the other hand, the recent hyperbolic graph neural network (HGNN) provides a way of learning high-quality hyperbolic node embeddings by capturing information from each node’s neighborhood. However, HGNN ignores the relation features and treats all neighboring nodes with equal importance. To this end, we propose RHGNN, which extends HGNN by including the relation features and performing a hyperbolic attention-based neighborhood aggregation. We then combine RHGNN with MuRP into a novel encoder–decoder hyperbolic embedding learning framework (which we call HyperGEL) for KG completion. RHGNN gathers information from the neighboring nodes to generate rich hyperbolic node embeddings, and MuRP uses these embeddings to predict new triples. Experimental results show the proposed framework’s effectiveness that consistently marks performance gains over several previous models on recent standard KG completion datasets.

Question-aware memory network for multi-hop question answering in human\u2013robot interaction

Knowledge graph question answering is an important technology in intelligent human–robot interaction, which aims at automatically giving answer to human natural language question with the given knowledge graph. For the multi-relation question with higher variety and complexity, the tokens of the question have different priority for the triples selection in the reasoning steps. Most existing models take the question as a whole and ignore the priority information in it. To solve this problem, we propose question-aware memory network for multi-hop question answering, named QA2MN, to update the attention on question timely in the reasoning process. In addition, we incorporate graph context information into knowledge graph embedding model to increase the ability to represent entities and relations. We use it to initialize the QA2MN model and fine-tune it in the training process. We evaluate QA2MN on PathQuestion and WorldCup2014, two representative datasets for complex multi-hop question answering. The result demonstrates that QA2MN achieves state-of-the-art Hits@1 accuracy on the two datasets, which validates the effectiveness of our model.

Knowledge graph embedding by relational and entity rotation

Knowledge graphs are typical large-scale multi-relational structures and useful for many artificial intelligence tasks. However, knowledge graphs often have missing facts, which limits the development of downstream tasks. To refine the knowledge graphs, knowledge graph embedding models have been developed. Knowledge graph embedding models aim to learn distributed representations for entities and relations and predict unknown triplets by scoring candidate triplets. Nevertheless, state-of-the-art works either aim to capture different relation patterns, or to model the multi-fold relations, and yet fail to consider these two aspects simultaneously. To fill this gap, in this paper, we propose a novel knowledge graph embedding model, MRotatE. It exploits triplet features from the perspective of relational and entity rotations, which can model and infer various relation patterns and handle with multi-fold relations at the same time. The experimental results demonstrate that MRotatE outperforms existing approaches and attains the state-of-the-art performance.

Improving Knowledge Graph Completion Using Soft Rules and Adversarial Learning

Knowledge graph is a useful resources and tools for describing entities and relationships in natural language processing tasks. However, the existing knowledge graph are incomplete. Therefore, knowledge graph completion technology has become a research hotspot in the field of artificial intelligence, but the traditional knowledge graph embedding method does not fully take into account the role of logic rules and the effect of false negative samples on knowledge embedding. Based on the logic rules of knowledge and the role of adversarial learning in knowledge embedding, we proposes a model to improve the completion of knowledge graph: soft Rules and graph adversarial learning (RUGA). Firstly, the traditional knowledge graph embedding model is trained as generator and discriminator by using adversarial learning method, and high-quality negative samples are obtained. Then these negative samples and the existing positive samples together constitute the label triple in the injection rule model. The whole model will benefit from both high-quality samples and logical rules. In addition, we evaluated the performance of link prediction task and triple classification task on Freebase and Yago datasets respectively. Finally, the experimental results show that the model can effectively improve the effect of knowledge graph completion.

Trans4E: Link prediction on scholarly knowledge graphs

The incompleteness of Knowledge Graphs (KGs) is a crucial issue affecting the quality of AI-based services. In the scholarly domain, KGs describing research publications typically lack important information, hindering our ability to analyse and predict research dynamics. In recent years, link prediction approaches based on Knowledge Graph Embedding models became the first aid for this issue. In this work, we present Trans4E, a novel embedding model that is particularly fit for KGs which include N to M relations with N ≫ M. This is typical for KGs that categorize a large number of entities (e.g., research articles, patents, persons) according to a relatively small set of categories. Trans4E was applied on two large-scale knowledge graphs, the Academia/Industry DynAmics (AIDA) and Microsoft Academic Graph (MAG), for completing the information about Fields of Study (e.g., ‘neural networks’, ‘machine learning’, ‘artificial intelligence’), and affiliation types (e.g., ‘education’, ‘company’, ‘government’), improving the scope and accuracy of the resulting data. We evaluated our approach against alternative solutions on AIDA, MAG, and four other benchmarks (FB15k, FB15k-237, WN18, and WN18RR). Trans4E outperforms the other models when using low embedding dimensions and obtains competitive results in high dimensions.

Rule-enhanced iterative complementation for knowledge graph reasoning

Knowledge graph (KG) reasoning aims to infer missing valid triples from observed triples, thereby improving the semantics of the whole KG. The general KG reasoning involves rule-based and embedding-based methods. The former can provide an interpretable reasoning process but has low efficiency, while the latter is the converse. Therefore, some hybrid methods have been proposed, but there are still two challenges: the completeness of rule learning and the determination of hidden triples. To address these challenges, this paper proposes a rule-enhanced iterative complementation (Rule-IC) method, which involves three components: rule learning, an embedding learner and a triple discriminator. Such an iterative process enriches the semantics of KG and further increases the completeness of rule learning to generate hidden triples. In order to precisely determine the validity of hidden triples, a multi-relational graph convolutional network (GCN) with attentive message passing is introduced as a triple discriminator. The embedding learner for KG reasoning and the GCN discriminator complement each other by valid hidden triples. In addition, the performance of these three components improves overall during the iterative process. Experimental results show that most evaluation metrics of Rule-IC are better than those of several baselines on four common KGs. Furthermore, it is scalable and can be extended to all KG embedding models theoretically.

TransET: Knowledge Graph Embedding with Entity Types

Knowledge graph embedding aims to embed entities and relations into low-dimensional vector spaces. Most existing methods only focus on triple facts in knowledge graphs. In addition, models based on translation or distance measurement cannot fully represent complex relations. As well-constructed prior knowledge, entity types can be employed to learn the representations of entities and relations. In this paper, we propose a novel knowledge graph embedding model named TransET, which takes advantage of entity types to learn more semantic features. More specifically, circle convolution based on the embeddings of entity and entity types is utilized to map head entity and tail entity to type-specific representations, then translation-based score function is used to learn the presentation triples. We evaluated our model on real-world datasets with two benchmark tasks of link prediction and triple classification. Experimental results demonstrate that it outperforms state-of-the-art models in most cases.

5* Knowledge Graph Embeddings with Projective Transformations

Performing link prediction using knowledge graph embedding models has become a popular approach for knowledge graph completion. Such models employ a transformation function that maps nodes via edges into a vector space in order to measure the likelihood of the links. While mapping the individual nodes, the structure of subgraphs is also transformed. Most of the embedding models designed in Euclidean geometry usually support a single transformation type -- often translation or rotation, which is suitable for learning on graphs with small differences in neighboring subgraphs. However, multi-relational knowledge graphs often include multiple subgraph structures in a neighborhood (e.g.~combinations of path and loop structures), which current embedding models do not capture well. To tackle this problem, we propose a novel KGE model 5*E in projective geometry, which supports multiple simultaneous transformations -- specifically inversion, reflection, translation, rotation, and homothety. The model has several favorable theoretical properties and subsumes the existing approaches. It outperforms them on most widely used link prediction benchmarks

Relation-based multi-type aware knowledge graph embedding

Knowledge graph (KG) embedding projects the graph into a low-dimensional space and preserves the graph information. An essential part of a KG is the ontology, which always is organized as a taxonomy tree, depicting the type (or multiple types) of each entity and the hierarchical relationships among these types. The importance of considering the ontology during KG embedding lies in its ability to provide side-information, improving the downstream applications’ accuracy (e.g., link prediction, entity alignment or recommendation). However, the ontology has yet to receive adequate attention during the KG embedding, especially for instances where each entity may belong to multiple types. This ontology-enhanced KG embedding’s main challenges are twofold: determining how to discover the relationships among these types and how to integrate them with the entities’ relationship network. Although it is common to see attention-based models used in KG embedding, they cannot settle the issues raised simultaneously. Only a single type is assigned to each entity and the correlation among types are ignored in those models, leading to information loss and encumbered downstream tasks. To overcome these challenges, we propose a composite multi-type aware KG embedding model, whose main components are a multi-type layer and entity embedding layer. We model it as a natural language processing task at the multi-type layer to discover each entity’s multi-type feature and automatically capture their correlations. Additionally, a relation-based attention mechanism is conducted at the entity embedding layer, which aggregates neighborhoods’ information and integrates the multi-type layer’s information through common entities of these two layers. Through extensive experiments on two real KGs, we demonstrate that, compared to several state-of-the-art baselines, our Multi-Type aware Embedding (MTE) model achieves substantial gain in both Mean Rank and Hit@N for the link prediction task and accuracy for multi-type classification.