Knowledge Graph Embedding Methods Research Articles

The RDF2vec family of knowledge graph embedding methods

Knowledge graph embeddings represent a group of machine learning techniques which project entities and relations of a knowledge graph to continuous vector spaces. RDF2vec is a scalable embedding approach rooted in the combination of random walks with a language model. It has been successfully used in various applications. Recently, multiple variants to the RDF2vec approach have been proposed, introducing variations both on the walk generation and on the language modeling side. The combination of those different approaches has lead to an increasing family of RDF2vec variants. In this paper, we evaluate a total of twelve RDF2vec variants on a comprehensive set of benchmark models, and compare them to seven existing knowledge graph embedding methods from the family of link prediction approaches. Besides the established GEval benchmark introducing various downstream machine learning tasks on the DBpedia knowledge graph, we also use the new DLCC (Description Logic Class Constructors) benchmark consisting of two gold standards, one based on DBpedia, and one based on synthetically generated graphs. The latter allows for analyzing which ontological patterns in a knowledge graph can actually be learned by different embedding. With this evaluation, we observe that certain tailored RDF2vec variants can lead to improved performance on different downstream tasks, given the nature of the underlying problem, and that they, in particular, have a different behavior in modeling similarity and relatedness. The findings can be used to provide guidance in selecting a particular RDF2vec method for a given task.

Enhancing Knowledge Graph Embedding with Hierarchical Self-Attention and Graph Neural Network Techniques for Drug-Drug Interaction Prediction in Virtual Reality Environments

In biomedicine, the critical task is to decode Drug–Drug Interactions (DDIs) from complex biomedical texts. The scientific community employs Knowledge Graph Embedding (KGE) methods, enhanced with advanced neural network technologies, including capsule networks. However, existing methodologies primarily focus on the structural details of individual entities or relations within Biomedical Knowledge Graphs (BioKGs), overlooking the overall structural context of BioKGs, molecular structures, positional features of drug pairs, and their critical Relational Mapping Properties. To tackle the challenges identified, this study presents HSTrHouse an innovative hierarchical self-attention BioKGs embedding framework. This architecture integrates self-attention mechanisms with advanced neural network technologies, including Convolutional Neural Network (CNN) and Graph Neural Network (GNN), for enhanced computational modeling in biomedical contexts. The model bifurcates the BioKGs into entity and relation layers for structural analysis. It employs self-attention across these layers, utilizing PubMedBERT and CNN for position feature extraction, and a GNN for drug pair molecular structure analysis. Then, we connect the position and molecular structure features to integrate them into the self-attention calculation of entity and relation. After that, the output of the self-attention layer is combined with the connected vectors of the position feature and molecular structure feature to obtain the final representation vector, and finally, to model the Relational Mapping Properties (RMPs), the representation vector is embedded into the complex vector space using Householder projections to obtain the BioKGs model. The paper validates HSTrHouse’s efficacy by comparing it with advanced models on three standard BioKGs for DDIs research.

ConvTKG: A query-aware convolutional neural network-based embedding model for temporal knowledge graph completion

Temporal knowledge graphs (TKGs) are still far from completeness although they have benefited many artificial intelligence applications successfully. To alleviate the incompleteness of TKGs, temporal knowledge graph completion (TKGC) aims to perform link prediction and reason new time-sensitive facts from existing facts. Among the various TKGC methods, temporal knowledge graph embedding (TKGE) methods represent entities, relations, and timestamps in low-dimensional spaces, which are known for their high reasoning performance and robust scalability. Some previous researches on TKGE aim to endow the embedding quadruple with the translational characteristic by modeling the global relationship in the embedding quadruple. However, the weights of same dimensional entries of the embedding quadruple are fixed and unique, which limits the ability to capture diverse global relationships. Besides, most TKGE models view timestamps as a kind of general feature and overlook their interrelations and correlations with the queries. Moreover, the distinct characteristics of an entity as the subject entity and the object entity of facts have not been fully leveraged. In this paper, we propose a query-aware embedding model based on the convolutional neural network (CNN) named ConvTKG to perform TKGE, thus tackling the TKGC task. We design a new temporal information encoder based on the Gated Recurrent Unit (GRU) and attention mechanism to learn the query-aware representation of temporal information. More importantly, we utilize a CNN-based decoder where multiple one-dimensional convolution kernels are operated on the matrix composed of entity embedding, relation embedding, and temporal representation to capture global relationships among them. In addition, we assign two independent vectors for each entity and take advantage of the inverse relation to allow them to be learned dependently. Experiments show that ConvTKG achieves better link prediction performance than previous state-of-the-art baseline methods for TKGC on three benchmark datasets: ICEWS14, ICEWS05-15, and GDELT.

Evolving Knowledge Graph Representation Learning with Multiple Attention Strategies for Citation Recommendation System

The growing number of publications in the field of artificial intelligence highlights the need for researchers to enhance their efficiency in searching for relevant articles. Most paper recommendation models either rely on simplistic citation relationships among papers or focus on content-based approaches, both of which overlook interactions within academic networks. To address the aforementioned problem, knowledge graph embedding (KGE) methods have been used for citation recommendations because recent research proves that graph representations can effectively improve recommendation model accuracy. However, academic networks are dynamic, leading to changes in the representations of users and items over time. The majority of KGE-based citation recommendations are primarily designed for static graphs, thus failing to capture the evolution of dynamic knowledge graph (DKG) structures. To address these challenges, we introduced the evolving knowledge graph embedding (EKGE) method. In this methodology, evolving knowledge graphs are input into time-series models to learn the patterns of structural evolution. The model has the capability to generate embeddings for each entity at various time points, thereby overcoming limitation of static models that require retraining to acquire embeddings at each specific time point. To enhance the efficiency of feature extraction, we employed a multiple attention strategy. This helped the model find recommendation lists that are closely related to a user’s needs, leading to improved recommendation accuracy. Various experiments conducted on a citation recommendation dataset revealed that the EKGE model exhibits a 1.13% increase in prediction accuracy compared to other KGE methods. Moreover, the model’s accuracy can be further increased by an additional 0.84% through the incorporation of an attention mechanism.

KDGene: knowledge graph completion for disease gene prediction using interactional tensor decomposition.

The accurate identification of disease-associated genes is crucial for understanding the molecular mechanisms underlying various diseases. Most current methods focus on constructing biological networks and utilizing machine learning, particularly deep learning, to identify disease genes. However, these methods overlook complex relations among entities in biological knowledge graphs. Such information has been successfully applied in other areas of life science research, demonstrating their effectiveness. Knowledge graph embedding methods can learn the semantic information of different relations within the knowledge graphs. Nonetheless, the performance of existing representation learning techniques, when applied to domain-specific biological data, remains suboptimal. To solve these problems, we construct a biological knowledge graph centered on diseases and genes, and develop an end-to-end knowledge graph completion framework for disease gene prediction using interactional tensor decomposition named KDGene. KDGene incorporates an interaction module that bridges entity and relation embeddings within tensor decomposition, aiming to improve the representation of semantically similar concepts in specific domains and enhance the ability to accurately predict disease genes. Experimental results show that KDGene significantly outperforms state-of-the-art algorithms, whether existing disease gene prediction methods or knowledge graph embedding methods for general domains. Moreover, the comprehensive biological analysis of the predicted results further validates KDGene's capability to accurately identify new candidate genes. This work proposes a scalable knowledge graph completion framework to identify disease candidate genes, from which the results are promising to provide valuable references for further wet experiments. Data and source codes are available at https://github.com/2020MEAI/KDGene.

Neuro-Symbolic Integration for Reasoning and Learning on Knowledge Graphs

The goal of this thesis is to address knowledge graph completion tasks using neuro-symbolic methods. Neuro-symbolic methods allow the joint utilization of symbolic information defined as meta-rules in ontologies and knowledge graph embedding methods that represent entities and relations of the graph in a low-dimensional vector space. This approach has the potential to improve the resolution of knowledge graph completion tasks in terms of reliability, interpretability, data-efficiency and robustness.

KGDM: A Diffusion Model to Capture Multiple Relation Semantics for Knowledge Graph Embedding

Knowledge graph embedding (KGE) is an efficient and scalable method for knowledge graph completion. However, most existing KGE methods suffer from the challenge of multiple relation semantics, which often degrades their performance. This is because most KGE methods learn fixed continuous vectors for entities (relations) and make deterministic entity predictions to complete the knowledge graph, which hardly captures multiple relation semantics. To tackle this issue, previous works try to learn complex probabilistic embeddings instead of fixed embeddings but suffer from heavy computational complexity. In contrast, this paper proposes a simple yet efficient framework namely the Knowledge Graph Diffusion Model (KGDM) to capture the multiple relation semantics in prediction. Its key idea is to cast the problem of entity prediction into conditional entity generation. Specifically, KGDM estimates the probabilistic distribution of target entities in prediction through Denoising Diffusion Probabilistic Models (DDPM). To bridge the gap between continuous diffusion models and discrete KGs, two learnable embedding functions are defined to map entities and relation to continuous vectors. To consider connectivity patterns of KGs, a Conditional Entity Denoiser model is introduced to generate target entities conditioned on given entities and relations. Extensive experiments demonstrate that KGDM significantly outperforms existing state-of-the-art methods in three benchmark datasets.

Towards Continual Knowledge Graph Embedding via Incremental Distillation

Traditional knowledge graph embedding (KGE) methods typically require preserving the entire knowledge graph (KG) with significant training costs when new knowledge emerges. To address this issue, the continual knowledge graph embedding (CKGE) task has been proposed to train the KGE model by learning emerging knowledge efficiently while simultaneously preserving decent old knowledge. However, the explicit graph structure in KGs, which is critical for the above goal, has been heavily ignored by existing CKGE methods. On the one hand, existing methods usually learn new triples in a random order, destroying the inner structure of new KGs. On the other hand, old triples are preserved with equal priority, failing to alleviate catastrophic forgetting effectively. In this paper, we propose a competitive method for CKGE based on incremental distillation (IncDE), which considers the full use of the explicit graph structure in KGs. First, to optimize the learning order, we introduce a hierarchical strategy, ranking new triples for layer-by-layer learning. By employing the inter- and intra-hierarchical orders together, new triples are grouped into layers based on the graph structure features. Secondly, to preserve the old knowledge effectively, we devise a novel incremental distillation mechanism, which facilitates the seamless transfer of entity representations from the previous layer to the next one, promoting old knowledge preservation. Finally, we adopt a two-stage training paradigm to avoid the over-corruption of old knowledge influenced by under-trained new knowledge. Experimental results demonstrate the superiority of IncDE over state-of-the-art baselines. Notably, the incremental distillation mechanism contributes to improvements of 0.2%-6.5% in the mean reciprocal rank (MRR) score. More exploratory experiments validate the effectiveness of IncDE in proficiently learning new knowledge while preserving old knowledge across all time steps.

Knowledge Graph Embedding: A Survey from the Perspective of Representation Spaces

Knowledge graph embedding (KGE) is an increasingly popular technique that aims to represent entities and relations of knowledge graphs into low-dimensional semantic spaces for a wide spectrum of applications such as link prediction, knowledge reasoning and knowledge completion. In this article, we provide a systematic review of existing KGE techniques based on representation spaces. Particularly, we build a fine-grained classification to categorise the models based on three mathematical perspectives of the representation spaces: (1) algebraic perspective, (2) geometric perspective and (3) analytical perspective. We introduce the rigorous definitions of fundamental mathematical spaces before diving into KGE models and their mathematical properties. We further discuss different KGE methods over the three categories, as well as summarise how spatial advantages work over different embedding needs. By collating the experimental results from downstream tasks, we also explore the advantages of mathematical space in different scenarios and the reasons behind them. We further state some promising research directions from a representation space perspective, with which we hope to inspire researchers to design their KGE models as well as their related applications with more consideration of their mathematical space properties.

Knowledge graph embedding based on dynamic adaptive atrous convolution and attention mechanism for link prediction

Knowledge graph embedding (KGE) is essential for various applications, particularly in link prediction and other downstream tasks. While existing convolutional neural network (CNN)-based methods have been effective, they face challenges in comprehensively capturing local and global contextual information from triplets. To address this challenge, we propose a KGE method based on a dynamic adaptive atrous convolution and attention mechanism (DTAE). In this study, we innovatively construct adaptive convolutional kernels derived from relation representations. Employing a parallel strategy, we apply a multidimensional attention mechanism across kernel space. This configuration allows our kernels to dynamically adapt and learn from different entity representations, ensuring more nuanced feature extraction. Moreover, a dynamic adaptive atrous convolutional network has been developed. This network leverages atrous convolution, efficiently broadening the receptive field to encompass global information while preserving essential details during the aggregation process. In addition, we enhance feature information by integrating a channel attention mechanism, further refining our model’s ability to discern and process relevant data. Via comprehensive experiments on five standard link prediction benchmark datasets, DTAE demonstrates superior performance over existing models. Compared to the best recent state-of-the-art models over the last three years, we achieve improvements across various metrics ranging from 0.4% to 14.4%.

SimRE: Simple contrastive learning with soft logical rule for knowledge graph embedding

Knowledge graphs serve as a pivotal framework for the structured representation of information regarding entities and relations. However, in the real world, these knowledge graphs are often incomplete and harboring missing facts. Knowledge graph completion (KGC) has emerged as a central research focus, entailing the automated prediction of these missing facts and garnering substantial scholarly attention in recent years. Text-based knowledge graph embedding methods have demonstrated considerable potential for tackling the challenges associated with KGC by employing pre-trained language models. However, their limitation lies in the lack of logical features, which constrains the efficacy of capturing intricate patterns within knowledge graphs. This paper proposed SimRE, a straightforward contrastive learning framework augmented with soft logic rules. SimRE introduces a self-supervised framework that leverages the input rule bodies to predict the corresponding rule heads through a contrastive objective. We introduced two rule sampling techniques to enhance the efficiency and accuracy of the model: in-batch rule negatives and pre-batch rule negatives. SimRE employs a simple method for integrating logical features with the text-based model. The experimental results on benchmark datasets demonstrate that the proposed approach outperforms state-of-the-art methods.

RoCS: Knowledge Graph Embedding Based on Joint Cosine Similarity

Knowledge graphs usually have many missing links, and predicting the relationships between entities has become a hot research topic in recent years. Knowledge graph embedding research maps entities and relations to a low-dimensional continuous space representation to predict links between entities. The present research shows that the key to the knowledge graph embedding approach is the design of scoring functions. According to the scoring function, knowledge graph embedding methods can be classified into dot product models and distance models. We find that the triple scores obtained using the dot product model or the distance model were unbounded, which leads to large variance. In this paper, we propose RotatE Cosine Similarity (RoCS), a method to compute the joint cosine similarity of complex vectors as a scoring function to make the triple scores bounded. Our approach combines the rotational properties of the complex vector embedding model RotatE to model complex relational patterns. The experimental results demonstrate that the newly introduced RoCS yields substantial enhancements compared to RotatE across various knowledge graph benchmarks, improving up to 4.0% in hits at 1 (Hits@1) on WN18RR and improving up to 3.3% in Hits@1 on FB15K-237. Meanwhile, our method achieves some new state-of-the-art (SOTA), including Hits@3 of 95.6%, Hits@10 of 96.4% on WN18, and mean reciprocal rank (MRR) of 48.9% and Hits@1 of 44.5% on WN18RR.

Multi-ontology embeddings approach on human-aligned multi-ontologies representation for gene-disease associations prediction

ObjectivesKnowledge graphs and ontologies in the biomedical domain provide rich contextual knowledge for a variety of challenges. Employing that for knowledge-driven NLP tasks such as gene-disease association prediction represents a promising way to increase the predictive power of a model. MethodsWe investigated the power of infusing the embedding of two aligned ontologies as prior knowledge to the NLP models. We evaluated the performance of different models on some large-scale gene-disease association datasets and compared it with a model without incorporating contextualized knowledge (BERT). ResultsThe experiments demonstrated that the knowledge-infused model slightly outperforms BERT by creating a small number of bridges. Thus, indicating that incorporating cross-references across ontologies can enhance the performance of base models without the need for more complex and costly training. However, further research is needed to explore the generalizability of the model. We expected that adding more bridges would bring further improvement based on the trend we observed in the experiments. In addition, the use of state-of-the-art knowledge graph embedding methods on a joint graph from connecting OGG and DOID with bridges also yielded promising results. ConclusionOur work shows that allowing language models to leverage structured knowledge from ontologies does come with clear advantages in the performance. Besides, the annotation stage brought out in this paper is constrained in reasonable complexity.

Attention-based exploitation and exploration strategy for multi-hop knowledge graph reasoning

Knowledge Graphs (KGs) typically suffer from incompleteness. A popular approach to solve this problem is multi-hop reasoning through Reinforcement Learning (RL) framework, which is an explainable and effective model to predict missing links in KGs. However, many previous RL-based models use the scoring function of the pre-trained Knowledge Graph Embedding (KGE) methods as the reward function, which will lead to the performance of the model be limited to the KGE methods. Moreover, the agent may reason a meaningless path if it cannot distinguish the different aspect of each entity in different triples. To solve both problems, we propose a multi-hop reasoning model named Ae2KGR, by applying two novel strategies: attention-based exploitation and attention-based exploration. The attention-based exploitation strategy incorporates historical and query information with the neighborhood of the current entity, and dynamically updates the current state during reasoning process to assign different semantic information to the entity to distinguish different triples. The attention-based exploration strategy designs a novel policy network and reward function to dynamically make decisions based on the constantly changing state. Extensive experiments on three standard datasets confirm the effectiveness of our innovations, and the performance of our proposed Ae2KGR is significantly improved compared to the state-of-the-art methods.

HPRE: Leveraging hierarchy-aware paired relation vectors for knowledge graph embedding

Knowledge graphs exhibit a typical hierarchical structure and find extensive applications in various artificial intelligence domains. However, large-scale knowledge graphs need to be completed, which limits the performance of knowledge graphs in downstream tasks. Knowledge graph embedding methods have emerged as a primary solution to enhance knowledge graph completeness. These methods aim to represent entities and relations as low-dimensional vectors, focusing on handling relation patterns and multi-relation types. Researchers need to pay more attention to the crucial feature of hierarchical relationships in real-world knowledge graphs. We propose a novel knowledge graph embedding model called Hierarchy-Aware Paired Relation Vectors Knowledge Graph Embedding (HPRE) to bridge this gap. By leveraging the power of 2D coordinates, HPRE adeptly model relation patterns, multi-relation types, and hierarchical features in the knowledge graph. Specifically, HPRE employs paired relation vectors to capture the distinct characteristics of head and tail entities, facilitating a better fit for relational patterns and multi-relation scenarios. Additionally, HPRE employs angular coordinates to differentiate entities at various levels of the hierarchy, effectively representing the hierarchical nature of the knowledge graph. The experimental results show that the HPRE model can effectively learn the hierarchical features of the knowledge graph and achieve state-of-the-art experimental results on multiple real-world datasets for the link prediction task.

Multi-Information Preprocessing Event Extraction With BiLSTM-CRF Attention for Academic Knowledge Graph Construction

Academic knowledge graph is an important application of knowledge graph in the vertical field of academia. At present, the construction of the academic knowledge graph is mainly completed by extracting published academic papers, authors, publications, and other information from related databases. However, academic information is not just information of published papers. Scholars’ academic activities include participation in academic conferences, visiting and making presentation, and so on. However, the above academic information is hidden in natural language texts and cannot be directly stored in academic knowledge graph. This article proposes an approach named construct-SCHOLAT knowledge graph to construct an academic event knowledge graph based on academic social network SCHOLAT. The construction framework mainly consists of two parts: data preprocessing and event extraction. In the data preprocessing, we propose a knowledge graph embedding method to represent scholars’ academic social feature. In the event extraction, we concatenate the preprocessed scholar vector with academic we-media blog text into the extraction model based on BiLSTM-CRF fused with attention mechanism. The extracted events are added to academic knowledge graph, and a public relationship exists between the event and the scholar. Compared to the previous methods, our framework has an excellent performance after experimental verification. To the best of our knowledge, this is the first study to use the scholar academic social information of the scholar who edited the text as the event extraction input information. In addition, we publish a Chinese event extraction dataset SCHOLAT academic event extraction.https://www.scholat.com/research/opendata The dataset includes academic we-media blog and the social behavior embedding vector of the scholar. All the data in this dataset are derived from the academic social network SCHOLAT.

A type-augmented knowledge graph embedding framework for knowledge graph completion

Knowledge graphs (KGs) are of great importance to many artificial intelligence applications, but they usually suffer from the incomplete problem. Knowledge graph embedding (KGE), which aims to represent entities and relations in low-dimensional continuous vector spaces, has been proved to be a promising approach for KG completion. Traditional KGE methods only concentrate on structured triples, while paying less attention to the type information of entities. In fact, incorporating entity types into embedding learning could further improve the performance of KG completion. To this end, we propose a universal Type-augmented Knowledge graph Embedding framework (TaKE) which could utilize type features to enhance any traditional KGE models. TaKE automatically captures type features under no explicit type information supervision. And by learning different type representations of each entity, TaKE could distinguish the diversity of types specific to distinct relations. We also design a new type-constrained negative sampling strategy to construct more effective negative samples for the training process. Extensive experiments on four datasets from three real-world KGs (Freebase, WordNet and YAGO) demonstrate the merits of our proposed framework. In particular, combining TaKE with the recent tensor factorization KGE model SimplE can achieve state-of-the-art performance on the KG completion task.

Research on a Knowledge Graph Embedding Method Based on Improved Convolutional Neural Networks for Hydraulic Engineering

In response to the shortcomings of existing knowledge graph embedding strategies, such as weak feature interaction and latent knowledge representation, a unique hydraulic knowledge graph embedding method is suggested. The proposed method incorporates spatial position features into the entity-relation embedding process, thereby enhancing the representation capability of latent knowledge. Furthermore, it utilizes a multi-layer convolutional neural network to fuse features at different levels, effectively capturing more abundant semantic information. Additionally, the method employs multi-scale dilated convolution kernels to capture rich explicit interaction features across different scales of space. In this study, the effectiveness of the proposed model was validated on the link prediction task. Experimental results demonstrated that, compared to the ConvE model, the proposed model achieved a significant improvement of 14.8% in terms of mean reciprocal rank (MRR) on public datasets. Additionally, the suggested model outperformed the ConvR model on the hydraulic dataset, leading to a 10.1% increase in MRR. The results indicate that the proposed approach exhibits good applicability and performance in the task of hydraulic knowledge graph complementation. This suggests that the method has the potential to offer significant assistance for knowledge discovery and application research in the field of hydraulics.

Knowledge graph embedding methods for entity alignment: experimental review

In recent years, we have witnessed the proliferation of knowledge graphs (KG) in various domains, aiming to support applications like question answering, recommendations, etc. A frequent task when integrating knowledge from different KGs is to find which subgraphs refer to the same real-world entity, a task largely known as the Entity Alignment. Recently, embedding methods have been used for entity alignment tasks, that learn a vector-space representation of entities which preserves their similarity in the original KGs. A wide variety of supervised, unsupervised, and semi-supervised methods have been proposed that exploit both factual (attribute based) and structural information (relation based) of entities in the KGs. Still, a quantitative assessment of their strengths and weaknesses in real-world KGs according to different performance metrics and KG characteristics is missing from the literature. In this work, we conduct the first meta-level analysis of popular embedding methods for entity alignment, based on a statistically sound methodology. Our analysis reveals statistically significant correlations of different embedding methods with various meta-features extracted by KGs and rank them in a statistically significant way according to their effectiveness across all real-world KGs of our testbed. Finally, we study interesting trade-offs in terms of methods’ effectiveness and efficiency.

Next POI Recommendation with Dynamic Graph and Explicit Dependency

Next Point-Of-Interest (POI) recommendation plays an important role in various location-based services. Its main objective is to predict the user's next interested POI based on her previous check-in information. Most existing methods directly use users' historical check-in trajectories to construct various graphs to assist sequential models to complete this task. However, as users' check-in data is extremely sparse, it is difficult to capture the potential relations between POIs by directly using these check-in data. To this end, we propose the Sequence-based Neighbour search and Prediction Model (SNPM) for next POI recommendation. In SNPM, the RotatE knowledge graph embedding and Eigenmap methods are used to extract POI relationships implied in check-in data, and build the POI similarity graph. Then, we enhance the model's generalized representations of POIs' general features by aggregating similar POIs. As the context is typically rich and valuable when making Next POI predictions, the sequence model selects which POIs to aggregate not only depends on the current state, but also needs to consider the previous POI sequence. Therefore, we construct a Sequence-based, Dynamic Neighbor Graph (SDNG) to find the similarity neighbourhood and develop a Multi-Step Dependency Prediction model (MSDP) inspired by RotatE, which explicitly leverage information from previous states. We evaluate the proposed model on two real-world datasets, and the experimental results show that the proposed method significantly outperforms existing state-of-the-art POI recommendation methods.