Knowledge Graph Representation Learning Research Articles

Knowledge graph representation learning with relation-guided aggregation and interaction

Knowledge graph representation learning (KGRL) aims to study the feature representations to solve the incompleteness of knowledge graphs (KGs). Recently, graph neural network (GNN) has achieved satisfactory performance in KGRL tasks. However, many GNN based KGRL approaches fail to capture the various semantic of entities, which may reduce the representation quality of embeddings. Furthermore, most of them treat relations equally when learning embeddings, which will lead to weakening of the specific semantic features. To alleviate the above problems, we propose a novel KGRL method (RGAI) with relation-guided aggregation (RGA) and relation-guided interaction (RGI). To capture rich attributes of entities, the RGA module generates multiple sub-structures according to relation types, and aggregates the semantic information among them. To mine specific semantic features and dig out the different contribution of relations, the RGI module calculates the importance of each relation to the central entity, and generates final entity features through aggregating representations of neighbors. Extensive experiments on standard KG datasets validate the effectiveness of the proposed innovations, and RGAI achieves state-of-the-art performance compared to existing approaches (e.g., compared to state-of-the-art approaches, RGAI improves Hits@10 from 0.548 to 0.562 on FB15K-237, Hits@3 from 0.561 to 0.582 on YAGO3-10, and Hits@1 from 0.449 to 0.455 on WN18RR).

Evolving Knowledge Graph Representation Learning with Multiple Attention Strategies for Citation Recommendation System

The growing number of publications in the field of artificial intelligence highlights the need for researchers to enhance their efficiency in searching for relevant articles. Most paper recommendation models either rely on simplistic citation relationships among papers or focus on content-based approaches, both of which overlook interactions within academic networks. To address the aforementioned problem, knowledge graph embedding (KGE) methods have been used for citation recommendations because recent research proves that graph representations can effectively improve recommendation model accuracy. However, academic networks are dynamic, leading to changes in the representations of users and items over time. The majority of KGE-based citation recommendations are primarily designed for static graphs, thus failing to capture the evolution of dynamic knowledge graph (DKG) structures. To address these challenges, we introduced the evolving knowledge graph embedding (EKGE) method. In this methodology, evolving knowledge graphs are input into time-series models to learn the patterns of structural evolution. The model has the capability to generate embeddings for each entity at various time points, thereby overcoming limitation of static models that require retraining to acquire embeddings at each specific time point. To enhance the efficiency of feature extraction, we employed a multiple attention strategy. This helped the model find recommendation lists that are closely related to a user’s needs, leading to improved recommendation accuracy. Various experiments conducted on a citation recommendation dataset revealed that the EKGE model exhibits a 1.13% increase in prediction accuracy compared to other KGE methods. Moreover, the model’s accuracy can be further increased by an additional 0.84% through the incorporation of an attention mechanism.

MvTuckER: Multi-view knowledge graphs representation learning based on tensor tucker model

Recently, more and more multi-view knowledge graphs with various important attributes are being constructed and applied, such as temporal information, geolocation, commonsense knowledge, multi-language and multi-modal. Knowledge graphs (KGs) have been widely applied in question answering systems, drug discovery, information retrieval, and so on. Knowledge graph representation learning (KGRL) is emerging as the most effective approach to tasks such as knowledge graph completion, entity alignment, knowledge reasoning and knowledge query. However, most previous work on KGs and KGRL are mainly designed for traditional KGs and only consider single-view, and the KGRL methods for special multi-view KGs faces the challenges of lacking a unified framework, insufficient interactive information and underutilizing important attributes. To bridge this gap, in this paper, we proposes a multi-view knowledge graphs (e.g., temporal KGs) representation learning method based on tensor tucker model (MvTuckER), which models the multi-view KGs from tuples to a nth-order binary tensor, and adopts the tensor n-mode product to capture complete interaction between and within different views. We also introduce low-rank and sparse approximations of core tensors to balance expressivity and complexity of MvTuckER. Moreover, we theoretically illustrate that our model is fully expressive for modeling multi-view KGs, discuss the working mechanism of MvTuckER from the perspective of logical operation, and explain how to migrate and extend our method to other types of views. We conducted extensive experiments on three multi-view knowledge graphs, and obtain 4.7%, 2.7%, and 5.3% effective improvement in Hit@1 respectively, which proves that the proposed MvTuckER can achieve state-of-the-art performance.

Relphormer: Relational Graph Transformer for Knowledge Graph Representations

Transformers have achieved remarkable performance in widespread fields, including natural language processing, computer vision and graph mining. However, vanilla Transformer architectures have not yielded promising improvements in the Knowledge Graph (KG) representations, where the translational distance paradigm dominates this area. Note that vanilla Transformer architectures struggle to capture the intrinsically heterogeneous structural and semantic information of knowledge graphs. To this end, we propose a new variant of Transformer for knowledge graph representations dubbed Relphormer. Specifically, we introduce Triple2Seq which can dynamically sample contextualized sub-graph sequences as the input to alleviate the heterogeneity issue. We propose a novel structure-enhanced self-attention mechanism to encode the relational information and keep the semantic information within entities and relations. Moreover, we utilize masked knowledge modeling for general knowledge graph representation learning, which can be applied to various KG-based tasks including knowledge graph completion, question answering, and recommendation. Experimental results on six datasets show that Relphormer can obtain better performance compared with baselines.22Code is available in https://github.com/zjunlp/Relphormer.

OERL: Enhanced Representation Learning via Open Knowledge Graphs

The sparseness and incompleteness of knowledge graphs (KGs) trigger considerable interest in enhancing the representation learning with external corpora. However, the difficulty of aligning entities and relations with external corpora leads to inferior performance improvement. Open knowledge graphs (OKGs) consist of entity-mentions and relation-mentions that are represented by noncanonicalized freeform phrases, which generally do not rely on the specification of ontology schema. The roughness of the nonontological construction method leads to a specific characteristic of OKGs: diversity, where multiple entity-mentions (or relation-mentions) have the same meaning but different expressions. The diversity of OKGs can provide potential textual and structural features for the representation learning of KGs. We speculate that leveraging OKGs to enhance the representation learning of KGs can be more effective than using pure text or pure structure corpora. In this paper, we propose a new <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">OERL</b> , <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O</b> pen knowledge graph <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">E</b> nhanced <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</b> epresentation <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</b> earning of KGs. OERL automatically extracts textual and structural connections between KGs and OKGs, models and transfers refined profitable features to enhance the representation learning of KGs. The strong performance improvement and exhaustive experimental analysis prove the superiority of OERL over state-of-the-art baselines.

Contrastive Multi-Modal Knowledge Graph Representation Learning

Representation learning of knowledge graphs (KGs) aims to embed both entities and relations as vectors in a continuous low-dimensional space, which has facilitated various applications such as link prediction and entity retrieval. Most existing KG embedding methods focus on modeling the structured fact triples independently and ignore the multi-type relations among triples as well as the variety of data types (e.g., texts and images) associated with entities in KGs, and thus fail to capture the complex and multi-modal information that is inherently inside the entity-relation triples. In this paper, we propose a novel approach for knowledge graph embedding named Contrastive Multi-modal Graph Neural Network (CMGNN), which can encapsulate comprehensive features from multi-modal content descriptions of entities and high-order connectivity structures. Specifically, CMGNN first learns entity embeddings from multi-modal content and then contrasts encodings from multi-relational local neighbors and high-order connectivities to obtain latent representations of entities and relations simultaneously. Experimental results demonstrate that CMGNN can effectively model the multi-modalities and multi-type structures in KGs, and significantly outperforms existing state-of-the-art methods on benchmark datasets for the tasks of link prediction and entity classification.

Knowledge Graph Representation Learning based on Entity Semantic Distance Classification

Knowledge Graph Representation Learning based on Entity Semantic Distance Classification

Infer the missing facts of D3FEND using knowledge graph representation learning

PurposeMITRE and the National Security Agency cooperatively developed and maintained a D3FEND knowledge graph (KG). It provides concepts as an entity from the cybersecurity countermeasure domain, such as dynamic, emulated and file analysis. Those entities are linked by applying relationships such as analyze, may_contains and encrypt. A fundamental challenge for collaborative designers is to encode knowledge and efficiently interrelate the cyber-domain facts generated daily. However, the designers manually update the graph contents with new or missing facts to enrich the knowledge. This paper aims to propose an automated approach to predict the missing facts using the link prediction task, leveraging embedding as representation learning.Design/methodology/approachD3FEND is available in the resource description framework (RDF) format. In the preprocessing step, the facts in RDF format converted to subject–predicate–object triplet format contain 5,967 entities and 98 relationship types. Progressive distance-based, bilinear and convolutional embedding models are applied to learn the embeddings of entities and relations. This study presents a link prediction task to infer missing facts using learned embeddings.FindingsExperimental results show that the translational model performs well on high-rank results, whereas the bilinear model is superior in capturing the latent semantics of complex relationship types. However, the convolutional model outperforms 44% of the true facts and achieves a 3% improvement in results compared to other models.Research limitations/implicationsDespite the success of embedding models to enrich D3FEND using link prediction under the supervised learning setup, it has some limitations, such as not capturing diversity and hierarchies of relations. The average node degree of D3FEND KG is 16.85, with 12% of entities having a node degree less than 2, especially there are many entities or relations with few or no observed links. This results in sparsity and data imbalance, which affect the model performance even after increasing the embedding vector size. Moreover, KG embedding models consider existing entities and relations and may not incorporate external or contextual information such as textual descriptions, temporal dynamics or domain knowledge, which can enhance the link prediction performance.Practical implicationsLink prediction in the D3FEND KG can benefit cybersecurity countermeasure strategies in several ways, such as it can help to identify gaps or weaknesses in the existing defensive methods and suggest possible ways to improve or augment them; it can help to compare and contrast different defensive methods and understand their trade-offs and synergies; it can help to discover novel or emerging defensive methods by inferring new relations from existing data or external sources; and it can help to generate recommendations or guidance for selecting or deploying appropriate defensive methods based on the characteristics and objectives of the system or network.Originality/valueThe representation learning approach helps to reduce incompleteness using a link prediction that infers possible missing facts by using the existing entities and relations of D3FEND.

A patent recommendation method based on KG representation learning

The number of patents is constantly increasing, providing an invaluable knowledge base. But retrieving this knowledge becomes ever more challenging with the increase in data. In this context, this study proposes a patent recommendation method based on knowledge graph (KG) representation. Although KG-based recommendation systems received broad attention in the literature, there is still a lack of literature that explores how it can be used for patent recommendations. First, and in terms of KG construction, the existing methods encounter issues when it comes to simultaneously accounting for accuracy and recall rate. This study proposes a combination of rule-based and deep learning methods for KG construction to solve this problem. Second, existing methods have performance and semantic association limitations when learning KG representations. This study therefore uses high-performance graph neural networks for encoding and models the semantic association limitations in graphs using traditional KG representation learning methods. Finally, existing recommendation systems usually focus on the semantic correlation between interaction items in recommendations, ignoring their direct relationships. This study therefore applies the KG representation learning method for the patent recommendation task, using KG representation learning to model the semantic association and constraints in the product–patent heterogeneous graph. Experiments based on a practical case example demonstrate that the proposed method outperforms baseline methods, and that it can guide innovative design processes. Findings have important implications for research and practice.

Research on Personalized Video Matching Algorithm Based on Implicit Feature Transfer and PTransE

With the rapid development of internet plus education, it is increasingly important to quickly and accurately match personalized videos for learners from massive learning videos. However, the existing resource recommendation methods have two problems. On the one hand, they do not make full use of the implicit interaction data of learners in the learning process, i.e., they rarely reflect the comprehensive consideration of learners' interest preferences and cognitive level when describing learners. On the other hand, they are mostly based on collaborative filtering algorithms, using the similarity between learners or videos to recommend, ignoring the impact of semantic relations between videos on the recommendation results. In view of this, we proposed a personalized learning video matching method (IFT‐PTransE) based on heterogeneous feature data transfer and knowledge reasoning from two aspects of learning behavior data mining and knowledge graph representation learning. In this method, firstly, a learner model is constructed. By analyzing and quantifying various implicit interaction behavior data of learners, the sparse video scoring matrix is filled as auxiliary data to complete the transfer of target scoring data. Secondly, the semantic close relationship between videos is introduced. PTransE algorithm is used to mine the multi‐path relationship between entities. All entity relationships are embedded into the low dimensional vector space, so that the semantic similarity between videos can be calculated using the distance between vectors. Finally, video score similarity and semantic similarity between frequencies are fused. video sorting is performed based on the improved collaborative filtering algorithm and then recommend the top N videos to the students. Through simulation and analysis, the effectiveness of this method in personalized video matching is proved. This method makes up for the shortcomings of collaborative filtering algorithm in using implicit information, enhances the recommendation effect at the semantic level, and solves the data sparse and cold start problems to a certain extent. © 2023 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

Representation learning of knowledge graphs with correlation-based methods

Knowledge graphs (KGs) are intrinsically incomplete. Knowledge representation learning methods deal with embedding components of a knowledge graph (i.e., entities and relations) into a low-dimensional vector space with the aim of predicting the missing links. The KG Embedding models use gradient descent-based optimization techniques to optimize their parameters, but they suffer from late convergence, making them inefficient for dynamic knowledge graphs. Moreover, generating and modeling negative samples has a great impact on the accuracy of the prediction model.To address these challenges, we propose a correlation-based knowledge representation learning (CKRL) method. We utilize three semantic spaces, namely entity, relation, and canonical spaces. The parameter values of these spaces, including transformation matrices and embedding of KG components, are determined by defining four optimization problems. Adapting least-squares and Lagrange multiplier techniques, these optimization problems are solved by time-efficient direct methods. To model positive and negative samples, we use two distinct relation spaces. Negative samples are generated by defining the concept of relational patterns used to measure the similarity between positive and negative samples. Evaluation results on link prediction task and training time analysis demonstrate the superiority of the proposed method in comparison with the state-of-the-art methods.

Knowledge graph-based recommendation system enhanced by neural collaborative filtering and knowledge graph embedding

Recommendation systems are an important and undeniable part of modern systems and applications. Recommending items and users to the users that are likely to buy or interact with them is a modern solution for AI-based applications. In this article, a novel architecture is used with the utilization of pre-trained knowledge graph embeddings of different approaches. The proposed architecture consists of several stages that have various advantages. In the first step of the proposed method, a knowledge graph from data is created, since multi-hop neighbors in this graph address the ambiguity and redundancy problems. Then knowledge graph representation learning techniques are used to learn low-dimensional vector representations for knowledge graph components. In the following a neural collaborative filtering framework is used which benefits from no extra weights on layers. It is only dependent on matrix operations. Learning over these operations uses the pre-trained embeddings, and fine-tune them. Evaluation metrics show that the proposed method is superior in over other state-of-the-art approaches. According to the experimental results, the criteria of recall, precision, and F1-score have been improved, on average by 3.87%, 2.42%, and 6.05%, respectively.

A Survey on Multimodal Knowledge Graphs: Construction, Completion and Applications

As an essential part of artificial intelligence, a knowledge graph describes the real-world entities, concepts and their various semantic relationships in a structured way and has been gradually popularized in a variety practical scenarios. The majority of existing knowledge graphs mainly concentrate on organizing and managing textual knowledge in a structured representation, while paying little attention to the multimodal resources (e.g., pictures and videos), which can serve as the foundation for the machine perception of a real-world data scenario. To this end, in this survey, we comprehensively review the related advances of multimodal knowledge graphs, covering multimodal knowledge graph construction, completion and typical applications. For construction, we outline the methods of named entity recognition, relation extraction and event extraction. For completion, we discuss the multimodal knowledge graph representation learning and entity linking. Finally, the mainstream applications of multimodal knowledge graphs in miscellaneous domains are summarized.

Knowledge graph representation learning with simplifying hierarchical feature propagation

Graph neural networks (GNN) have emerged as a new state-of-the-art for learning knowledge graph representations. Although they have shown impressive performance in recent studies, how to efficiently and effectively aggregate neighboring features is not well designed. To tackle this challenge, we propose the simplifying heterogeneous graph neural network (SHGNet), a generic framework that discards the two standard operations in GNN, including the transformation matrix and nonlinear activation. SHGNet, in particular, adopts only the essential component of neighborhood aggregation in GNN and incorporates relation features into feature propagation. Furthermore, to capture complex structures, SHGNet utilizes a hierarchical aggregation architecture, including node aggregation and relation weighting. Thus, the proposed model can treat each relation differently and selectively aggregate informative features. SHGNet has been evaluated for link prediction tasks on three real-world benchmark datasets. The experimental results show that SHGNet significantly promotes efficiency while maintaining superior performance, outperforming all the existing models in 3 out of 4 metrics on NELL-995 and in 4 out of 4 metrics on FB15k-237 dataset.

Knowledge graph representation learning model based on meta-information and logical rule enhancements

Existing knowledge graph representation learning (KGRL) models rely on explicit semantic information of triple structure and cannot fully mine the implicit semantic information in the knowledge graph (KG). Aiming to improve KGRL model performance and accuracy, making up for the disadvantages of existing research, we propose Melo (Meta-information and Logical rules), a novel KGRL model that leverages meta-information and logical rules of entities and relations.Melo first utilizes neighborhood structures of entities to obtain meta-information and ontological information, then it mines logical rules from the KG to infer high-confidence triples and expand the training samples. Finally, Melo realizes accurate and reliable representations of entities and relations with help of meta-, logical, and triple structure information. Experimental results on regular and sparse datasets show its enhanced performance when compared with baselines in terms of multiple evaluation metrics. Visualization methods are also utilized to demonstrate how meta-information, logical rules, and triple structure mutually and separately enhance training.

Knowledge Graph Representation Learning Based on Automatic Network Search for Link Prediction

Link prediction, also known as Knowledge Graph Completion (KGC), is the common task in Knowledge Graphs (KGs) to predict missing connections between entities. Most existing methods focus on designing shallow, scalable models, which have less expressive than deep, multi-layer models. Furthermore, most operations like addition, matrix multiplications or factorization are handcrafted based on a few known relation patterns in several well-known datasets, such as FB15k, WN18, etc. However, due to the diversity and complex nature of real-world data distribution, it is inherently difficult to preset all latent patterns. To address this issue, we propose KGE-ANS, a novel knowledge graph embedding framework for general link prediction tasks using automatic network search. KGE-ANS can learn a deep, multi-layer effective architecture to adapt to different datasets through neural architecture search. In addition, the general search space we designed is tailored for KG tasks. We perform extensive experiments on benchmark datasets and the dataset constructed in this paper. The results show that our KGE-ANS outperforms several state-of-the-art methods, especially on these datasets with complex relation patterns.

Companion Animal Disease Diagnostics Based on Literal-Aware Medical Knowledge Graph Representation Learning

Companion Animal Disease Diagnostics Based on Literal-Aware Medical Knowledge Graph Representation Learning

Comprehensive Analysis of Knowledge Graph Embedding Techniques Benchmarked on Link Prediction

In knowledge graph representation learning, link prediction is among the most popular and influential tasks. Its surge in popularity has resulted in a panoply of orthogonal embedding-based methods projecting entities and relations into low-dimensional continuous vectors. To further enrich the research space, the community witnessed a prolific development of evaluation benchmarks with a variety of structures and domains. Therefore, researchers and practitioners face an unprecedented challenge in effectively identifying the best solution to their needs. To this end, we propose the most comprehensive and up-to-date study to systematically assess the effectiveness and efficiency of embedding models for knowledge graph completion. We compare 13 models on six datasets with different sizes, domains, and relational properties, covering translational, semantic matching, and neural network-based encoders. A fine-grained evaluation is conducted to compare each technique head-to-head in terms of standard metrics, training and evaluation times, memory consumption, carbon footprint, and space geometry. Our results demonstrate the high dependence between performance and graph types, identifying the best options for each scenario. Among all the encoding strategies, the new generation of translational models emerges as the most promising, bringing out the best and most consistent results across all the datasets and evaluation criteria.

UniSKGRep: A unified representation learning framework of social network and knowledge graph

The human-oriented applications aim to exploit behaviors of people, which impose challenges on user modeling of integrating social network (SN) with knowledge graph (KG), and jointly analyzing two types of graph data. However, existing graph representation learning methods merely represent one of two graphs alone, and hence are unable to comprehensively consider features of both SN and KG with profiling the correlation between them, resulting in unsatisfied performance in downstream tasks. Considering the diverse gap of features and the difficulty of associating of the two graph data, we introduce a Unified Social Knowledge Graph Representation learning framework (UniSKGRep), with the goal to leverage the multi-view information inherent in the SN and KG for improving the downstream tasks of user modeling. To the best of our knowledge, we are the first to present a unified representation learning framework for SN and KG. Concretely, the SN and KG are organized as the Social Knowledge Graph (SKG), a unified representation of SN and KG. For the representation learning of SKG, first, two separate encoders in the Intra-graph model capture both the social-view and knowledge-view in two embedding spaces, respectively. Then the Inter-graph model is learned to associate the two separate spaces via bridging the semantics of overlapping node pairs. In addition, the overlapping node enhancement module is designed to effectively align two spaces with the consideration of a relatively small number of overlapping nodes. The two spaces are gradually unified by continuously iterating the joint training procedure. Extensive experiments on two real-world SKG datasets have proved the effectiveness of UniSKGRep in yielding general and substantial performance improvement compared with the strong baselines in various downstream tasks.

Leveraging Multi-Modal Information for Cross-Lingual Entity Matching across Knowledge Graphs

In recent years, the scale of knowledge graphs and the number of entities have grown rapidly. Entity matching across different knowledge graphs has become an urgent problem to be solved for knowledge fusion. With the importance of entity matching being increasingly evident, the use of representation learning technologies to find matched entities has attracted extensive attention due to the computability of vector representations. However, existing studies on representation learning technologies cannot make full use of knowledge graph relevant multi-modal information. In this paper, we propose a new cross-lingual entity matching method (called CLEM) with knowledge graph representation learning on rich multi-modal information. The core is the multi-view intact space learning method to integrate embeddings of multi-modal information for matching entities. Experimental results on cross-lingual datasets show the superiority and competitiveness of our proposed method.