Graph Embedding Model Research Articles

Lifelong Embedding Learning and Transfer for Growing Knowledge Graphs

Existing knowledge graph (KG) embedding models have primarily focused on static KGs. However, real-world KGs do not remain static, but rather evolve and grow in tandem with the development of KG applications. Consequently, new facts and previously unseen entities and relations continually emerge, necessitating an embedding model that can quickly learn and transfer new knowledge through growth. Motivated by this, we delve into an expanding field of KG embedding in this paper, i.e., lifelong KG embedding. We consider knowledge transfer and retention of the learning on growing snapshots of a KG without having to learn embeddings from scratch. The proposed model includes a masked KG autoencoder for embedding learning and update, with an embedding transfer strategy to inject the learned knowledge into the new entity and relation embeddings, and an embedding regularization method to avoid catastrophic forgetting. To investigate the impacts of different aspects of KG growth, we construct four datasets to evaluate the performance of lifelong KG embedding. Experimental results show that the proposed model outperforms the state-of-the-art inductive and lifelong embedding baselines.

A neuro-symbolic system over knowledge graphs for link prediction

Neuro-Symbolic Artificial Intelligence (AI) focuses on integrating symbolic and sub-symbolic systems to enhance the performance and explainability of predictive models. Symbolic and sub-symbolic approaches differ fundamentally in how they represent data and make use of data features to reach conclusions. Neuro-symbolic systems have recently received significant attention in the scientific community. However, despite efforts in neural-symbolic integration, symbolic processing can still be better exploited, mainly when these hybrid approaches are defined on top of knowledge graphs. This work is built on the statement that knowledge graphs can naturally represent the convergence between data and their contextual meaning (i.e., knowledge). We propose a hybrid system that resorts to symbolic reasoning, expressed as a deductive database, to augment the contextual meaning of entities in a knowledge graph, thus, improving the performance of link prediction implemented using knowledge graph embedding (KGE) models. An entity context is defined as the ego network of the entity in a knowledge graph. Given a link prediction task, the proposed approach deduces new RDF triples in the ego networks of the entities corresponding to the heads and tails of the prediction task on the knowledge graph (KG). Since knowledge graphs may be incomplete and sparse, the facts deduced by the symbolic system not only reduce sparsity but also make explicit meaningful relations among the entities that compose an entity ego network. As a proof of concept, our approach is applied over a KG for lung cancer to predict treatment effectiveness. The empirical results put the deduction power of deductive databases into perspective. They indicate that making explicit deduced relationships in the ego networks empowers all the studied KGE models to generate more accurate links.

LogicENN: A Neural Based Knowledge Graphs Embedding Model with Logical Rules.

Knowledge graph embedding models have gained significant attention in AI research. The aim of knowledge graph embedding is to embed the graphs into a vector space in which the structure of the graph is preserved. Recent works have shown that the inclusion of background knowledge, such as logical rules, can improve the performance of embeddings in downstream machine learning tasks. However, so far, most existing models do not allow the inclusion of rules. We address the challenge of including rules and present a new neural based embedding model (LogicENN). We prove that LogicENN can learn every ground truth of encoded rules in a knowledge graph. To the best of our knowledge, this has not been proved so far for the neural based family of embedding models. Moreover, we derive formulae for the inclusion of various rules, including (anti-)symmetric, inverse, irreflexive and transitive, implication, composition, equivalence, and negation. Our formulation allows avoiding grounding for implication and equivalence relations. Our experiments show that LogicENN outperforms the existing models in link prediction.

Employing Latent Categories of Entities for Knowledge Graph Embeddings With Contrastive Learning

Knowledge graph embedding aims to represent entities and relations as low-dimensional vectors, which is an effective way for robotics to learn and reason about semantic knowledge. It is crucial for knowledge graph embedding models to infer various relation patterns, such as symmetry/antisymmetry. However, most existing approaches ignore the latent semantic categories information of entities. For example, when robots use a faucet with relation <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">OperatesOn</i> , its tail entity category should be washable objects, such as apple, bowl, rather than non-washable objects, such as mobile phone, computer. To address such issues, we propose a novel and simple framework for knowledge graph embedding, which utilizes contrastive learning to cluster entities based on the constraints arising from relations, and enhances the discriminative ability for entities with the latent same category. Experimental results demonstrate the effectiveness of our proposed method on four standard knowledge graph completion benchmarks. It is noteworthy that our method can yield some new state-of-the-art results, achieving 88.2% MRR, 84.9% Hits@1 on the AI2Thor dataset, and 39.8% MRR, 30.0% Hits@1 on the FB15k-237 dataset.

Relationship Prediction in a Knowledge Graph Embedding Model of the Illicit Antiquities Trade

ABSTRACTThe transnational networks of the illicit and illegal antiquities trade are hard to perceive. We suggest representing the trade as a knowledge graph with multiple kinds of relationships that can be transformed by a neural architecture into a “knowledge graph embedding model.” The result is that the vectorization of the knowledge represented in the graph can be queried for missing “knowledge” of the trade by virtue of the various entities’ proximity in the multidimensional embedding space. In this article, we build a knowledge graph about the antiquities trade using a semantic annotation tool, drawing on the series of articles in the Trafficking Culture Project's online encyclopedia. We then use the AmpliGraph package, a series of tools for supervised machine learning (Costabello et al. 2019) to turn the graph into a knowledge graph embedding model. We query the model to predict new hypotheses and to cluster actors in the trade. The model suggests connections between actors and institutions hitherto unsuspected and not otherwise present in the original knowledge graph. This approach could hold enormous potential for illuminating the hidden corners of the illicit antiquities trade. The same method could be applied to other kinds of archaeological knowledge.

Graph Representation Learning and Its Applications: A Survey.

Graphs are data structures that effectively represent relational data in the real world. Graph representation learning is a significant task since it could facilitate various downstream tasks, such as node classification, link prediction, etc. Graph representation learning aims to map graph entities to low-dimensional vectors while preserving graph structure and entity relationships. Over the decades, many models have been proposed for graph representation learning. This paper aims to show a comprehensive picture of graph representation learning models, including traditional and state-of-the-art models on various graphs in different geometric spaces. First, we begin with five types of graph embedding models: graph kernels, matrix factorization models, shallow models, deep-learning models, and non-Euclidean models. In addition, we also discuss graph transformer models and Gaussian embedding models. Second, we present practical applications of graph embedding models, from constructing graphs for specific domains to applying models to solve tasks. Finally, we discuss challenges for existing models and future research directions in detail. As a result, this paper provides a structured overview of the diversity of graph embedding models.

A contrastive knowledge graph embedding model with hierarchical attention and dynamic completion

A contrastive knowledge graph embedding model with hierarchical attention and dynamic completion

Autonomic computing and incremental learning for the management of big services

AbstractRecent years have witnessed the emergence of big services, as a large‐scale big data‐centric service model, that resulted from the synergy between powerful computing paradigms (big data processing, service and cloud computing, Internet of Things, etc.). Big services are seen as a heterogeneous combination of physical and virtualized domain‐specific resources, with a huge volume of data and complex functionalities, all encapsulated and offered as services. This complexity of big services (composition units' heterogeneity, cross‐domain orientation, data massiveness), coupled with other environmental factors (cloud dynamicity, providers' policies, customer requirements) makes their management tasks beyond humans' capability. Therefore, endowing big service ecosystems with self‐adaptive behavior is a natural solution. To achieve this goal, this article models big services as autonomic computing systems, and structures their behavioral aspects (functional behavior, quality of service/data levels, management policies) as a multi‐view knowledge graph. To infer useful knowledge (e.g., conflicts between policies) for the autonomic big service's management tasks, we process the big service's knowledge graph (BSKG) via a graph neural network‐based graph embedding model. This latter is reinforced by an incremental learning method, that helps capturing the big services' frequent changes (e.g., QoS deviations, service failures, new policies), and drives autonomic managers to continuously update and enrich their knowledge w.r.t. the managed big service's current state. Finally, a flexible decision mechanism explores the BSKG structure and the latent knowledge, to locate and trigger the appropriate management policies, according to the big service's produced events.

Knowledge graph embedding model with attention-based high-low level features interaction convolutional network

Knowledge graphs are sizeable graph-structured knowledge with both abstract and concrete concepts in the form of entities and relations. Recently, convolutional neural networks have achieved outstanding results for more expressive representations of knowledge graphs. However, existing deep learning-based models exploit semantic information from single-level feature interaction, potentially limiting expressiveness. We propose a knowledge graph embedding model with an attention-based high-low level features interaction convolutional network called ConvHLE to alleviate this issue. This model effectively harvests richer semantic information and generates more expressive representations. Concretely, the multilayer convolutional neural network is utilized to fuse high-low level features. Then, features in fused feature maps interact with other informative neighbors through the criss-cross attention mechanism, which expands the receptive fields and boosts the quality of interactions. Finally, a plausibility score function is proposed for the evaluation of our model. The performance of ConvHLE is experimentally investigated on six benchmark datasets with individual characteristics. Extensive experimental results prove that ConvHLE learns more expressive and discriminative feature representations and has outperformed other state-of-the-art baselines over most metrics when addressing link prediction tasks. Comparing MRR and Hits@1 on FB15K-237, our model outperforms the baseline ConvE by 13.5% and 16.0%, respectively.

A collaborative central domain adaptation approach with multi-order graph embedding for bearing fault diagnosis under few-shot samples

Effective fault diagnosis is a prerequisite for ensuring the safe, stable and long-term operation of many rotating machinery. With the rapid development of measurement, sensor and computing technologies, measurement data presents a high-dimensional and massive distribution. This makes the valuable fault information in samples sparse. Moreover, industrial data can only present the distribution state of few-shot unlabeled information. In addition, the vibration signal of bearing faults contains noise interference, leading to poor stability and low efficiency of most models. In this study, we propose an approach for rolling bearing faults diagnosis under few-shot samples. It consists of a multi-order graph embedding stacked denoising auto encoder optimized by an improved sine–cosine​ algorithm (MGE-ISCA-SDAE) and a collaborative central domain adaptation (CCDA). First, a multi-order graph embedding model and an ISCA-based strategy are designed to improve the SDAE, thereby improving the feature extraction effect. To overcome the sparseness of valuable information, we design a CCDA model that learns the fault features using the labeled samples. Subsequently, it is transferred to the target domain of few-shot labeled samples for adaptation. Finally, the intelligent diagnosis is achieved under few-shot samples. We conduct experiments with four datasets. The results show that the MGE-ISCA-SDAE can extract the time–frequency high-level fault features. The CCDA model can transfer the fault samples well. When there are fewer fault samples, our approach has outstanding advantages.

A Quick Prototype for Assessing OpenIE Knowledge Graph-Based Question-Answering Systems

Due to the rapid growth of knowledge graphs (KG) as representational learning methods in recent years, question-answering approaches have received increasing attention from academia and industry. Question-answering systems use knowledge graphs to organize, navigate, search and connect knowledge entities. Managing such systems requires a thorough understanding of the underlying graph-oriented structures and, at the same time, an appropriate query language, such as SPARQL, to access relevant data. Natural language interfaces are needed to enable non-technical users to query ever more complex data. The paper proposes a question-answering approach to support end users in querying graph-oriented knowledge bases. The system pipeline is composed of two main modules: one is dedicated to translating a natural language query submitted by the user into a triple of the form &lt;subject, predicate, object&gt;, while the second module implements knowledge graph embedding (KGE) models, exploiting the previous module triple and retrieving the answer to the question. Our framework delivers a fast OpenIE-based knowledge extraction system and a graph-based answer prediction model for question-answering tasks. The system was designed by leveraging existing tools to accomplish a simple prototype for fast experimentation, especially across different knowledge domains, with the added benefit of reducing development time and costs. The experimental results confirm the effectiveness of the proposed system, which provides promising performance, as assessed at the module level. In particular, in some cases, the system outperforms the literature. Finally, a use case example shows the KG generated by user questions in a graphical interface provided by an ad-hoc designed web application.

GOGCN: Graph Convolutional Network on Gene Ontology for Functional Similarity Analysis of Genes.

The measurement of gene functional similarity plays a critical role in numerous biological applications, such as gene clustering, the construction of gene similarity networks. However, most existing approaches still rely heavily on traditional computational strategies, which are not guaranteed to achieve satisfactory performance. In this study, we propose a novel computational approach called GOGCN to measure gene functional similarity by modeling the Gene Ontology (GO) through Graph Convolutional Network (GCN). GOGCN is a graph-based approach that performs sufficient representation learning for terms and relations in the GO graph. First, GOGCN employs the GCN-based knowledge graph embedding (KGE) model to learn vector representations (i.e., embeddings) for all entities (i.e., terms). Second, GOGCN calculates the semantic similarity between two terms based on their corresponding vector representations. Finally, GOGCN estimates gene functional similarity by making use of the pair-wise strategy. During the representation learning period, GOGCN promotes semantic interaction between terms through GCN, thereby capturing the rich structural information of the GO graph. Further experimental results on various datasets suggest that GOGCN is superior to the other state-of-the-art approaches, which shows its reliability and effectiveness.

Fuzzy RDF Knowledge Graph Embeddings Through Vector Space Model

Resource description framework (RDF) is a World Wide Web consortium recommendation and has been widely accepted for semantic data models (e.g., knowledge graphs). As such, a large amount of RDF data is readily available. Information in the real world is often uncertain, and uncertain RDF data models have been proposed and received increasing attention. Knowledge graph embedding is an effective way to perform vector projection of structured knowledge, after which we can infer a large-scale knowledge graph. Nowadays, knowledge graph embedding models are widely proposed, but most of them can only handle crisp information. In this article, we concentrate on modeling fuzzy RDF graphs in the vector space. We propose a fuzzy RDF knowledge graph embedding model (FRKGE), which can project the fuzzy entities and fuzzy relations with membership degree to the vector space. Our proposed model preserves the structure of the fuzzy RDF knowledge graph and the explicit expression of membership degree in the vector space, which provides a natural expression of fuzzy RDF tuples. Our experimental results demonstrate that the FRKGE model is competent for the task of representation learning for fuzzy RDF knowledge graphs.

Dual Subgraph-Based Graph Neural Network for Friendship Prediction in Location-Based Social Networks

With the wide use of Location-Based Social Networks (LBSNs), predicting user friendship from online social relations and offline trajectory data is of great value to improve the platform service quality and user satisfaction. Existing methods mainly focus on some hand-crafted features or graph embedding models based on the user-location bipartite graph, which cannot precisely capture the latent mobility similarity for the majority of users who have no explicit co-visit behaviors and also fail to balance the tradeoff between social features and mobility features for friendship prediction. In this regard, we propose a dual subgraph-based pairwise graph neural network (DSGNN) for friendship prediction in LBSNs, which extracts a pairwise social subgraph and a trajectory subgraph to model the social proximity and mobility similarity, respectively. Specifically, to overcome the co-visit data sparsity, we design an entropy-based random walk to construct a location graph that captures the high-level correlation between locations. Based on this, we characterize the pairwise mobility similarity from trajectory level instead of location level, which is modeled by a graph neural network (GNN) on a labeled trajectory subgraph composed of the two trajectories of the target user pair. Besides, we also utilize another GNN to extract social proximity based on social subgraph of the target user pair. Finally, we propose a gate layer to adaptively balance the fusion of the social and mobility features for friendship prediction. We conduct extensive experiments on the real-world datasets and demonstrate the superiority of our approach, which outperforms other state-of-the-art methods. In particular, the comparative experiments on the trajectory level mobility similarity further validate the effectiveness of the designed trajectory subgraph-based method, which can extract predictive mobility features.

Adaptive Manifold Graph representation for Two-Dimensional Discriminant Projection

Two-dimensional (2D) local discriminant analysis is one of the popular techniques for image representation and recognition. Conventional 2D methods extract features of images relying on the affinity relationships among samples. However, affinity graph learning and dimensionality reduction are two separate processes, and the predefined affinity graph is sensitive to noisy and irrelevant features. To this end, we propose a graph embedding model, named as Two-Dimensional Discriminant Projection with Adaptive Manifold Graph (2DDP-AMG). In our model, the local manifold structure of data is characterized by the k-nearest-neighbor (kNN) graph within each class. More importantly, the above graph is updated adaptively in the process of dimensionality reduction to eliminate the interference of irrelevant features and noise. Furthermore, F-norm is introduced to measure the distance between homogeneous samples and improve the robustness of the model to outliers. To solve this model, an efficient algorithm is developed to update each variable iteratively. Extensive experiments conducted on benchmark face image data sets show the superiority and robustness of our model compared to state-of-the-art alternatives.

Preference-aware Graph Attention Networks for Cross-Domain Recommendations with Collaborative Knowledge Graph

Knowledge graphs (KGs) can provide users with semantic information and relations among numerous entities and nodes, which can greatly facilitate the performance of recommender systems. However, existing KG-based approaches still suffer from severe data sparsity and may not be effective in capturing the preference features of similar entities across domains. Therefore, in this article, we propose a P reference-aware G raph A ttention network model with C ollaborative K nowledge G raph ( PGACKG ) for cross-domain recommendations. Preference-aware entity embeddings with some collaborative signals are first obtained by exploiting the graph-embedding model, which can transform entities and items in the collaborative knowledge graph into semantic preference spaces. To better learn user preference features, we devise a preference-aware graph attention network framework that aggregates the preference features of similar entities within domains and across domains. In this framework, multi-hop reasoning is employed to assist in the generation of preference features within domains, and the node random walk based on frequency visits is proposed to gather similar preferences across domains for target entities. Then, the final preference features of entities are fused, while a novel C ross-domain B ayesian P ersonalized R anking ( CBPR ) is proposed to improve cross-domain recommendation accuracy. Extensive empirical experiments on four real-world datasets demonstrate that our proposed approach consistently outperforms state-of-the-art baselines. Furthermore, our PGACKG achieves strong performance in different ablation scenarios, and the interaction sparsity experiments also demonstrate that our proposed approach can significantly alleviate the data sparsity issue.

PTKE: Translation-based temporal knowledge graph embedding in polar coordinate system

Knowledge graph embedding has received widespread attention in recent years. Most existing models represent time-independent facts as low dimensional embeddings. Nevertheless, knowledge graphs with temporal information provide more accurate and timely data. Hence, we propose Polar Temporal Knowledge Graph Embedding (PTKE), a novel temporal knowledge graph (TKG) embedding model which belongs to the translation-based model family and embeds time-aware facts into polar coordinate system. PTKE defines time as a constraint of the entity and synchronously embeds the starting and ending timestamps. The fact is divided into the modulus and the angular parts to avoid generating similar time-constrained entities. We use the modulus part to distinguish different time-constrained entities, and the angular part to distinguish time-constrained entities with the same modulus. Experiments on the temporal datasets show that PTKE outperforms prior state-of-the-art static knowledge graph (SKG) embedding models and temporal knowledge graph (TKG) embedding models in the link prediction task and the relation prediction task. Furthermore, the analysis of different time units and semantic expressive ability test on time embeddings prove that PTKE has a great ability on time expression.

Efficient Knowledge Graph Embedding Training Framework with Multiple GPUs

When training a large-scale knowledge graph embedding (KGE) model with multiple graphics processing units (GPUs), the partition-based method is necessary for parallel training. However, existing partition-based training methods suffer from low GPU utilization and high input/output (10) overhead between the memory and disk. For a high 10 overhead between the disk and memory problem, we optimized the twice partitioning with fine-grained GPU scheduling to reduce the 10 overhead between the CPU memory and disk. For low GPU utilization caused by the GPU load imbalance problem, we proposed balanced partitioning and dynamic scheduling methods to accelerate the training speed in different cases. With the above methods, we proposed fine-grained partitioning KGE, an efficient KGE training framework with multiple GPUs. We conducted experiments on some benchmarks of the knowledge graph, and the results show that our method achieves speedup compared to existing framework on the training of KGE.

Graph Representation Learning Meets Computer Vision: A Survey

A graph structure is a powerful mathematical abstraction, which can not only represent information about individuals but also capture the interactions between individuals for reasoning. Geometric modeling and relational inference based on graph data is a long-standing topic of interest in the computer vision community. In this article, we provide a systematic review of graph representation learning and its applications in computer vision. First, we sort out the evolution of representation learning on graphs, categorizing them into the nonneural network and neural network methods based on the way the nodes are encoded. Specifically, nonneural network methods, such as graph embedding and probabilistic graphical models, are introduced, and neural network methods, such as graph recurrent neural networks, graph convolutional networks, and variants of graph neural networks, are also presented. Then, we organize the applications of graph representation algorithms in various vision tasks (such as image classification, semantic segmentation, object detection, and tracking) for review and reference, and the typical graph construction approaches in computer vision are also summarized. Finally, on the background of biology and brain inspiration, we discuss the existing challenges and future directions of graph representation learning and computer vision.

An attention‐based representation learning model for multiple relational knowledge graph

AbstractKnowledge graph embedding models are used to learn low‐dimensional representations of entities and relations in knowledge graphs. In this paper, we propose Multi‐RAttE, an attention‐based learning method for multiple relational knowledge graph embedding representation, which divides the information transfer in the knowledge graph into cross‐relational information transfer and relation‐specific information transfer, and divides the embedding of knowledge graph entities into structural embedding and multi‐relational embedding for joint learning. To objectively analyse the performance of the Multi‐RAttE model, we select two typical datasets and different representative baseline models for experimental evaluation on several tasks such as link prediction, multi‐relation prediction and node classification. The experimental results show that the Multi‐RAttE model improves 8% over the state‐of‐the‐art model Composition‐based Multi‐Relational Graph Convolutional Networks (CompGCN) in terms of Hits@1 metric on the link prediction task on FB15k‐237 dataset; on the multi‐relation prediction task, the accuracy improves by 1.8% and 3.7% in the auc metric and F1 metric, respectively. The experimental results have proved that the Multi‐RAttE model can effectively perform the representation of multiple relations.