Graph Embedding Model Research Articles

Universal Knowledge Graph Embedding Framework Based on High-Quality Negative Sampling and Weighting

The traditional model training approach based on negative sampling randomly samples a portion of negative samples for training, which can easily overlook important negative samples and adversely affect the training of knowledge graph embedding models. Some researchers have explored non-sampling model training frameworks that use all unobserved triples as negative samples to improve model training performance. However, both training methods inevitably introduce false negative samples and easy-to-separate negative samples that are far from the model’s decision boundary, and they do not consider the adverse effects of long-tail entities and relations during training, thus limiting the improvement of model training performance. To address this issue, we propose a universal knowledge graph embedding framework based on high-quality negative sampling and weighting, called HNSW-KGE. First, we conduct pre-training based on the NS-KGE non-sampling training framework to quickly obtain an initial set of relatively high-quality embedding vector representations for all entities and relations. Second, we design a candidate negative sample set construction strategy that samples a certain number of negative samples that are neither false negatives nor easy-to-separate negatives for all positive triples, based on the embedding vectors obtained from pre-training. This ensures the provision of high-quality negative samples for model training. Finally, we apply weighting to the loss function based on the frequency of the entities and relations appearing in the triples to mitigate the adverse effects of long-tail entities and relations on model training. Experiments conducted on benchmark datasets FB15K237 and WN18RR using various knowledge graph embedding models demonstrate that our proposed framework HNSW-KGE, based on high-quality negative sampling and weighting, achieves better training performance and exhibits versatility, making it applicable to various types of knowledge embedding models.

Joint embedding in Hierarchical distance and semantic representation learning for link prediction

The link prediction task aims to predict missing entities or relations in the knowledge graph and is essential for the downstream application. Existing well-known models deal with this task by mainly focusing on representing knowledge graph triplets in the distance space or semantic space. However, they can not fully capture the information of head and tail entities, nor even make good use of hierarchical level information. Thus, in this paper, we propose a novel knowledge graph embedding model for the link prediction task, namely, HIE, which models each triplet (h, r, t) into distance measurement space and semantic measurement space, simultaneously. Moreover, HIE is introduced into hierarchical-aware space to leverage rich hierarchical information of entities and relations for better representation learning. Specifically, we apply distance transformation operation on the head entity in distance space to obtain the tail entity instead of translation-based or rotation-based approaches. Experimental results of HIE on four real-world datasets show that HIE outperforms several existing state-of-the-art knowledge graph embedding methods on the link prediction task and deals with complex relations accurately.

A representation learning-based approach to enhancing manufacturing quality for low-voltage electrical products

In low-voltage electrical product manufacturing, resolving quality issues is heavily reliant on engineering experience, and can be time-consuming and error-prone. Through quality management systems, a large number of historical defect cases can be consolidated for analysis along with relevant causes. However, these defect descriptions are often casually described with a mix of Chinese and English language, containing domain-specific terms. Additionally, defect product features have varieties and complex relationships. Therefore, historical defect cases have not been effectively utilized to support manufacturing quality issues. To address this challenge, this study proposes a representation learning-based approach to enhance manufacturing quality. Key research contributions include: (1) A two-stage word embedding technique based on the pre-trained language model. First, TSDAE is utilized for unsupervised pre-training on a large amount of unlabeled data. Then, Sentence-BERT is utilized for fine-tuning on a small set of labeled similar sentence pairs. This process yields a pre-trained language model specific to low-voltage electrical product defects. (2) NSHPSAGE graph embedding model based on the constructed product feature knowledge graph. We select more valuable neighboring nodes during sampling and explore different aggregation functions to enhance graph embedding performance. This model effectively aggregates product feature information into “Defect_Case” nodes, yielding graph embedding vectors. The model exhibits good Weighted-Precision and Weighted-Recall with a short training duration, and it can handle new nodes, addressing the issue of heterogeneous graph embedding. (3) A defect case recommendation technique that fuses word embedding and graph embedding. We use Multi-Head Attention Fusion in the late-fusion to obtain defect case vectors. This approach comprehensively considers defect description semantic knowledge and complex product feature relationships, enabling accurate defect case recommendation with the prototype system.

A multi-entity reinforced main path analysis: Heterogeneous network embedding considering knowledge proximity

Main path analysis (MPA) is an important approach in detecting the trajectory of knowledge diffusion in a specific research domain. Previous studies always focus on citation-based relationships, overlooking other structural forms in citation network. This study introduces a multi-entity reinforced MPA model by constructing a knowledge graph from paper metadata, including citations, authors, journals, and keywords. We construct heterogeneous network to reveal relationships among various entities. Different knowledge graph embedding models are employed to train the network, thereby obtaining entity and relation embeddings. The cosine similarity algorithm is adopted to measure the knowledge proximity between these embeddings. We take the Internet of Thing domain as an example to verify the performance of the multi-entity reinforced MPA through both quantitative and qualitative analysis. Our findings indicate that the adjusted MPA exhibits stronger topic relevance, demonstrating the effectiveness of the method in capturing complex knowledge relationships.

FuseLinker: Leveraging LLM’s pre-trained text embeddings and domain knowledge to enhance GNN-based link prediction on biomedical knowledge graphs

ObjectiveTo develop the FuseLinker, a novel link prediction framework for biomedical knowledge graphs (BKGs), which fully exploits the graph’s structural, textual and domain knowledge information. We evaluated the utility of FuseLinker in the graph-based drug repurposing task through detailed case studies. MethodsFuseLinker leverages fused pre-trained text embedding and domain knowledge embedding to enhance the graph neural network (GNN)-based link prediction model tailored for BKGs. This framework includes three parts: a) obtain text embeddings for BKGs using embedding-visible large language models (LLMs), b) learn the representations of medical ontology as domain knowledge information by employing the Poincaré graph embedding method, and c) fuse these embeddings and further learn the graph structure representations of BKGs by applying a GNN-based link prediction model. We evaluated FuseLinker against traditional knowledge graph embedding models and a conventional GNN-based link prediction model across four public BKG datasets. Additionally, we examined the impact of using different embedding-visible LLMs on FuseLinker’s performance. Finally, we investigated FuseLinker’s ability to generate medical hypotheses through two drug repurposing case studies for Sorafenib and Parkinson’s disease. ResultsBy comparing FuseLinker with baseline models on four BKGs, our method demonstrates superior performance. The Mean Reciprocal Rank (MRR) and Area Under receiver operating characteristic Curve (AUROC) for KEGG50k, Hetionet, SuppKG and ADInt are 0.969 and 0.987, 0.548 and 0.903, 0.739 and 0.928, and 0.831 and 0.890, respectively. ConclusionOur study demonstrates that FuseLinker is an effective novel link prediction framework that integrates multiple graph information and shows significant potential for practical applications in biomedical and clinical tasks. Source code and data are available at https://github.com/YKXia0/FuseLinker.

Intelligent Asset Allocation Portfolio Division and Recommendation

With the continuous development of financial markets, intelligent asset allocation has become a topic of great concern in the investment field. However, traditional asset allocation methods often face difficulties in grasping the relationship between diversity, risk and return, which limits its application in complex market environments. To solve this problem, this study introduces deep learning and knowledge graphs and proposes an intelligent asset allocation model. Our model makes full use of the advantages of the Knowledge Graph Embedding Model (KGE), LSTM, and Genetic Algorithm (GA) to build a multi-level and multi-dimensional asset allocation model. KGE helps capture the complex relationships between different assets, LSTM is used to learn key patterns of historical portfolio performance, and GA finds the optimal asset allocation combination by simulating natural selection and genetic mechanisms. Experimental findings indicate that our model has demonstrated substantial improvements across various performance metrics and outperforms conventional approaches.

TAP with ease: a generic recommendation system for trigger-action programming based on multi-modal representation learning

The escalating popularity of smart devices has given rise to an increasing trend wherein users leverage customized trigger-action programming (TAP) rules within the Internet of Things (IoT) to automate various aspects of their lives. This article addresses the challenge of effectively combining functions provided by many smart devices and online services by introducing a novel multi-modal representation learning model called TAP-TAG. This model integrates both textual and graph structures inherent in TAP rules, offering a holistic method to rule recommendation. TAP-TAG comprises two branches: the Knowledge Graph Embedding model, which projects triplets extracted from the TAP dataset into embeddings, and convolution neural networks that extract semantic features from the textual content of TAP rules. Extensive experiments are conducted on real-world TAP datasets to evaluate our model’s ability to recommend relevant rules based on user preferences. The experimental results show that TAP-TAG can outperform the state-of-the-art method by 5% in Precision@5, indicating that TAP-TAG is highly effective in providing accurate and diverse recommendations for TAP rules.

Self-X heterogeneous attributed graph embedding-based product configuration framework for cognitive mass personalization

Cognitive mass personalization (CMP) is a promising manufacturing paradigm; equipped with cognitive capabilities like reasoning, CMP satisfies changeable needs via configuring personalized products at scale. In CMP, knowledge graphs (KGs) are exploited by smart product-service systems (SPSS) to support cognitive configuration/reconfiguration processes. However, the extant KG-enabled SPSSs are built upon fixed configurations and hybrid frameworks due to lacking a graph embedding (GE) model to render cognitive configuration decisions. In fact, GE is scarcely used in SPSS configuration, because it is not only compromised by the heterogeneity of KGs entailed by content-related specifications and complex structures but also influenced by the feature randomness and feature drift problems, which are triggered by accumulative errors and inconsistent objectives due to noisy assignments and different configuration tasks, separately. To address these limitations, a Self-X Heterogeneous Attributed Graph Embedding (SXHAGE) model is proposed in a Self-X architecture, which includes 1) self-attention graph attention networks, 2) a self-adaptive autoencoder, and 3) self-optimizing training objectives, to present heterogeneous data through jointly optimizing heterogeneous attributed entities and relations. A systematic SXHAGE-based configuration framework, in which product family design and configuration recommending are enabled by graph clustering and link prediction, is developed as a continuous updating loop to proactively configure personalized products. A real-world case study, i.e., configure personalized electric clippers via a web-based sustainable configuration platform, is performed to validate the applicability of the proposed framework in the CMP context. Moreover, extensive experiments on the case study dataset demonstrate the superiority of SXHAGE over the state-of-the-art algorithms, e.g., surpassing Deep Neighbor-Aware Embedding (DNENC) by 18 % in F1-score for graph clustering and by 5 % in ROC-AUC for link prediction.

ERT-GFAN: A multimodal drug–target interaction prediction model based on molecular biology and knowledge-enhanced attention mechanism

In drug discovery, precisely identifying drug–target interactions is crucial for finding new drugs and understanding drug mechanisms. Evolving drug/target heterogeneous data presents challenges in obtaining multimodal representation in drug–target prediction(DTI). To deal with this, we propose ‘ERT-GFAN’, a multimodal drug–target interaction prediction model inspired by molecular biology. Firstly, it integrates bio-inspired principles to obtain structure feature of drugs and targets using Extended Connectivity Fingerprints(ECFP). Simultaneously, the knowledge graph embedding model RotatE is employed to discover the interaction feature of drug–target pairs. Subsequently, Transformer is utilized to refine the contextual neighborhood features from the obtained structure feature and interaction features, and multi-modal high-dimensional fusion features of the three-modal information constructed. Finally, the final DTI prediction results are outputted by integrating the multimodal fusion features into a graphical high-dimensional fusion feature attention network (GFAN) using our innovative multimodal high-dimensional fusion feature attention. This multimodal approach offers a comprehensive understanding of drug–target interactions, addressing challenges in complex knowledge graphs. By combining structure feature, interaction feature, and contextual neighborhood features, ‘ERT-GFAN’ excels in predicting DTI. Empirical evaluations on three datasets demonstrate our method’s superior performance, with AUC of 0.9739, 0.9862, and 0.9667, AUPR of 0.9598, 0.9789, and 0.9750, and Mean Reciprocal Rank(MRR) of 0.7386, 0.7035, and 0.7133. Ablation studies show over a 5% improvement in predictive performance compared to baseline unimodal and bimodal models. These results, along with detailed case studies, highlight the efficacy and robustness of our approach.

An radicals construction technique based on dual quaternions and hierarchical transformers

At present, the exploration of Knowledge Graph Embedding (KGE) mainly focuses on static KGE models. Due to the diversity and complexity of vertical domain data, it is difficult for existing KGE models to achieve excellent performance. Specific KGE models must be constructed for research on vertical domain knowledge graphs. This paper explores the relationships between the composition of Chinese characters and their sounds, meanings, and forms by constructing the Chinese Radical Knowledge Graph (CRKG). CRKG was constructed from a unique perspective according to the characteristics of Chinese radicals and knowledge graphs. The minor components of Chinese characters are separated and constructed into the fourth dimension of the knowledge graph. In this paper, we propose a new Chinese Radical Knowledge Graph Embedding (CRKGE) model, RotAL, which introduces dual quaternions and hierarchical transformers into KGE. RotAL performs a translation-rotation operation on the head entity with the relation as a parameter, then does a translation operation on the head entity with the minor component as a parameter to be close to the tail entity. Finally, we feed the trained quaternions into hierarchical transformers to jointly learn entity-related composition and relational contextualization based on a head entity’s neighborhood. We demonstrate that our method can model and infer various relational patterns of CRKG, such as symmetric/antisymmetric, reflexive, inversion, and multiple. The experimental results on four CRKG datasets demonstrate that the RotAL model achieves the highest Hit@1 score, outperforming existing KGE models in accurately identifying the target entity on the first attempt. Furthermore, the RotAL model consistently achieves superior performance across other evaluation metrics, substantiating its significant advantage over existing models.

Concept graph embedding models for enhanced accuracy and interpretability

Concept graph embedding models for enhanced accuracy and interpretability

Manufacturing resilience through disruption mitigation using attention-based consistently-attributed graph embedded decision support system

With global supply chains experiencing significant disruptions, there is increasing emphasis on enhancing manufacturing resilience by mitigating supply chain-propagated impacts on production entities in a holistic manner. However, existing mitigative strategies typically rely on implicit experience, which may be biased and suboptimal. Furthermore, the lack of consideration towards the chain of custody, multi-disruption management, and transportation networks results in unintuitive systems. To overcome these challenges, this study proposes a graph embedding-based mitigation decision support system (GEM-DSS) using a heterogeneous industrial knowledge graph to facilitate disruption management. Addressing the prevailing industrial challenge of knowledge representations and graph incompleteness, an attention-based consistently-attributed graph embedding (ACAGE) model is proposed, with the capability to learn and factor in multi-attributes and multi-structures within the knowledge graph. Through supplier selection and material substitution mitigation strategies, two computational pipelines are designed as part of a dual-stage search mechanism to generate feasible solutions. The experimental results of the ACAGE model for knowledge graph completion outperform several attributed graph embedding baselines, achieving 0.999 AUC-ROC and 0.990 AUC-PR on the constructed real-world industrial knowledge graph. The functionality and applicability of the GEM-DSS are demonstrated in an automotive case study with dynamic considerations via a cognitive computing methodology. This study aims to provide valuable insights for both academia and industry towards enabling manufacturing continuity in the face of global supply chain challenges.

A Knowledge Graph Embedding Model for Answering Factoid Entity Questions

Factoid entity questions (FEQ), which seek answers in the form of a single entity from knowledge sources such as DBpedia and Wikidata, constitute a substantial portion of user queries in search engines. This paper introduces the Knowledge Graph Embedding model for Factoid Entity Question answering (KGE-FEQ). Leveraging a textual knowledge graph derived from extensive text collections, KGE-FEQ encodes textual relationships between entities. The model employs a two-step process: (1) Triple Retrieval, where relevant triples are retrieved from the textual knowledge graph based on semantic similarities to the question, and (2) Answer Selection, where a knowledge graph embedding approach is utilized for answering the question. This involves positioning the embedding for the answer entity close to the embedding of the question entity, incorporating a vector representing the question and textual relations between entities. Extensive experiments evaluate the performance of the proposed approach, comparing KGE-FEQ to state-of-the-art baselines in factoid entity question answering and the most advanced open-domain question answering techniques applied to FEQs. The results show that KGE-FEQ outperforms existing methods across different datasets. Ablation studies highlights the effectiveness of KGE-FEQ when both the question and textual relations between entities are considered for answering questions.

Alzheimer's Disease Knowledge Graph Enhances Knowledge Discovery and Disease Prediction.

Alzheimer's disease (AD), a progressive neurodegenerative disorder, continues to increase in prevalence without any effective treatments to date. In this context, knowledge graphs (KGs) have emerged as a pivotal tool in biomedical research, offering new perspectives on drug repurposing and biomarker discovery by analyzing intricate network structures. Our study seeks to build an AD-specific knowledge graph, highlighting interactions among AD, genes, variants, chemicals, drugs, and other diseases. The goal is to shed light on existing treatments, potential targets, and diagnostic methods for AD, thereby aiding in drug repurposing and the identification of biomarkers. We annotated 800 PubMed abstracts and leveraged GPT-4 for text augmentation to enrich our training data for named entity recognition (NER) and relation classification. A comprehensive data mining model, integrating NER and relationship classification, was trained on the annotated corpus. This model was subsequently applied to extract relation triplets from unannotated abstracts. To enhance entity linking, we utilized a suite of reference biomedical databases and refine the linking accuracy through abbreviation resolution. As a result, we successfully identified 3,199,276 entity mentions and 633,733 triplets, elucidating connections between 5,000 unique entities. These connections were pivotal in constructing a comprehensive Alzheimer's Disease Knowledge Graph (ADKG). We also integrated the ADKG constructed after entity linking with other biomedical databases. The ADKG served as a training ground for Knowledge Graph Embedding models with the high-ranking predicted triplets supported by evidence, underscoring the utility of ADKG in generating testable scientific hypotheses. Further application of ADKG in predictive modeling using the UK Biobank data revealed models based on ADKG outperforming others, as evidenced by higher values in the areas under the receiver operating characteristic (ROC) curves. The ADKG is a valuable resource for generating hypotheses and enhancing predictive models, highlighting its potential to advance AD's disease research and treatment strategies.

Towards assessing the quality of knowledge graphs via differential testing

Knowledge graphs (KG) can aggregate data and make information resources easier to calculate and understand. With tremendous advancements in knowledge graphs, they have been incorporated into plenty of software systems to assist various tasks. However, while KGs determine the performance of downstream software systems, their quality is often measured by the accuracy of test data. Considering the limitation of accessible high-quality test data, an automated quality assessment technique could fundamentally improve the testing efficiency of KG-driven software systems and save plenty of manual labeling resources.In this paper, we propose an automated approach to quantify the quality of KGs via differential testing. It first constructs multiple Knowledge Graph Embedding Models (KGEM) and conducts head prediction tasks on models. Then, it can produce a differential score that reflects the quality of KGs by comparing the proximity of output results. To validate the effectiveness of this approach, we experiment with four open-sourced knowledge graphs. The experiment results show that our approach is capable of accurately evaluating the quality of KGs and producing reliable results on different datasets. Moreover, we compared our method with existing methods and achieved certain advantages. The potential usefulness of our approach sheds light on the development of various KG-driven software systems.

Spatio-temporal knowledge embedding method considering the lifecycle of geographical entities

With the emergence of substantial amounts of data featuring spatio-temporal characteristics, research on spatio-temporal knowledge has garnered widespread attention. However, the existing knowledge graph embedding methods are difficult to embed spatio-temporal knowledge in continuous time, failing to meet the unique embedding requirements of geographic entities for uninterrupted, constantly changing, and smoothly continuous embeddings. To address this issue, we begin by outlining a version-based representation of spatio-temporal knowledge. Building on this foundation, we propose a novel and effective knowledge graph embedding model called Version Embedding (VerE), which embeds continuous time into vector space, transforms geographic entities into corresponding versions using an attention mechanism, and calculates spatio-temporal knowledge scores under the constraint of version similarity regularization. Subsequently, we introduce a method for link prediction based on unknown time with the aim of assessing the model’s generalization capabilities in time representation. Finally, experiments conducted on two real-world datasets demonstrate significant performance improvement of VerE compared to most existing models, and its ability to maintain stability and continuity in link prediction under unknown time. These experiments confirmed the effectiveness of the proposed model and provided new perspectives and methods for spatio-temporal knowledge embedding.

Improving static and temporal knowledge graph embedding using affine transformations of entities

To find a suitable embedding for a knowledge graph (KG) remains a big challenge nowadays. By measuring the distance or plausibility of triples and quadruples in static and temporal knowledge graphs, many reliable knowledge graph embedding (KGE) models are proposed. However, these classical models may not be able to represent and infer various relation patterns well, such as TransE cannot represent symmetric relations, DistMult cannot represent inverse relations, RotatE cannot represent multiple relations, etc.. In this paper, we improve the ability of these models to represent various relation patterns by introducing the affine transformation framework. Specifically, we first utilize a set of affine transformations related to each relation or timestamp to operate on entity vectors, and then these transformed vectors can be applied not only to static KGE models, but also to temporal KGE models. The main advantage of using affine transformations is their good geometry properties with interpretability. Our experimental results demonstrate that the proposed intuitive design with affine transformations provides a statistically significant increase in performance with adding a few extra processing steps and keeping the same number of embedding parameters. Taking TransE as an example, we employ the scale transformation (the special case of an affine transformation). Surprisingly, it even outperforms RotatE to some extent on various datasets. We also introduce affine transformations into RotatE, Distmult, ComplEx, TTransE and TComplEx respectively, and experiments demonstrate that affine transformations consistently and significantly improve the performance of state-of-the-art KGE models on both static and temporal knowledge graph benchmarks.

Social Perception with Graph Attention Network for Recommendation

Recommendation systems are designed to uncover users’ potential preferences and make recommendations. However, they often face challenges such as data sparsity and the cold start problem. Although the introduction of knowledge graphs has partially addressed the issue of data sparsity, the challenge of cold start has not been effectively resolved. In this paper, a novel approach called Social Perception with Graph Attention Network (SPGAT) for Recommendation is proposed. In SPGAT, we aim to leverage social perception to solve the cold start effectively for more accurate recommendations. The approach utilizes a multi-layer graph attention network to aggregate user preference features from collaborative knowledge graphs and social perception graphs. By analyzing the social network of a new user, associated friend users can be identified. The interaction data of these friend users is then provided as side information to recommend to the new user. To handle one-to-many and many-to-many relations, we introduce the TransD graph embedding model, which maps different types of relations and entities to different spaces. To optimize the proposed SPGAT, self-adversarial negative sampling is utilized to implement entity and relation embedding and generate negative samples. Experimental results demonstrate that SPGAT has achieved superior performance compared to several advanced methods.

A knowledge graph embedding model based on multi-level analogical reasoning

A knowledge graph embedding model based on multi-level analogical reasoning

A Personalized Collaborative Filtering Recommendation System Based on Bi-Graph Embedding and Causal Reasoning.

The integration of graph embedding technology and collaborative filtering algorithms has shown promise in enhancing the performance of recommendation systems. However, existing integrated recommendation algorithms often suffer from feature bias and lack effectiveness in personalized user recommendation. For instance, users' historical interactions with a certain class of items may inaccurately lead to recommendations of all items within that class, resulting in feature bias. Moreover, accommodating changes in user interests over time poses a significant challenge. This study introduces a novel recommendation model, RCKFM, which addresses these shortcomings by leveraging the CoFM model, TransR graph embedding model, backdoor tuning of causal inference, KL divergence, and the factorization machine model. RCKFM focuses on improving graph embedding technology, adjusting feature bias in embedding models, and achieving personalized recommendations. Specifically, it employs the TransR graph embedding model to handle various relationship types effectively, mitigates feature bias using causal inference techniques, and predicts changes in user interests through KL divergence, thereby enhancing the accuracy of personalized recommendations. Experimental evaluations conducted on publicly available datasets, including "MovieLens-1M" and "Douban dataset" from Kaggle, demonstrate the superior performance of the RCKFM model. The results indicate a significant improvement of between 3.17% and 6.81% in key indicators such as precision, recall, normalized discount cumulative gain, and hit rate in the top-10 recommendation tasks. These findings underscore the efficacy and potential impact of the proposed RCKFM model in advancing recommendation systems.