Embedding Knowledge Research Articles

MISMS: a multistep-ahead and interpretable sequential modelling scheme for the long-term dynamic forecasting in process industries

This paper introduces Multistep-ahead and Interpretable Sequential Modeling Scheme (MISMS), a pioneering approach for long-term dynamic forecasting that enhances model interpretability in process industries. MISMS is distinguished by its innovative integration of multivariate analysis, multistep-ahead forecasting, and sequential model interpretability. The scheme incorporates knowledge embedding and Exploratory Data Analysis (EDA) to structure multiple variables and extract their temporal attributes. Based on this groundwork, sequence-to-sequence (Seq2Seq) submodels are developed to capture domain-wide dynamics rather than one single value. Crucially, MISMS has pioneered the application of sequence interpretation for model decomposition and transparency, improving interpretability. The proposed scheme employs different temporal models in practical scenarios and can ensure a better prediction with long-term accuracy and stability. Additionally, discussions on the model explainer's expected and unexpected outcomes are conducted, providing potential avenues for future research in process industries.

A conversational ambiguity resolution model for contextual knowledge embedding

A conversational ambiguity resolution model for contextual knowledge embedding

Knowledge diffusion as a key element in smart governance

This study examines the critical role of knowledge diffusion in smart governance within complex and globalized contexts, emphasizing its importance in driving technological advancement and fostering an information-driven society. By integrating smart governance with knowledge diffusion, the research aims to identify strategies for building a more equitable and inclusive society. A narrative review of documents from 2015 to 2023 was conducted using Boolean operators across major databases, followed by a rigorous selection and analysis using a traceability matrix. The findings highlight the innovative culture of smart governance, showcasing successful integration of citizen participation, public value creation, open government, and interorganizational collaboration. The results demonstrate that effective knowledge diffusion is essential for establishing robust governance systems that contribute significantly to economic and social development. The study concludes that embedding knowledge diffusion within smart governance frameworks can drive substantial societal improvements, promoting greater equity and justice

Construction and control parameter matching method of hydraulic pile hammer surrogate model combining transfer algorithm and knowledge embedding

Due to its complex application environment, hydraulic pile hammers are difficult to model and accurately grasp their motion laws in automation control. Therefore, to address the high-precision prediction problem of the motion law of hydraulic pile hammer systems under complex working conditions, a knowledge embedding surrogate model based on a dual neural network is designed, which can be divided into two parts: feature extraction network and prediction network. On this basis, the model introduces physics knowledge embedding technology, adds inequality constraints based on physical laws to the loss function, and introduces transfer learning algorithms to maintain high accuracy of the model under limited data conditions. The results showed that the PSM (γ = 0.5) model designed in the study had a median of -0.001 and a variance of 0.038 under undisturbed conditions, demonstrating high prediction stability. Under perturbation conditions, the median of the PSM (γ = 1.5) model was -0.003, with a variance of 0.031, indicating the highest prediction accuracy. From this, the research model has good prediction accuracy under complex working conditions, which can lay a technical foundation for predicting the motion law of hydraulic pile hammers and automatic control.

Recognition of building group patterns using GCN and knowledge graph

Effective identification of building patterns can improve the quality of automated map generalization. Established methods are limited by complex rule definitions and cannot fully consider the local domain information of buildings patterns, and few studies have focused on recognizing multiple building group patterns using GCN. Therefore, we propose a new model for recognizing building patterns by combining knowledge graph methods and GCN methods. First, the features of individual buildings are acquired, then the graph structure of building is constructed, and finally, by means of GCN and knowledge embedding, the features of buildings are efficiently learned on the basis of the graph structure of building patterns, and the pattern features of building are extracted, so as to realize the recognition of building patterns. The results show that the training accuracy and testing accuracy reach 99.03% and 95.89%, respectively. Compared with other methods, the proposed model can effectively utilize the local spatial information of building patterns and accurately recognize building patterns.

Model can be subtle: Two important mechanisms for Social Media Popularity Prediction

Social media popularity prediction is an important channel to explore content sharing and communication on social networks. It aims to capture informative cues by analyzing multi-type data (such as user profile, image, and text) to decide the popularity of a specified post. In this paper, we divide social network users into two categories (i.e., active and inactive users) and find a dilemma in existing models: If an active user publishes the low-popularity post, the model will habitually predict the high score. On the contrary, if an inactive user provides the high-popularity post, the model still gives the low score incorrectly. Therefore, how to make the model more subtle to users is important. Comparing to existing methods that directly leverage multi-modal features for regression training, this paper stresses more on two novel mechanisms. The first method aims to prevent the over-fitting on user IDs. We propose the attribute-sensitive interactive mechanism (M1) by incorporating explicit user-attribute and post-attribute interaction. It can analyze which type of features a user cares the most and weaken the model’s dependence on user IDs. The second method aims to strengthen the influence of post content. We propose the knowledge embedding mechanism (M2) to revise the popularity scores in existing models by fusing the statistical frequency over multi-type data. Note that both mechanisms are model-agnostic, which can be applicable in any popularity prediction model. Extensive experiments conducted on the Social Media Prediction Dataset further validate the effectiveness.

Knowledge Embedding Networks Based on Deep Learning for Automatic Modulation Classification in Cognitive Radio

Knowledge Embedding Networks Based on Deep Learning for Automatic Modulation Classification in Cognitive Radio

A Prior Knowledge Embedding Contrastive Attention Learning Network for Variable Working Conditions Bearing Fault Diagnosis With Small Samples

A Prior Knowledge Embedding Contrastive Attention Learning Network for Variable Working Conditions Bearing Fault Diagnosis With Small Samples

Domain-aware Multimodal Dialog Systems with Distribution-based User Characteristic Modeling

Textual response generation is a pivotal yet challenging task for multimodal task-oriented dialog systems, which targets at generating the appropriate textual response given the multimodal context. Although existing efforts have obtained remarkable advancements, they ignore the potential of the domain information in revealing the key points of the user intention and the user's history dialogs in indicating the user's characteristics. To address this issue, in this work, we propose a novel domain-aware multimodal dialog system with distribution-based user characteristic modeling (named DMDU). In particular, DMDU contains three vital components: context-knowledge embedding extraction , domain-aware response generation and distribution-based user characteristic injection . Specifically, the context-knowledge embedding extraction component aims to extract the embedding of multimodal context and related knowledge following existing studies. The domain-aware response generation component targets at conducting domain-aware fine-grained intention modeling based on the context and knowledge embedding, and thus fulfills the textual response generation. Moreover, the distribution-based user characteristic injection component first captures the user's characteristics and current intention with the Gaussian distribution, and then conducts the sampling-based contrastive semantic regularization to promote the context representation learning. Experimental results on the public dataset demonstrate the effectiveness of DMDU. We release codes to promote other researchers.

Aggregation or separation? Adaptive embedding message passing for knowledge graph completion

Knowledge graph completion intends to infer information within knowledge graphs, thereby bolstering the functionality of knowledge-driven applications. Recently, there has been a significant increase in the utilization of graph convolutional networks (GCNs) for knowledge graph completion. These GCN-based models primarily focus on aggregating information from neighboring entities and relations. Nonetheless, a fundamental question arises: is it beneficial to consider all neighbor information, and should some neighbor features be separated? We tackle this issue and present an adaptive graph convolutional network (AdaGCN) for knowledge graph completion, which can adaptively aggregate or separate neighbor information for knowledge embedding learning. Specifically, AdaGCN utilizes the adaptive message-passing mechanism to determine the importance of each relation, allocating weights to neighbor entity embeddings. This adaptive approach facilitates the propagation of valuable information while effectively separating less relevant or unnecessary details. Experimental results demonstrate that AdaGCN can efficiently acquire the embeddings of various triplets within knowledge graphs, and it achieves competitive performance compared to SOTA models on six datasets for the tasks of knowledge graph completion.

Universal Knowledge Graph Embedding Framework Based on High-Quality Negative Sampling and Weighting

The traditional model training approach based on negative sampling randomly samples a portion of negative samples for training, which can easily overlook important negative samples and adversely affect the training of knowledge graph embedding models. Some researchers have explored non-sampling model training frameworks that use all unobserved triples as negative samples to improve model training performance. However, both training methods inevitably introduce false negative samples and easy-to-separate negative samples that are far from the model’s decision boundary, and they do not consider the adverse effects of long-tail entities and relations during training, thus limiting the improvement of model training performance. To address this issue, we propose a universal knowledge graph embedding framework based on high-quality negative sampling and weighting, called HNSW-KGE. First, we conduct pre-training based on the NS-KGE non-sampling training framework to quickly obtain an initial set of relatively high-quality embedding vector representations for all entities and relations. Second, we design a candidate negative sample set construction strategy that samples a certain number of negative samples that are neither false negatives nor easy-to-separate negatives for all positive triples, based on the embedding vectors obtained from pre-training. This ensures the provision of high-quality negative samples for model training. Finally, we apply weighting to the loss function based on the frequency of the entities and relations appearing in the triples to mitigate the adverse effects of long-tail entities and relations on model training. Experiments conducted on benchmark datasets FB15K237 and WN18RR using various knowledge graph embedding models demonstrate that our proposed framework HNSW-KGE, based on high-quality negative sampling and weighting, achieves better training performance and exhibits versatility, making it applicable to various types of knowledge embedding models.

SSQ[formula omitted]: A Subgraph-based Semantic Query Approach for Temporal Knowledge Graph

Real-world knowledge graphs are growing in size with the explosion of data and rapid expansion of knowledge. There are some studies on knowledge graph query, but temporal knowledge graph (TKG) query is still a relatively unexplored field. A temporal knowledge graph is a knowledge graph that contains temporal information and contains knowledge that is likely to change over time. It introduces a temporal dimension that can characterize the changes and evolution of entities and relationships at different points in time. However, in the existing temporal knowledge graph query, the entity labels are one-sided, which cannot accurately reflect the semantic relationships of temporal knowledge graphs, resulting in incomplete query results. For the processing of temporal information in temporal knowledge graphs, we propose a temporal frame filtering approach and measure the acceptability of temporal frames by the new definition simtime based on the proposed three temporal frames and nine rules. For measuring the semantic relationship of predicates between entities, we vectorize the semantic similarity between predicates, i.e., edges, using the knowledge embedding model, and propose the new definition simpre to measure the semantic similarity of predicates. Based on these, we propose a new semantic temporal knowledge graph query method SSQTKG, and perform pruning operations to optimize the query efficiency of the algorithm based on connectivity. Extensive experiments show that SSQTKG can return more accurate and complete results that meet the query conditions in the semantic query and can improve the performance of the querying on the temporal knowledge graph.

Aesthetic image cropping meets VLP: Enhancing good while reducing bad

Aesthetic Image Cropping (AIC) enhances the visual appeal of an image by adjusting its composition and aesthetic elements. People make these adjustments based on these elements, aiming to enhance appealing aspects while minimizing detrimental factors. Motivated by these observations, we propose a novel approach called CLIPCropping, which simulates the human decision-making process in AIC. CLIPCropping leverages Contrastive Language–Image Pre-training (CLIP) to align visual perception with textual description. It consists of three branches: composition embedding, aesthetic embedding, and image cropping. The composition embedding branch learns principles based on Composition Knowledge Embedding (CKE), while the aesthetic embedding branch learns principles based on Aesthetic Knowledge Embedding (AKE). The image cropping branch evaluates the quality of candidate crops by aggregating knowledge from CKE and AKE; an MLP produces the best result. Extensive experiments on three benchmark datasets — GAICD-1236, GAICD-3336, and FCDB — show that CLIPCropping outperforms state-of-the-art methods and provides insightful interpretations.

Towards Data-and Knowledge-Driven AI: A Survey on Neuro-Symbolic Computing.

Neural-symbolic computing (NeSy), which pursues the integration of the symbolic and statistical paradigms of cognition, has been an active research area of Artificial Intelligence (AI) for many years. As NeSy shows promise of reconciling the advantages of reasoning and interpretability of symbolic representation and robust learning in neural networks, it may serve as a catalyst for the next generation of AI. In the present paper, we provide a systematic overview of the recent developments and important contributions of NeSy research. Firstly, we introduce study history of this area, covering early work and foundations. We further discuss background concepts and identify key driving factors behind the development of NeSy. Afterward, we categorize recent landmark approaches along several main characteristics that underline this research paradigm, including neural-symbolic integration, knowledge representation, knowledge embedding, and functionality. Next, we briefly discuss the successful application of modern NeSy approaches in several domains. Then, we benchmark several NeSy methods on three representative application tasks. Finally, we identify the open problems together with potential future research directions. This survey is expected to help new researchers enter this rapidly evolving field and accelerate the progress towards data-and knowledge-driven AI.

Knowledge Graph Embedding for Hierarchical Entities Based on Auto-Embedding Size

Knowledge graph embedding represents entities and relations as low-dimensional continuous vectors. Recently, researchers have attempted to leverage the potential semantic connections between entities with hierarchical relationships in the knowledge graph. However, existing knowledge embedding methods that model entity hierarchical structures face the following challenges: (1) Setting the same embedding dimension for entities at different hierarchical levels can lead to overfitting issues and the waste of storage and computational resources; (2) Manually searching the embedding dimension in discrete space makes it easy to miss the optimal parameters and increases training costs. Therefore, we propose a knowledge embedding method, HEAES, that automatically learns the embedding dimensions for entities based on their hierarchical structure. Specifically, we first propose a new modeling method to capture the hierarchical relationships of entities, and we then train the model on a triple classification task. Then, we adaptively learn the pruning threshold to trim the embedding vectors, automatically learning different embedding dimensions for entities at different hierarchical levels. Experiments on the YAGO26K and DB111K datasets verify that introducing entity hierarchical information helps boost the model performance.

KnoMol: A Knowledge-Enhanced Graph Transformer for Molecular Property Prediction.

Molecular property prediction (MPP) techniques are pivotal in reducing drug development costs by preemptively predicting bioactivity and ADMET properties. Despite the application of numerous deep learning approaches, enhancing the representational capacity of these models remains a significant challenge. This paper presents a novel knowledge-based Transformer framework, KnoMol, designed to improve the understanding of molecular structures. KnoMol integrates expert chemical knowledge into the Transformer, emulating the analytical methods of medicinal chemists. Additionally, the multiperspective attention mechanism provides a more precise way to represent ring systems. In the evaluation experiments, KnoMol achieved state-of-the-art performance on both MoleculeNet and small-scale data sets, surpassing existing models in terms of accuracy and generalization. Further research indicated that the incorporation of knowledge significantly reduces KnoMol's reliance on data volumes, offering a solution to the challenge of data scarcity. Moreover, KnoMol identified several new inhibitors of HER2 in a case study, demonstrating its value in real-world applications. Overall, this research not only provides a powerful tool for MPP but also serves as a successful precedent for embedding knowledge into Transformers, with positive implications for computer-aided drug discovery and the development of MPP algorithms.

FuseLinker: Leveraging LLM’s pre-trained text embeddings and domain knowledge to enhance GNN-based link prediction on biomedical knowledge graphs

ObjectiveTo develop the FuseLinker, a novel link prediction framework for biomedical knowledge graphs (BKGs), which fully exploits the graph’s structural, textual and domain knowledge information. We evaluated the utility of FuseLinker in the graph-based drug repurposing task through detailed case studies. MethodsFuseLinker leverages fused pre-trained text embedding and domain knowledge embedding to enhance the graph neural network (GNN)-based link prediction model tailored for BKGs. This framework includes three parts: a) obtain text embeddings for BKGs using embedding-visible large language models (LLMs), b) learn the representations of medical ontology as domain knowledge information by employing the Poincaré graph embedding method, and c) fuse these embeddings and further learn the graph structure representations of BKGs by applying a GNN-based link prediction model. We evaluated FuseLinker against traditional knowledge graph embedding models and a conventional GNN-based link prediction model across four public BKG datasets. Additionally, we examined the impact of using different embedding-visible LLMs on FuseLinker’s performance. Finally, we investigated FuseLinker’s ability to generate medical hypotheses through two drug repurposing case studies for Sorafenib and Parkinson’s disease. ResultsBy comparing FuseLinker with baseline models on four BKGs, our method demonstrates superior performance. The Mean Reciprocal Rank (MRR) and Area Under receiver operating characteristic Curve (AUROC) for KEGG50k, Hetionet, SuppKG and ADInt are 0.969 and 0.987, 0.548 and 0.903, 0.739 and 0.928, and 0.831 and 0.890, respectively. ConclusionOur study demonstrates that FuseLinker is an effective novel link prediction framework that integrates multiple graph information and shows significant potential for practical applications in biomedical and clinical tasks. Source code and data are available at https://github.com/YKXia0/FuseLinker.

An implicit aspect-based sentiment analysis method using supervised contrastive learning and knowledge embedding

Aspect-based sentiment analysis aims to analyze and understand people’s opinions from different aspects. Some comments do not contain explicit opinion words but still convey a clear human-perceived emotional orientation, which is known as implicit sentiment. Most previous research relies on contextual information from a text for implicit aspect-based sentiment analysis. However, little work has integrated external knowledge with contextual information. This paper proposes an implicit aspect-based sentiment analysis model combining supervised contrastive learning with knowledge-enhanced fine-tuning on BERT (BERT-SCL+KEFT). In the pre-training phase, the model utilizes supervised contrastive learning (SCL) on large-scale sentiment-annotated corpora to acquire sentiment knowledge. In the fine-tuning phase, the model uses a knowledge-enhanced fine-tuning (KEFT) method to capture explicit and implicit aspect-based sentiments. Specifically, the model utilizes knowledge embedding to embed external general knowledge information into textual entities by using knowledge graphs, enriching textual information. Finally, the model combines external knowledge and contextual features to predict the implicit sentiment in a text. The experimental results demonstrate that the proposed BERT-SCL+KEFT model outperforms other baselines on the general implicit sentiment analysis and implicit aspect-based sentiment analysis tasks. In addition, ablation experimental results show that the proposed BERT-SCL+KEFT model without the knowledge embedding module or supervised contrastive learning module significantly decreases performance, indicating the importance of these modules. All experiments validate that the proposed BERT-SCL+KEFT model effectively achieves implicit aspect-based sentiment classification.

Prior knowledge embedding convolutional autoencoder: A single-source domain generalized fault diagnosis framework under small samples

The proposed transfer learning-based fault diagnosis models have achieved good results in multi-source domain generalization (MDG) tasks. However, research on single-source domain generalization (SDG) is relatively scarce, and the limited availability of small training samples is seldom considered. Therefore, to address the insufficient feature extraction capability and poor generalization performance of existing models on single-source domain small sample data, a novel single-source domain generalization fault diagnosis (SDGFD) framework, the prior knowledge embedded convolutional autoencoder (PKECA), is proposed. During the training phase, first, single-source domain data are used to construct prior features based on the time domain, frequency domain, and time-frequency domain. Second, a prior knowledge embedding structure based on the convolutional autoencoder is built, which compresses the prior knowledge and original vibration data into a high-dimensional space of consistent dimensions, embedding the prior knowledge into the features corresponding to the vibration data using a mean squared error loss function. Subsequently, the proposed centroid-based self-supervised learning (CBSSL) strategy further constrains high-dimensional features, improving the generalization ability. The designed sparse regularized activation (SRA) function significantly enhances the regularization effect on features. During the testing phase, it is only necessary to input the data from the unknown domain to identify the fault types. The experimental results show that the proposed method achieves superior performance in fault diagnosis tasks involving cross-speed, time-varying speed, and small sample data in SDGFD, demonstrating that PKECA has strong generalizability. The code can be found here: https://github.com/John-520/PKECA. © 2024 Elsevier Science. All rights reserved

Data‐Knowledge Driven Hybrid Deep Learning for Earthquake Early Warning

AbstractEarthquake early warning (EEW) is of great significance in mitigating seismic disasters. Traditional EEW algorithms, which are knowledge‐driven approaches, rely on seismologists' analysis. The limited intensity measures were extracted by seismologists from P‐wave signals. And there is considerable uncertainty for predicting epicentral distance, magnitude, peak ground acceleration (PGA), and peak ground velocity (PGV). Currently, data‐driven deep learning methods with the strong learning abilities do not consider knowledge information from seismologists in EEW; thus, there is unexplored potential in enhancing the performance of deep learning models for EEW. Here, we construct the Data‐knowledge driven Hybrid deep Learning network (DHLnet) for EEW using the waveform input, knowledge embedding, convolutional neural network and graph convolutional network, aiming to integrate knowledge information from knowledge‐driven methods and the strong learning ability of data‐driven deep learning methods, that is, improving the performance of EEW. For the same test data set, compared with knowledge‐driven methods and data‐driven deep learning models, we demonstrate that DHLnet enhances the timeliness and robustness in predicting the epicentral distance, magnitude, PGA, and PGV during 10 s time window following the arrival of P‐wave. Furthermore, to validate the generalization and robustness of the DHLnet in EEW, we applied the trained DHLnet to an independent data set, within first few seconds after an earthquake occurs, DHLnet can provide robust magnitude estimation, epicentral distance estimation and high alarm accuracy. The potential of the proposed network is to enhance the performance of EEW systems and provides new insights into the exploration of deep learning methods for EEW domain.