Graph Transformation Research Articles

MPCD: A Multitask Graph Transformer for Molecular Property Prediction by Integrating Common and Domain Knowledge.

Molecular property prediction with deep learning often employs self-supervised learning techniques to learn common knowledge through masked atom prediction. However, the common knowledge gained by masked atom prediction dramatically differs from the graph-level optimization objective of downstream tasks, which results in suboptimal problems. Particularly for properties with limited data, the failure to consider domain knowledge results in a direct search in an immense common space, rendering it infeasible to identify the global optimum. To address this, we propose MPCD, which enhances pretraining transferability by aligning the optimization objectives between pretraining and fine-tuning with domain knowledge. MPCD also leverages multitask learning to improve data utilization and model robustness. Technically, MPCD employs a relation-aware self-attention mechanism to capture molecules' local and global structures comprehensively. Extensive validation demonstrates that MPCD outperforms state-of-the-art methods for absorption, distribution, metabolism, excretion, and toxicity (ADMET) and physicochemical prediction across various data sizes.

Enhancing Video-Language Representations With Structural Spatio-Temporal Alignment.

While pre-training large-scale video-language models (VLMs) has shown remarkable potential for various downstream video-language tasks, existing VLMs can still suffer from certain commonly seen limitations, e.g., coarse-grained cross-modal aligning, under-modeling of temporal dynamics, detached video-language view. In this work, we target enhancing VLMs with a fine-grained structural spatio-temporal alignment learning method (namely Finsta). First of all, we represent the input texts and videos with fine-grained scene graph (SG) structures, both of which are further unified into a holistic SG (HSG) for bridging two modalities. Then, an SG-based framework is built, where the textual SG (TSG) is encoded with a graph Transformer, while the video dynamic SG (DSG) and the HSG are modeled with a novel recurrent graph Transformer for spatial and temporal feature propagation. A spatial-temporal Gaussian differential graph Transformer is further devised to strengthen the sense of the changes in objects across spatial and temporal dimensions. Next, based on the fine-grained structural features of TSG and DSG, we perform object-centered spatial alignment and predicate-centered temporal alignment respectively, enhancing the video-language grounding in both the spatiality and temporality. We design our method as a plug&play system, which can be integrated into existing well-trained VLMs for further representation augmentation, without training from scratch or relying on SG annotations in downstream applications. On 6 representative VL modeling tasks over 12 datasets in both standard and long-form video scenarios, Finsta consistently improves the existing 13 strong-performing VLMs persistently, and refreshes the current state-of-the-art end task performance significantly in both the fine-tuning and zero-shot settings.

GACT-PPIS: Prediction of protein-protein interaction sites based on graph structure and transformer network

The prediction of protein-protein interaction sites (PPIS) is currently crucial for regulating many biological activities in cells and developing drugs for various diseases. Deep learning-based methods have been proposed for predicting PPIS, significantly reducing the manpower and time costs associated with traditional experimental methods such as yeast two-hybrid, mass spectrometry, and affinity purification. However, the predictive accuracy of these deep learning methods has not yet reached the expected level. Therefore, we introduce a model called GACT-PPIS.The design of the GACT-PPIS algorithm aims to utilize combined information from protein sequences and structures as input to predict protein-protein interaction sites. The core of GACT-PPIS utilizes an Enhanced Graph Attention Network (EGAT) with initial residual and identity mappings, along with a deep Transformer network as the basic units, supplemented by Graph Convolutional Networks (GCN), effectively aggregating information from neighboring nodes for each node. After multiple network layers, the information of the entire protein is also fused into the nodes, and the Transformer network further enhances the model's performance.Experimental results show that GACT-PPIS outperforms the most representative models in terms of Recall, F1-measure, MCC, AUROC, and AUPRC on the benchmark test set (Test-60). Additionally, on other independent test sets (UBTest-31-6), GACT-PPIS leads in terms of Accuracy, Precision, Recall, F1-measure, MCC, AUROC, and AUPRC compared to the most representative models. It is worth noting that GACT-PPIS demonstrates excellent generalization and versatility across different test sets, showcasing good performance on multiple test sets for the same trained GACT-PPIS model.

A LLM-driven and motif-informed linearizing graph transformer for Web API recommendation

The rapid growth in the number of Web APIs makes it difficult for developers to find the appropriate ones. To tackle this issue, researchers have created various powerful automatic approaches for recommendations. Recently, a range of graph neural networks, drawing inspiration from Transformers, have incorporated global attention to enhance recommendations. However, these approaches still have limitations in terms of processing information between nodes and utilizing other valid information, which reduces their effectiveness in large-scale Web API recommendations. To tackle these problems, this paper introduces a novel positional and semantic encoding method and motif-based linearizing graph Transformer for automatic Web API recommendation. We integrate the semantic information of the nodes into the positional information by using a fine-tuned large language model and graph attention mechanism. Furthermore, we leverage motif information to alter the computational sequence of the Vanilla Transformer, achieving linear time complexity. Experimental results on the two real-world datasets demonstrate the suitability of our model for Web API recommendation, surpassing existing state-of-the-art methods. In summary, our proposed technique exhibits promising results for Web API recommendation and underscores the potential of global attention in this field.

A heterogeneous graph transformer framework for accurate cancer driver gene prediction and downstream analysis

Accurately predicting cancer driver genes remains a formidable challenge amidst the burgeoning volume and intricacy of cancer genomic data. In this investigation, we propose HGTDG, an innovative heterogeneous graph transformer framework tailored for precisely predicting cancer driver genes and exploring downstream tasks. A heterogeneous graph construction module is central to the framework, which assembles a gene-protein heterogeneous network leveraging the Kyoto Encyclopedia of Genes and Genomes (KEGG) pathways and protein-protein interactions sourced from the STRING (search tool for recurring instances of neighboring genes) database. Moreover, our framework introduces a pioneering heterogeneous graph transformer module, harnessing multi-head attention mechanisms for nuanced node embedding. This transformative module proficiently captures distinct representations for both nodes and edges, thereby enriching the model's predictive capacity. Subsequently, the generated node embeddings are seamlessly integrated into a classification module, facilitating the discrimination between driver and non-driver genes. Our experimental findings evince the superiority of HGTDG over existing methodologies, as evidenced by the enhanced performance metrics, including the area under the receiver operating characteristic curves (AUROC) and the area under the precision-recall curves (AUPRC). Furthermore, the downstream analysis utilizing the newly identified cancer driver genes underscores the efficacy and versatility of our proposed framework.

A gradient-enhanced univariate dimension reduction method for uncertainty propagation

The univariate dimension reduction (UDR) method stands as a way to estimate the statistical moments of the output that is effective in a large class of uncertainty quantification (UQ) problems. UDR's fundamental strategy is to approximate the original function using univariate functions so that the UQ cost only scales linearly with the dimension of the problem. Nonetheless, UDR's effectiveness can diminish when uncertain inputs have high variance, particularly when assessing the output's second and higher-order statistical moments. This paper proposes a new method, gradient-enhanced univariate dimension reduction (GUDR), that enhances the accuracy of UDR by incorporating univariate gradient function terms into the UDR approximation function. Theoretical results indicate that the GUDR approximation is expected to be one order more accurate than UDR in approximating the original function, and it is expected to generate more accurate results in computing the output's second and higher-order statistical moments. Our proposed method uses a computational graph transformation strategy to efficiently evaluate the GUDR approximation function on tensor-grid quadrature inputs, and use the tensor-grid input-output data to compute the statistical moments of the output. With an efficient automatic differentiation method to compute the gradients, our method preserves UDR's linear scaling of computation time with problem dimension. Numerical results show that the GUDR is more accurate than UDR in estimating the standard deviation of the output and has a performance comparable to the method of moments using a third-order Taylor series expansion.

Graph-accelerated non-intrusive polynomial chaos expansion using partially tensor-structured quadrature rules for uncertainty quantification

Recently, the graph-accelerated non-intrusive polynomial chaos (NIPC) method has been proposed for solving uncertainty quantification (UQ) problems. This method leverages the full-grid integration-based NIPC method to address UQ problems while employing the computational graph transformation approach, AMTC, to accelerate the tensor-grid evaluations. This method exhibits remarkable efficacy on a broad range of low-dimensional (three dimensions or less) UQ problems featuring multidisciplinary models. However, it often does not scale well with problem dimensions due to the exponential increase in the number of quadrature points when using the full-grid quadrature rule. To expand the applicability of this method to a broader range of UQ problems, this paper introduces a new framework for generating a tailored, partially tensor-structured quadrature rule to use with the graph-accelerated NIPC method. This quadrature rule, generated through the designed quadrature approach, possesses a tensor structure that is tailored for the computational model. The selection of the tensor structure is guided by an analysis of the computational graph, ensuring that the quadrature rule effectively capitalizes on the sparsity within the computational graph when paired with the AMTC method. This method has been tested on one 4D and one 6D UQ problem, both originating from aircraft design scenarios and featuring multidisciplinary models. Numerical results show that, when using with graph-accelerated NIPC method, our approach generates a partially tensor-structured quadrature rule that outperforms the full-grid Gauss quadrature and the designed quadrature methods (more than 40% reduction in computational costs) in both of the test problems.

Lane extraction from trajectories at road intersections based on Graph Transformer Network

Lane-level road networks are crucial components of high-precision maps and play a significant role in intelligent transportation systems. Extracting lane-level road networks at intersections presents considerable challenges due to the complex structures of intersections and diverse driving behaviors. A graph-learning based method is proposed for extracting lanes from high-precision trajectories at road intersections. A trajectory relation graph is designed to encode the directional, shape, and distance features of trajectories, capturing both the intrinsic and extrinsic relationships between trajectories. Subsequently, a Graph Transformer Network is developed to extract a representative subset of trajectories as lanes. To alleviate the problem of generating missing and extraneous lanes, a set-based lane extraction loss is introduced to achieve implicit pruning of redundancy through the attention mechanism. Comprehensive experimental results demonstrate that the proposed method outperforms state-of-the-art methods in three positional and topological accuracy metrics. The method achieves lane extraction with minimal omissions and redundancies and exhibits strong performance in complex scenarios such as U-turns, lane merging, and lane diverging regions.

GT-NMR: a novel graph transformer-based approach for accurate prediction of NMR chemical shifts

In this work, inspired by the graph transformer, we presented an improved protocol, termed GT-NMR, which integrates 2D molecular graph representation with Transformer architecture, for accurate yet efficient prediction of NMR chemical shifts. The effectiveness of the GT-NMR was thoroughly examined with the standard nmrshiftdb2 dataset, 37 natural products and structural elucidation of 11 pairs of natural products. Systematical analysis affirms that GT-NMR outperforms traditional graph-based methods in all aspects, achieving state-of-the-art performance, with the mean absolute error of 0.158 and 1.189 ppm in predicting 1H and 13C NMR chemical shifts, respectively, for the standard nmrshiftdb2 dataset. Further scrutiny of its practical applications indicates that GT-NMR's efficacy is closely tied to molecular complexity, as quantified by the size-normalized spatial score (nSPS). For relatively simple molecules (nSPS < = 27.71), GT-NMR performs comparably to the best density functional while its effectiveness significantly diminishes with complex molecules characterized by higher nSPS values (nSPS > = 38.42). This trend is consistent across other graph-based NMR chemical shift prediction methods as well. Therefore, while employing GT-NMR or other graph-based methods for the rapid and routine prediction of NMR chemical shifts, it is advisable to utilize nSPS to assess their suitability. The source codes and trained model of GT-NMR are publicly available at GitHub.Scientific contributionGT-NMR, which combines the 2D molecular graph representation with the Transformer architecture, was implemented for the first time to predict atom-level NMR chemical shifts, achieving state-of-the-art performance. More importantly, the reliability of the GT-NMR and graph-based methods was assessed for the first time in terms of molecular complexity, as quantified by the size-normalized spacial score (nSPS). Systematical scrutiny demonstrated that GT-NMR offer a valuable way for routine application in structural screening and elucidation of relatively simple molecules.

Directed Knowledge Graph Embedding Using a Hybrid Architecture of Spatial and Spectral GNNs

Knowledge graph embedding has been identified as an effective method for node-level classification tasks in directed graphs, the objective of which is to ensure that nodes of different categories are embedded as far apart as possible in the feature space. The directed graph is a general representation of unstructured knowledge graphs. However, existing methods lack the ability to simultaneously approximate high-order filters and globally pay attention to the task-related connectivity between distant nodes for directed graphs. To address this limitation, a directed spectral graph transformer (DSGT), a hybrid architecture model, is constructed by integrating the graph transformer and directed spectral graph convolution networks. The graph transformer leverages multi-head attention mechanisms to capture the global connectivity of the feature graph from different perspectives in the spatial domain, which bridges the gap between frequency responses and, further, naturally couples the graph transformer and directed graph convolutional neural networks (GCNs). In addition to the inherent hard inductive bias of DSGT, we introduce directed node positional and structure-aware edge embedding to provide topological prior knowledge. Extensive experiments demonstrate that the DSGT exhibits state-of-the-art (SOTA) or competitive node-level representation capabilities across datasets of varying attributes and scales. Furthermore, the experimental results indicate that the homophily and degree of correlation of the nodes significantly influence the classification performance of the model. This finding opens significant avenues for future research.

CrossFormer: Cross-Modal Representation Learning via Heterogeneous Graph Transformer

Transformers have been recognized as powerful tools for various cross-modal tasks due to their superior ability to perform representation learning through self-attention. Existing transformer-based cross-modal models can be categorized into single-stream and dual-stream ones. By performing fine-grained interaction with self-attention on the cross-modal concatenated features, the former can simultaneously learn intra- and inter-modal correlations. However, this simple concatenation treats the inputs of different modalities equally; as a result, the heterogeneous differences between modalities are ignored, leading to a modality gap. The latter process the inputs of different modalities separately, then perform cross-modal interaction on the subsequently fused networks, resulting in a failure to integrate the fine-grained correlations of both intra- and inter-modality in a uniform module. To this end, we propose an effective heterogeneous graph transformer for dual-stream cross-modal representation learning, named CrossFormer, which constructs a heterogeneous graph as a bridge to achieve fine-grained intra- and inter-modal interaction on a dual-stream network. Specifically, we first represent multi-modal data with a heterogeneous graph, then develop a dual-positional encoding strategy that enables the heterogeneous graph to obtain the relative positional information. Finally, a dual-stream self-attention is performed on the heterogeneous graph, bridging the gap between modalities and effectively capturing fine-grained intra- and inter-modal interactions simultaneously. Extensive experiments on various cross-modal tasks demonstrate the superiority of our method.

A Mixed Neuro Graph Approach with Gradient Boosting to Hybrid Job-Shop Scheduling to Minimize a Regular Function of Job Completion Times and Numbers of Used Machines

The paper considers a multi-stage processing system including sets of identical (parallel) machines and a set of dedicated machines processing different operations of the given jobs in any sectors of economy. Based on the weighted Mixed Neuro graph model, the paper proposes adaptive algorithms for solving this problem via appropriate Mixed Neuro graph transformations. The main novelty is (1) low demands on the source data-unlike classical machine learning algorithms, the approach can offer stable interpretable results even with a short dataset size; (2) the number of new matrix multiplication operations that make up the main load when training models increases linearly with the number of new data from 0 to 999 time periods; (3) the results of the model are repeatable due to the stability of the coefficients of the model. These algorithms are able to solve (exactly or heuristically) the tested instances with N jobs and W types of parallel identical machines within on the personal computer. The gradient boosting result is in interval 5.9677410-3.4982093.

Incremental model transformations with triple graph grammars for multi-version models and multi-version pattern matching

AbstractLike conventional software projects, projects in model-driven software engineering require adequate management of multiple versions of development artifacts, importantly allowing living with temporary inconsistencies. In previous work, we have introduced multi-version models for model-driven software engineering, which allow checking well-formedness and finding merge conflicts for multiple versions of the same model at once. However, situations where different models are linked via automatic model transformations also have to be handled for multi-version models. In this paper, we propose a technique for jointly handling the transformation of multiple versions of a source model into corresponding versions of a target model. This enables the use of a more compact representation that may afford improved execution time of both the transformation and further analysis. Our approach is based on the well-known formalism of triple graph grammars and the aforementioned encoding of model version histories called multi-version models. In addition to batch transformation of an entire history, the technique covers incremental synchronization of changes in the framework of multi-version models. Our solution is complemented by a dedicated pattern matching technique for multi-version models. We show the correctness of our approach with respect to the standard semantics of triple graph grammars and conduct an empirical evaluation to investigate the performance of our technique regarding execution time and memory consumption. Our results indicate that the proposed solution affords lower memory consumption and may improve execution time for batch transformation of large version histories, but can also come with computational overhead in unfavorable cases.

Termination of Graph Transformation Systems Using Weighted Subgraph Counting

We introduce a termination method for the algebraic graph transformation framework PBPO+, in which we weigh objects by summing a class of weighted morphisms targeting them. The method is well-defined in rm-adhesive quasitoposes (which include toposes and therefore many graph categories of interest), and is applicable to non-linear rules. The method is also defined for other frameworks, including SqPO and left-linear DPO, because we have previously shown that they are naturally encodable into PBPO+ in the quasitopos setting. We have implemented our method, and the implementation includes a REPL that can be used for guiding relative termination proofs.

Towards harmonization of SO(3)-equivariance and expressiveness: a hybrid deep learning framework for electronic-structure Hamiltonian prediction

Abstract Deep learning for predicting the electronic-structure Hamiltonian of quantum systems necessitates satisfying the covariance laws, among which achieving SO(3)-equivariance without sacrificing the non-linear expressive capability of networks remains unsolved. To navigate the harmonization between SO(3)-equivariance and expressiveness, we propose HarmoSE, a deep learning method synergizing two distinct categories of neural mechanisms as a two-stage encoding and regression framework. The first stage corresponds to group theory-based neural mechanisms with inherent SO(3)-equivariant properties prior to the parameter learning process, while the second stage is characterized by a non-linear 3D graph Transformer network we propose, featuring high capability on non-linear expressiveness. Their combination lies in the point that, the first stage predicts baseline Hamiltonians with abundant SO(3)-equivariant features extracted, assisting the second stage in empirical learning of equivariance; and in turn, the second stage refines the first stage’s output as a fine-grained prediction of Hamiltonians using powerful non-linear neural mappings, compensating for the intrinsic weakness on non-linear expressiveness capability of mechanisms in the first stage. Our method enables precise, generalizable predictions while capturing SO(3)-equivariance under rotational transformations, and achieves state-of-the-art performance in Hamiltonian prediction tasks under multiple mean absolute error (MAE) metrics, such as the average MAE across all samples and matrix elements, the MAE for challenging samples, the MAE for different Hamiltonian blocks, and the MAE for the challenging blocks. It also demonstrates significant improvements in accuracy for downstream quantities, such as occupied orbital energy and the electronic wavefunction, as measured by MAE and cosine similarity, respectively.

Category-Based Administrative Access Control Policies

As systems evolve, security administrators need to review and update access control policies. Such updates must be carefully controlled due to the risks associated with erroneous or malicious policy changes. We propose a category-based access control (CBAC) model, called Admin-CBAC , to control administrative actions. Since most of the access control models in use nowadays (including the popular RBAC and ABAC models) are instances of CBAC, from Admin-CBAC , we derive administrative models for RBAC and ABAC, too. We present a graph-based representation of Admin-CBAC policies and a formal operational semantics for administrative actions via graph rewriting. We also discuss implementations of Admin-CBAC exploiting the graph-based representation. Using the formal semantics, we show how properties (such as safety, liveness, and effectiveness of policies) and constraints (such as separation of duties) can be checked, and discuss the impact of policy changes. Although the most interesting properties of policies are generally undecidable in dynamic access control models, we identify particular cases where reachability properties are decidable and can be checked using our operational semantics, generalising previous results for RBAC and ABAC α .

Research on lane detection algorithms based on GTNs

Abstract. Lane detection technology plays an important role in the lane departure warning and adaptive cruise control functions of autonomous vehicle systems. Traditional lane line detection includes experimental steps such as preyreatment, color space conversion, feature extraction, and lane line tracking, but there are still some problems such as recognition accuracy in complex environments and parameter adjustment dependency. To overcome these limitations, this study uses a novel method that can directly predict the parameters of the lane shape model, thus avoiding complex post-processing steps. This research method uses the network structure based on Graph Transformer Networks (GTNS) to capture richer structural information and contextual relationships, thereby improving the accuracy of detection. In terms of feature extraction, Graph Transformer Networks is used to accurately capture the structural information of the lane through the attention mechanism. At the same time, the results in the later stage show that our method has better advantages in accuracy and speed. In short, this optimization scheme not only improves the detection speed but also ensures a certain degree of accuracy. The method will perform better in the future after further optimization such as multi-sensor fusion and real-time optimization.

ScGraphformer: unveiling cellular heterogeneity and interactions in scRNA-seq data using a scalable graph transformer network

The precise classification of cell types from single-cell RNA sequencing (scRNA-seq) data is pivotal for dissecting cellular heterogeneity in biological research. Traditional graph neural network (GNN) models are constrained by reliance on predefined graphs, limiting the exploration of complex cell-to-cell relationships. We introduce scGraphformer, a transformer-based GNN that transcends these limitations by learning an all-encompassing cell-cell relational network directly from scRNA-seq data. Through an iterative refinement process, scGraphformer constructs a dense graph structure that captures the full spectrum of cellular interactions. This comprehensive approach enables the identification of subtle and previously obscured cellular patterns and relationships. Evaluated on multiple datasets, scGraphformer demonstrates superior performance in cell type identification compared to existing methods and showcases its scalability with large-scale datasets. Our method not only provides enhanced cell type classification ability but also reveals the underlying cell interactions, offering deeper insights into functional cellular relationships. The scGraphformer thus holds the potential to significantly advance the field of single-cell analysis and contribute to a more nuanced understanding of cellular behavior.

Baseline-free assisted lamb wave-based damage detection in CFRP composites using graph convolutional networks and Transformer models

Carbon fiber-reinforced polymer (CFRP) composites are widely used in aerospace but are susceptible to damage, threatening structural safety. Thus, developing an effective structural health monitoring system is essential. This study presents CFRP-former, a novel deep learning framework that detects damage in CFRP composites by constructing spatio-temporal representation graphs of Lamb waves (LWs). The CFRP-former comprises three key modules: graph convolutional neural network (GCNN), a multi-layer perceptron (MLP), and a Transformer Encoder. The GCNN models the topological relationships between LW nodes and sensors. The MLP reduces data dimensionality while preserving key information for subsequent analysis. Meanwhile, the Transformer Encoder, employing multi-head attention mechanisms, captures global time-series patterns in the LW signals. Additionally, the CFRP-former framework includes a parallel module that separately predicts both the size and location of damage. Experiments using only four sensors show that CFRP-former achieves high accuracy in detecting and quantifying damage, with robustness confirmed through ablation tests.

Data-driven multivariate time series prediction of in-vehicle equipment failure rates

Effectively predicting the failure rate of train-controlled on-board equipment is of great significance for rationally allocating equipment spares, drawing up maintenance plans, and reducing the occurrence of failures. In order to tackle the problem of the intricate attributes and limited predictive precision of the sample data pertaining to the failure rate of on-board equipment, in this paper, a multivariate time series prediction model for the failure rate of train-controlled on-board equipment is based on the combination of multivariate variational modal decomposition (MVMD) graph neural network (GNN) and transformer. First, the original failure rate time series, air temperature, humidity and sand and dust time series are modally decomposed using MVMD, and the intrinsic modal functions (IMFs) of each series are obtained; then, the dynamic graph of the GNN network is defined according to the IMFs, and Transformer’s self-attention mechanism is utilised to capture the dynamic graph’s temporal and spatial dependencies. Finally, the failure rate is output through the feed-forward neural network prediction value. Experiments using the historical fault data of CTCS3-300T train-controlled on-board equipment are carried out to confirm the efficacy of the suggested method, and it is contrasted with other conventional machine learning techniques. The outcomes of the experiment show that compared with other train-controlled on-board equipment failure prediction models, the proposed method has a very good superiority, as evidenced by its mean absolute error (MAE) of 0.0489 and root mean square error (RMSE) of 0.0510, which is of certain reference value for equipment operation and maintenance.