Graph transformer with disease subgraph positional encoding for improved comorbidity prediction

Structural and positional ensembled encoding for Graph Transformer

The attention mechanism is a powerful and effective method utilized in natural language processing. This mechanism allows the model to focus on important parts of the input sequence. Transformer model utilizes attention mechanisms to replace recurrent and convolutional neural networks, which eliminates the need for increasingly complex operations as the distance between words in a sequence increases. However, this method is notably insensitive to positional information. Positional encoding is crucial for Transformer-like models that heavily rely on the attention mechanism. To make the models position-aware, the position information of the input words is typically incorporated to the input token embeddings as an additional embedding. The purpose of the paper is to conduct a systematic study to understand different position encoding methods. We briefly describe the components of the attention mechanism, its role in the Transformer model, and the encoder-decoder architecture of the Transformer. We also study how sharing position encodings across various heads and layers of a Transformer affects the model performance. Methodology of the study is based on general research methods of analysis and synthesis, experimental testing, and quantitative analysis to comprehensively examine and compare the efficacy and performance of different positional encoding techniques utilized in Transformer models. The obtained results show that using absolute and relative encodings results in similar performance for the model, while relative encodings worked much better with longer sentences. We found the original encoder-decoder form worked best for the tasks of machine translation and question answering. Despite using twice as many parameters as "encoder-only" or "decoder-only" architectures, an encoder-decoder model has a similar computational cost. Besides that, the number of learnable parameters can often be reduced without performance loss. Practical implications.Positional encoding is essential for enabling Transformer models to effectively process data by preserving sequence order, handling variable-length sequences, and improving generalization. Its inclusion significantly contributes to the success of Transformer-based architectures in various natural language processing tasks. Value/originality.Positional encoding is such a critical issue for Transformer-like models. However, it has not been explored how positional encoding establishes positional dependencies within a sequence. We chose to analyze several approaches to position encoding in the context of question answering and machine translation tasks because the influence of positional encoding on NLP models in terms of word order remains ambiguous and requires further exploration.

Confidence intervals for the receiver operating characteristic area in studies with small samples.

We present an approach for the extraction of Graph Grammars (GGs) from Java source code. A GG consists of an initial graph, describing the initial state of a system, and a set of rules, modeling the possible changes of state. We generate a GG based on execution traces collected from annotated code, following the main ideas from an existing approach for extracting Labelled Transition Systems (LTS) based on context information (combination of block of code, values of attributes, and evaluated path conditions). Since GGs are data-driven, in contrast to the action-based formalism of LTS, we have adapted the existing technique to focus on data information. The approach is partially supported by a tool and the generated GGs can serve as input to existing analysis tools. We illustrate the approach with a case study and compare the resulting GG with a GG manually created by an expert for the same system.

The use of magnetic resonance imaging (MRI) for assessment of Crohn's disease (CD) is expanding. The aim of this study is to define and provide an external validation of the MRI predictors of active CD, severe CD, and a quantitative Magnetic Resonance Index of Activity (MaRIA). In all, 48 patients with clinically active (n = 29) or inactive (n = 19) CD underwent ileocolonoscopy (reference standard) and MRI. T2-weighted and pre- and postcontrast-enhanced T1-weighted sequences were acquired. Endoscopic activity was evaluated by the Crohn's Disease Endoscopic Index of Severity (CDEIS), and also classified as absent, mild (inflammation without ulcers), or severe (presence of ulceration). In complete agreement with a previous derivation study, independent predictors of disease severity using CDEIS as a reference were wall thickness, relative contrast enhancement (RCE), presence of edema, and ulcers on MRI. Estimation of activity in each segment using this regression model, or another with simplified coefficients (MaRIA(S) = 1.5*wall thickness + 0.02*RCE + 5*edema + 10*ulceration) correlated with CDEIS (r = 0.798, P< 0.001; r = 0.80 P < 0.001, respectively). In the validation cohort both indexes had a high and equal accuracy for diagnosis of active disease: receiver operator characteristic (ROC) area 0.93, sensitivity 0.87, specificity 0.87 using a cutoff point ≥ 7, and for diagnosis of severe disease: ROC area 0.96, sensitivity 0.92, specificity 0.92 using a cutoff point ≥ 11. The total of segment values (MaRIA(T)) correlated with global CDEIS (r = 0.83, P< 0.001). The MRI variables that should be evaluated in clinical practice to diagnose active CD and severe CD are validated, as well as the quantitative index of activity for use in research studies.

This paper presents an approach for model specification in the context of Model Checking, using graph grammars. The choice for this formal method is justified by the visual language provided by Graph Grammar and the possibility of using it to check properties of a system. Our approach is based on the analysis of a graph grammar obtained from trace information generated by Java applications. The process of creating such graph grammar from execution traces follows the main ideas of an existing approach that allows the generation of labeled transition systems from Java code. The main objective of this work is to find a way to automate the process of generating a graph grammar, using the information extracted from the source code and execution traces. For this, we developed an algorithm that maps system information, such as values of variables and possible transitions between states of a program, to rules of a graph grammar. To evaluate the methodology used in our approach, we conducted two experiments with toy applications and discuss the results obtained.

The rapid development of Open‐Source Software (OSS) has resulted in a significant demand for code changes to maintain OSS. Symptoms of poor design and implementation choices in code changes often occur, thus heavily hindering code reviewers to verify correctness and soundness of code changes. Researchers have investigated how to learn meaningful code changes to assist developers in anticipating changes that code reviewers may suggest for the submitted code. However, there are two main limitations to be addressed, including the limitation of long‐range dependencies of the source code and the missing syntactic structural information of the source code. To solve these limitations, a novel method is proposed, named Graph Transformer for learning meaningful Code Transformations (GTCT), to provide developers with preliminary and quick feedback when developers submit code changes, which can improve the quality of code changes and improve the efficiency of code review. GTCT comprises two components: code graph embedding and code transformation learning. To address the missing syntactic structural information of the source code limitation, the code graph embedding component captures the types and patterns of code changes by encoding the source code into a code graph structure from the lexical and syntactic representations of the source code. Subsequently, the code transformation learning component uses the multi‐head attention mechanism and positional encoding mechanism to address the long‐range dependencies limitation. Extensive experiments are conducted to evaluate the performance of GTCT by both quantitative and qualitative analyses. For the quantitative analysis, GTCT relatively outperforms the baseline on six datasets by 210%, 342.86%, 135%, 29.41%, 109.09%, and 91.67% in terms of perfect prediction. Meanwhile, the qualitative analysis shows that each type of code change by GTCT outperforms that of the baseline method in terms of bug fixed, refactoring code and others' taxonomy of code changes.

During human pregnancy, the DNA methylation of placental tissue is highly relevant to the normal growth and development of the fetus; therefore, methylomic analysis of the placental tissue possesses high research and clinical value in prenatal testing and monitoring. Thus, our aim is to develop an approach for reconstruction of the placental methylome, which should be completely noninvasive and achieve high accuracy and resolution. We propose a novel size-based algorithm, FEtal MEthylome Reconstructor (FEMER), to noninvasively reconstruct the placental methylome by genomewide bisulfite sequencing and size-based analysis of maternal plasma DNA. By applying FEMER on a real clinical dataset, we demonstrate that FEMER achieves both high accuracy and resolution, thus provides a high-quality view of the placental methylome from maternal plasma DNA. FEtal MEthylome Reconstructor could also predict the DNA methylation profile of CpG islands with high accuracy, thus shows potential in monitoring of key genes involved in placental/fetal development. Source code and testing datasets for FEMER are available at http://sunlab.cpy.cuhk.edu.hk/FEMER/. FEtal MEthylome Reconstructor could enhance the noninvasive fetal/placental methylomic analysis and facilitate its application in prenatal testing and monitoring.

Signed Graph Neural Networks (SGNNs) have been shown to be effective in analyzing complex patterns in real-world situations where positive and negative links coexist. However, SGNN models suffer from poor explainability, which limit their adoptions in critical scenarios that require understanding the rationale behind predictions. To the best of our knowledge, there is currently no research work on the explainability of the SGNN models. Our goal is to address the explainability of decision-making for the downstream task of link sign prediction specific to signed graph neural networks. Since post-hoc explanations are not derived directly from the models, they may be biased and misrepresent the true explanations. Therefore, in this paper we introduce a Self-Explainable Signed Graph transformer (SE-SGformer) framework, which can not only outputs explainable information while ensuring high prediction accuracy. Specifically, we propose a new Transformer architecture for signed graphs and theoretically demonstrate that using positional encoding based on signed random walks has greater expressive power than current SGNN methods and other positional encoding graph Transformer-based approaches. We construct a novel explainable decision process by discovering the K-nearest (farthest) positive (negative) neighbors of a node to replace the neural network-based decoder for predicting edge signs. These K positive (negative) neighbors represent crucial information about the formation of positive (negative) edges between nodes and thus can serve as important explanatory information in the decision-making process. We conducted experiments on several real-world datasets to validate the effectiveness of SE-SGformer, which outperforms the state-of-the-art methods by improving 2.2% prediction accuracy and 73.1% explainablity accuracy in the best-case scenario.

In this work, we propose a Three-Dimensional Transmissible Attention Network (3DTANet) for Person Re-Identification, which can transmit the attention information from layer to layer and attend to the person image from a three-dimensional perspective. Main contributions of the 3DTANet are: (i) A novel Transmissible Attention (TA) mechanism is introduced, which can transfer attention information between convolution layers. Different from traditional attention mechanism, not only can it convey accumulated attention information layer by layer but also guide the network to retain holistic attention information. (ii) We propose a Three-Dimension Attention (3DA) mechanism, which is capable of extracting a three-dimensional attention map. While previous researches on image attention mechanism extracts channel or spatial attention information separately, 3DA mechanism pays attention to channel and spatial information simultaneously, thereby making them play better complementary role in attention extraction. (iii) A new loss function named L2-norm Multi-labels Loss (L2ML) is applied to acquire higher recognition accuracy calculated by multi labels of same ID and corresponding feature representation. Quite different from the common loss functions, L2-norm Multi-labels Loss is specifically good at optimizing feature distance. In brief, 3DTANet gains two-fold benefit toward higher accuracy. For one thing, the attention information is informative and can be transmitted, feature being more representative. For another, our model is computationally lightweight and can be easily applied to real scenarios. We extensively conduct experiments on four Person Re-Identification benchmark datasets. Our model achieves rank-1 accuracy of 87.50% on CUHK03, 96.23% on Market-1501, 92.50% on DukeMTMC-reID and 76.60% on MSMT17-V2 respectively. The results confirm that the 3DTANet can extract more representative features and attain a higher recognition accuracy, outperforming the state-of-the-art methods.

Graph Transformers have proved their advantages in graph data mining with elaborate Positional Encodings, especially in graph-level tasks. However, their application in the node classification task has not been fully exploited yet. In the node classification task, existing Graph Transformers with Positional Encodings are limited by the following issues: (i) PEs describing the node’s positional identities are insufficient for the node classification task on complex graphs, where a full portrayal of the local node property is needed. (ii) PEs for graphs are integrated with Transformers in a constant schema, resulting in the ignorance of local patterns that may vary among different nodes. In this paper, we propose Adaptive Graph Transformer (AGT) to tackle above issues. AGT consists of a Learnable Centrality Encoding and a Kernelized Local Structure Encoding. The two modules extract structural patterns from centrality and subgraph views in a learnable and scalable manner. Further, we design the Adaptive Transformer Block to adaptively integrate the attention scores and Structural Encodings in a node-specific manner. AGT achieves state-of-the-art performances on nine real-world web graphs (up to 1.6 million nodes). Furthermore, AGT shows outstanding results on two series of synthetic graphs with ranges of heterophily and noise ratios.

The inference of gene regulatory networks (GRNs) is critical for understanding the regulatory mechanisms underlying cellular development, functional specialization, and disease progression. Predicting regulatory gene interactions—often framed as a link prediction task—is a foundational step toward modeling cellular behavior. However, GRN inference from gene coexpression data alone is limited by noise, low interpretability, and difficulty in capturing indirect regulatory signals. Additionally, challenges such as data sparsity, nonlinearity, and complex gene interactions hinder accurate network reconstruction. To address these issues, we propose, a novel graph transformer (GT) based framework (GT-GRN) that enhances GRN inference by integrating multimodal gene embeddings. Our method combines three complementary sources of information: (i) autoencoder-based embeddings, which capture high-dimensional gene expression patterns while preserving biological signals; (ii) structural embeddings, derived from previously inferred GRNs and encoded via random walks and a Bidirectional Encoder Representations from Transformers (BERT) based language model to learn global gene representations; (iii) positional encodings, capturing each gene’s role within the network topology . These heterogeneous features are fused and processed using a GT, allowing the joint modeling of both local and global regulatory structures. Experimental results on benchmark datasets show that GT-GRN outperforms existing GRN inference methods in predictive accuracy and robustness. Furthermore, it reconstructs cell-type-specific GRNs with high fidelity and produces gene embeddings that generalize to other tasks such as cell-type annotation.

The prediction of fluid through well logging is a cornerstone in guiding exploratory efforts in the energy sector. Comprehending the fluid composition beneath the surface empowers exploration teams to effectively gauge the extent, reserves, and caliber of oil and gas resources. This leads to enhanced strategies in exploration and the judicious use of resources. We introduce an innovative machine learning framework named “Graph Transformer” for predicting fluid. This model melds graph convolutional layers with a Transformer module. It excels in decoding spatial and temporal patterns within well logging data, thus unraveling complex geological dependencies by factoring in the interconnectedness of various data points. Additionally, it features a Positional Encoding module to enhance understanding of sequential data points in terms of depth, thereby overcoming the limitations of sequence independence. The Transformer's Multi-Head Self-Attention mechanism is pivotal in discerning and integrating spatial and temporal interconnections across various depths, elevating its capability to represent geological structures. Initially, the model harnesses key well log data like Density, Acoustic, Gamma-ray, and Compensated Neutron Logs for extracting geological features. These insights are then processed through the Graph Transformer to establish relationship between fluid characteristics and logging parameters. Furthermore, we compare this model with other leading models using precision, recall, and accuracy metrics. Experimental findings affirm the model's high accuracy in predicting fluid within intricate geological settings. Its exceptional adaptability makes it apt for various geological conditions and logging tools. Thus, our Graph Transformer model stands out as a sophisticated, efficient machine learning solution in the realm of well logging fluid prediction, offering geologists and engineers precise tools for exploration and development.

Following the significant success of the transformer in NLP and computer vision, this paper attempts to extend it to 3D triangle mesh. The aim is to determine the shape’s global representation using the transformer and capture the inherent manifold information. To this end, this paper proposes a novel learning framework named Navigation Geodesic Distance Transformer (NGD-Transformer) for 3D mesh. Specifically, this approach combined farthest point sampling with the Voronoi segmentation algorithm to spawn uniform and non-overlapping manifold patches. However, the vertex number of these patches was inconsistent. Therefore, self-attention graph pooling is employed for sorting the vertices on each patch and screening out the most representative nodes, which were then reorganized according to their scores to generate tokens and their raw feature embeddings. To better exploit the manifold properties of the mesh, this paper further proposed a novel positional encoding called navigation geodesic distance positional encoding (NGD-PE), which encodes the geodesic distance between vertices relatively and spatial symmetrically. Subsequently, the raw feature embeddings and positional encodings were summed as input embeddings fed to the graph transformer encoder to determine the global representation of the shape. Experiments on several datasets were conducted, and the experimental results show the excellent performance of our proposed method.

A biomedical relation statement is commonly expressed in multiple sentences and consists of many concepts, including gene, disease, chemical and mutation. To automatically extract information from biomedical literature, existing biomedical text-mining approaches typically formulate the problem as a cross-sentence n-ary relation-extraction task that detects relations among n entities across multiple sentences, and use either a graph neural network (GNN) with long short-term memory (LSTM) or an attention mechanism. Recently, Transformer has been shown to outperform LSTM on many natural language processing (NLP) tasks. In this work, we propose a novel architecture that combines Bidirectional Encoder Representations from Transformers with Graph Transformer (BERT-GT), through integrating a neighbor-attention mechanism into the BERT architecture. Unlike the original Transformer architecture, which utilizes the whole sentence(s) to calculate the attention of the current token, the neighbor-attention mechanism in our method calculates its attention utilizing only its neighbor tokens. Thus, each token can pay attention to its neighbor information with little noise. We show that this is critically important when the text is very long, as in cross-sentence or abstract-level relation-extraction tasks. Our benchmarking results show improvements of 5.44% and 3.89% in accuracy and F1-measure over the state-of-the-art on n-ary and chemical-protein relation datasets, suggesting BERT-GT is a robust approach that is applicable to other biomedical relation extraction tasks or datasets. the source code of BERT-GT will be made freely available athttps://github.com/ncbi/bert_gt upon publication. Supplementary data are available at Bioinformatics online.