Node Embedding Research Articles

HGNN−BRFE: Heterogeneous Graph Neural Network Model Based on Region Feature Extraction

With the strong capability of heterogeneous graphs in accurately modeling various types of nodes and their interactions, they have gradually become a research hotspot, promoting the rapid development of the field of heterogeneous graph neural networks (HGNNs). However, most existing HGNN models rely on meta−paths for feature extraction, which can only utilize part of the data from the graph for training and learning. This not only limits the data generalization ability of deep learning models but also affects the application effect of data−driven adaptive technologies. In response to this challenge, this study proposes a new model—heterogeneous graph neural network based on regional feature extraction (HGNN−BRFE). This model enhances performance through an “extraction−fusion” strategy in three key aspects: first, it efficiently extracts features of neighboring nodes of the same type according to specific regions; second, it effectively fuses information from different regions and hierarchical neighbors using attention mechanisms; third, it specially designs a process for feature extraction and fusion targeting heterogeneous type nodes, ensuring that the rich semantic and heterogeneity information within the heterogeneous graph is retained while maintaining the node’s own characteristics during the node embedding process to prevent the loss of its own features and potential over−smoothing issues. Experimental results show that HGNN−BRFE achieves a performance improvement of 1–3% over existing methods on classification tasks across multiple real−world datasets.

Multi-view graph contrastive representation learning for bundle recommendation

Bundle recommendation can recommend a collection of associated items that can be consumed together to a user rather than recommending these items separately, making it extremely suitable for some scenarios such as product bundle recommendation and game bundle recommendation. Recent bundle recommendation approaches consider auxiliary data to mitigate sparse user-bundle interactions. However, these approaches obtain the node embeddings directly from the established user-bundle graph and do not explicitly exploit the relationships between users (bundles) when constructing recommendation models. Moreover, bundle recommendation approaches based on graph contrastive learning usually construct contrastive views by randomly discarding nodes (edges) in the graph, while discarding some essential nodes or edges will destroy the structure of the original graph, thereby deteriorating the quality of the learned node embeddings. Aiming at these limitations, we propose a bundle recommendation approach based on multi-view graph contrastive representation learning. First, we present a multi-view modeling method to model the relations between entities as several views from different perspectives. These views serve as inputs of graph neural networks for graph representation learning and provide contrastive views for the contrastive learning tasks. Second, we propose a novel framework for bundle recommendation. This framework obtains the user (bundle) embeddings from different views by performing multi-view graph representation learning and enhances the learned user and bundle embeddings through a two-level contrastive learning strategy. On this basis, the enhanced user (bundle) embeddings are fused for prediction. Finally, we design a joint optimization objective to optimize the model parameters, combining the prediction loss that supports multiple negative samples and the contrastive losses. Experiments on the Netease and Youshu datasets reveal that our approach outperforms the state-of-the-art (SOTA) baselines. Furthermore, the average improvements of Recall@K and NDCG@K of our approach over the SOTA baselines are approximately 3.38% and 2.80% on Netease and 3.94% and 4.84% on Youshu.

Heterogeneous graph representation learning via mutual information estimation for fraud detection

In the fraud detection, fraudsters frequently engage with numerous benign users to disguise their activities. Consequently, the fraud graph exhibits not only homogeneous connections between the fraudsters and the same labelled nodes, but also heterogeneous connections, where fraudsters interact with the legitimate nodes. Heterogeneous graph representation learning aims at extracting the structural and semantic information and embed it into the low-dimensional node representation. Recently, maximizing the mutual information between the local node embedding and the summary representation has achieved the promising results on node classification tasks. However, existing deep graph infomax methods still have the following limitations. Firstly, attribute information of nodes in the graph is not fully utilized for capturing the semantic relationships between nodes. Secondly, the local and global supervision signal are not simultaneously exploited for the node embedding learning. Thirdly, the multiplex heterogeneous relations among nodes are ignored. To address these issues, a heterogeneous graph representation learning model by mutual information estimation (MIE-HetGRL) is proposed in this paper to identify the fraudsters in the fraud review graph. Concretely, a high-order mutual information estimation is proposed to integrate the local and global mutual information as the supervision signal. Then we devise a semantic attention fusion module to aggregate the relation-aware node embeddings into a compact node representation. Finally, a joint contrastive learning is designed for facilitating the training and optimization of model. The experimental results show that our proposed model significantly outperforms state-of-the-art baselines for fraud detection.

Heterogeneous Graph Neural Network with Hierarchical Attention for Group-Aware Paper Recommendation in Scientific Social Networks

In recent years, the academic groups established in Scientific Social Networks (SSNs) have not only facilitated collaboration among researchers but also enriched the relations in SSNs, providing valuable information for paper recommendation tasks. However, existing paper recommendation methods rarely consider group information and they fail to fully leverage the group information due to the heterogeneous and complex relations between researchers, papers, and groups. In this paper, a heterogeneous graph neural network with hierarchical attention, named HHA-GPR, is proposed for group-aware paper recommendation. Firstly, a heterogeneous graph is constructed based on the interactions of researchers, papers, and groups in SSNs. Secondly, a random walk-based sampling strategy is utilized to sample highly correlated heterogeneous neighbors for researchers and papers. Thirdly, a hierarchical attention network with intra-type and inter-type attention mechanisms is designed to aggregate the sampled neighbors and comprehensively model the complex relations among the heterogeneous neighbors. More specifically, an intra-type attention mechanism is introduced to aggregate the neighbors of the same type, and an inter-type attention mechanism is employed to combine the embeddings of different types to form the ultimate node embedding. Extensive experiments are conducted on the real-world CiteULike and AMiner datasets, and the experimental results demonstrate that our proposed method outperforms other benchmark methods with an average improvement of 5.3% in Precision, 5.6% in Recall, and 5.1% in Normalized Discounted Cumulative Gain (NDCG) across both datasets.

Anomaly Detection System Based on Graph Neural Network and Node Embedding for Cybersecurity

Anomaly Detection System Based on Graph Neural Network and Node Embedding for Cybersecurity

EEG-based patient-specific seizure prediction based on Spatial–Temporal Hypergraph Attention Transformer

Background and Objectives:Patient-specific seizure prediction based on Electroencephalogram (EEG) is a challenging and significant neuromedicine study, which could potentially safeguard drug-resistant patients from unforeseen risks in their daily lives. However, prevalent seizure prediction frameworks based on convolutional neural networks (CNNs) prioritize local information while overlooking long-range global dependencies and high-order spatial correlations prevalent among brain leads. Methods:To address the limitations above, we propose a novel Spatial–Temporal Hypergraph Attention Transformer (STHAT) framework for extracting the key features in EEG signals. Aimed at the characteristics of epilepsy EEG data, our framework comprises two primary branches: (1) Long-range Temporal Feature Dependencies: This branch aims to capture temporal dependencies within EEG signals by leveraging Swin Transformer and 2D-CNN. Utilizing shifting windows and convolution operations, we encode local and global content dependencies within segmented EEG samples. (2) High-Order Spatial Information Correlations: We construct a raw graph structure using the Pearson correlation coefficient, enabling the extraction of initial node embeddings through graph convolution operations (GCO). Subsequently, a dynamic hypergraph is formed using K-Nearest Neighbors (KNN) and K-means clustering, allowing us to employ a Hypergraph Attention Network (HAN) to capture structural context relationships among brain regions. By integrating information from both branches, we utilize a linear layer to derive the final seizure prediction outcome. Experiment and Results:Extensive evaluations are conducted on the widely used CHB-MIT dataset. Our comprehensive experimental results demonstrate the effectiveness of the proposed framework in achieving superior performance in automatic seizure data prediction, outperforming state-of-the-art algorithms. The STHAT framework effectively exploits both temporal and spatial information within EEG signals, enhancing seizure prediction accuracy.

Maximizing Influence in Social Networks Using Combined Local Features and Deep Learning-Based Node Embedding.

The influence maximization problem has several issues, including low infection rates and high time complexity. Many proposed methods are not suitable for large-scale networks due to their time complexity or free parameter usage. To address these challenges, this article proposes a local heuristic called Embedding Technique for Influence Maximization (ETIM) that uses shell decomposition, graph embedding, and reduction, as well as combined local structural features. The algorithm selects candidate nodes based on their connections among network shells and topological features, reducing the search space and computational overhead. It uses a deep learning-based node embedding technique to create a multidimensional vector of candidate nodes and calculates the dependency on spreading for each node based on local topological features. Finally, influential nodes are identified using the results of the previous phases and newly defined local features. The proposed algorithm is evaluated using the independent cascade model, showing its competitiveness and ability to achieve the best performance in terms of solution quality. Compared with the collective influence global algorithm, ETIM is significantly faster and improves the infection rate by an average of 12%.

Trustworthiness-Driven Graph Convolutional Networks for Signed Network Embedding

The problem of representing nodes in a signed network as low-dimensional vectors, known as signed network embedding (SNE), has garnered considerable attention in recent years. While several SNE methods based on graph convolutional networks (GCNs) have been proposed for this problem, we point out that they significantly rely on the assumption that the decades-old balance theory always holds in the real-world. To address this limitation, we propose a novel GCN-based SNE approach, named as TrustSGCN, which corrects for incorrect embedding propagation in GCN by utilizing the trustworthiness on edge signs for high-order relationships inferred by the balance theory. The proposed approach consists of three modules: (M1) generation of each node’s extended ego-network; (M2) measurement of trustworthiness on edge signs; and (M3) trustworthiness-aware propagation of embeddings. Specifically, TrustSGCN leverages topological information to measure trustworthiness on edge sign for high-order relationships inferred by balance theory. It then considers structural properties inherent to an input network, such as the ratio of triads, to correct for incorrect embedding propagation. Furthermore, TrustSGCN learns the node embeddings by leveraging two well-known social theories, i.e., balance and status, to jointly preserve the edge sign and direction between nodes connected by existing edges in the embedding space. The experiments on six real-world signed network datasets demonstrate that TrustSGCN consistently outperforms six state-of-the-art GCN-based SNE methods. The code is available at https://github.com/kmj0792/TrustSGCN .

Cluster-guided denoising graph auto-encoder for enhanced traffic data imputation and fault detection

In an era of increasing digitalization, the integration of diverse sensors has become commonplace. Within the transportation sector, state Departments of Transportation have extensively deployed traffic sensors to support various engineering endeavors. However, these sensors are prone to faults such as data loss, random noise, bias, and drift due to sensor aging, defects, or environmental influences. Therefore, detecting these faults is imperative to maintain data integrity. This paper presents a novel approach for detecting faults in traffic data based on a cluster-guided data reconstruction process. First, a traffic sensor clustering module is designed using a dual-encoding attention graph auto-encoder (DA-GAE) to identify clusters of traffic sensors. This module leverages a joint embedding of node and edge features in a low-dimensional vector space. Subsequently, utilizing the identified clusters, a cluster-guided denoising graph auto-encoder (CG-DGAE) is devised and trained for data reconstruction. The CG-DGAE employs a diffusion graph convolutional network (DGCN) and is trained with a cluster-wise sampling strategy. Extensive experiments were conducted using traffic data obtained from a real-world large sensor network, demonstrating superior performance of the CG-DGAE model in data reconstruction compared to various baseline methods. For fault detection, a score function is devised to discern potential faults by contrasting the sensor data sequence and the reconstructed data sequence. The fault detection results show that our proposed CG-DGAE model achieved an impressive overall accuracy of 99.09%, a precision of 99.13%, a recall of 99.53%, and a F1 score of 99.53%. The superior capability of CG-DGAE in data reconstruction and fault detection is attributable to the cluster-wise spatial–temporal context leveraged by the model.

Research on Heterogeneous Information Network Link Prediction Based on Representation Learning

A heterogeneous information network, which is composed of various types of nodes and edges, has a complex structure and rich information content, and is widely used in social networks, academic networks, e-commerce, and other fields. Link prediction, as a key task to reveal the unobserved relationships in the network, is of great significance in heterogeneous information networks. This paper reviews the application of presentation-based learning methods in link prediction of heterogeneous information networks. This paper introduces the basic concepts of heterogeneous information networks, and the theoretical basis of representation learning, and discusses the specific application of the deep learning model in node embedding learning and link prediction in detail. The effectiveness and superiority of these methods on multiple real data sets are demonstrated by experimental verification.

Plant lncRNA-miRNA Interaction Prediction Based on Counterfactual Heterogeneous Graph Attention Network.

Identifying interactions between long non-coding RNAs (lncRNAs) and microRNAs (miRNAs) provides a new perspective for understanding regulatory relationships in plant life processes. Recently, computational methods based on graph neural networks (GNNs) have been widely employed to predict lncRNA-miRNA interactions (LMIs), which compensate for the inadequacy of biological experiments. However, the low-semantic and noise of graph limit the performance of existing GNN-based methods. In this paper, we develop a novel Counterfactual Heterogeneous Graph Attention Network (CFHAN) to improve the robustness to against the noise and the prediction of plant LMIs. Firstly, we construct a real-world based lncRNA-miRNA (L-M) heterogeneous network. Secondly, CFHAN utilizes the node-level attention, the semantic-level attention, and the counterfactual links to enhance the node embeddings learning. Finally, these embeddings are used as inputs for Multilayer Perceptron (MLP) to predict the interactions between lncRNAs and miRNAs. Evaluating our method on a benchmark dataset of plant LMIs, CFHAN outperforms five state-of-the-art methods, and achieves an average AUC and average ACC of 0.9953 and 0.9733, respectively. This demonstrates CFHAN's ability to predict plant LMIs and exhibits promising cross-species prediction ability, offering valuable insights for experimental LMI researches.

Heterogeneous graph contrastive learning with adaptive data augmentation for semi‐supervised short text classification

AbstractShort text classification has been widely used in many fields. Due to the scarcity of labelled data, implementing short text classification under semi‐supervised learning setting has become increasingly popular. Semi‐supervised short text classification methods based on graph neural networks can achieve state‐of‐the‐art classification performance by utilizing the expressive power of graph neural networks. However, these methods usually fail to mine the hidden patterns of a large amount of short text node data in the graph to optimize the short text node embeddings, which limits the semantic representation power of the short texts, thus leading to suboptimal classification performance. To overcome the limitation, this paper proposes a novel semi‐supervised short text classification method called the Heterogeneous Graph Contrastive Learning with Adaptive Data Augmentation (HGCLADA). In the knowledge bases guided soft prompt‐based data augmentation component, the related words of the tag words are used to optimize the soft prompts for generating diverse augmented samples. In the heterogeneous graph contrastive learning framework component, a heterogeneous graph that is constructed using short texts and keywords and an effective edge augmentation scheme based on a short text clustering algorithm are proposed. The optimized short text embeddings can be obtained to achieve the effective semi‐supervised short text classification. Extensive experiments on six benchmark datasets show that our HGCLADA method outperforms four classes of state‐of‐the‐art methods in terms of classification accuracy, especially with significant performance improvements of 8.74% on the TagMyNews dataset when each class only contains 20 labelled data.

Graph node classification algorithm based on similarity random walk aggregation

Aiming at the relatively low accuracy of methods such as MLP and GCN in heterogeneous graph node classification tasks, this paper proposes a graph neural network based on similarity random walk aggregation (SRW-GNN). Most existing node classification methods usually take neighbor nodes as neighborhoods, but the target node and its neighbors in heterogeneous graphs usually belong to different categories. To reduce the impact of heterogeneity on node embedding, SRW-GNN uses the similarity between nodes as probability to perform random walks and takes the sampled paths as neighborhoods to obtain more homogeneous information. The order in which nodes appear in the path is particularly critical for capturing neighborhood information. However, most existing GNN aggregators are insensitive to node order. This paper introduces a path aggregator based on recurrent neural network (RNN) to simultaneously extract the features and order information of nodes in the path. In addition, nodes have different preferences for different paths. In order to adaptively learn the importance of different paths in node encoding, an attention mechanism is used to dynamically adjust the contribution of each path to the final embedding. Experimental results on multiple commonly used heterogeneous graph datasets show that the accuracy of this method is significantly better than that of MLP, GCN, H2GCN, HOG-GCN and other methods, verifying its effectiveness in heterogeneous graph node classification tasks.

Adaptive multi-graph contrastive learning for bundle recommendation

Recently, recommending bundles - sets of items that complement each other - instead of individual items to users has drawn much attention in both academia and industry. Models based on Graph Neural Networks (GNNs) for bundle recommendation have achieved great success in capturing users’ preferences by modeling pairwise correlations among users, bundles, and items via information propagation on graphs. However, a notable limitation lies in their insufficient focus on explicitly modeling intricate ternary relationships. Additionally, the loose combination of node embeddings from different graphs tends to introduce noise, as it fails to consider disparities among the graphs. To this end, we propose a novel approach called Adaptive Multi-Graph Contrastive Learning for Bundle Recommendation (AMCBR). Specifically, AMCBR models ternary interactions by constructing multiple graphs, including a bundle preference graph based on direct user-bundle interactions, a collaborative neighborhoods graph featuring user-level and bundle-level subgraphs, and an item-level preference hypergraph capturing indirect user-bundle relationships through items. Then, (hyper)graph convolution is applied to each (hyper)graph to encode diverse potential preferences into node embeddings. To enhance the model’s robustness, an adaptive aggregation module is employed to assign varying weights to node embeddings from different graphs during the fusion process, which enriches the semantic and comprehensive information in the embeddings while mitigating potential noise. Finally, a contrastive learning strategy is proposed to jointly optimize the model, strengthening collaborative links between individual graphs. Extensive experiments on three real datasets demonstrate that AMCBR can outperform the state-of-the-art baselines on the Top-K recommendations.

Provenance-based APT campaigns detection via masked graph representation learning

Advanced Persistent Threats (APTs) are well-planned, persistent, and highly stealthy cyberattacks designed to steal confidential information or disrupt specific target systems. Recent studies have used system audit logs to construct provenance graphs that describe system interactions to detect potentially malicious activities. Although they are effective, they still suffer from problems such as the need for a priori knowledge, lack of attack data, and high computational overhead that limit their application. In this paper, we propose a self-supervised learning-based APT detection model, APT-MGL, which learns the embedded representations of nodes through a graph mask self-encoder and transforms the detection problem into an outlier detection problem for malicious nodes. APT-MGL characterizes the behavior of nodes based on node type, action, and interaction frequency, and fuses the features through a multi-head self-attention mechanism. Then the node embedding is obtained by combining graph features and structural information using masked graph representation learning. Finally, the unsupervised outlier detection method is used to analyze the computed embeddings and obtain the final detection results. The experimental results show that APT-MGL outperforms existing monitoring models and achieves a small overhead.

A heterogeneous graph transformer framework for accurate cancer driver gene prediction and downstream analysis

Accurately predicting cancer driver genes remains a formidable challenge amidst the burgeoning volume and intricacy of cancer genomic data. In this investigation, we propose HGTDG, an innovative heterogeneous graph transformer framework tailored for precisely predicting cancer driver genes and exploring downstream tasks. A heterogeneous graph construction module is central to the framework, which assembles a gene-protein heterogeneous network leveraging the Kyoto Encyclopedia of Genes and Genomes (KEGG) pathways and protein-protein interactions sourced from the STRING (search tool for recurring instances of neighboring genes) database. Moreover, our framework introduces a pioneering heterogeneous graph transformer module, harnessing multi-head attention mechanisms for nuanced node embedding. This transformative module proficiently captures distinct representations for both nodes and edges, thereby enriching the model's predictive capacity. Subsequently, the generated node embeddings are seamlessly integrated into a classification module, facilitating the discrimination between driver and non-driver genes. Our experimental findings evince the superiority of HGTDG over existing methodologies, as evidenced by the enhanced performance metrics, including the area under the receiver operating characteristic curves (AUROC) and the area under the precision-recall curves (AUPRC). Furthermore, the downstream analysis utilizing the newly identified cancer driver genes underscores the efficacy and versatility of our proposed framework.

Subgraph Structure Membership Inference Attacks against Graph Neural Networks

Graph Neural Networks (GNNs) have been widely applied to various applications across different domains. However, recent studies have shown that GNNs are susceptible to the membership inference attacks (MIAs) which aim to infer if some particular data samples were included in the model’s training data. While most previous MIAs have focused on inferring the membership of individual nodes and edges within the training graph, we introduce a novel form of membership inference attack called the Structure Membership Inference Attack (SMIA) which aims to determine whether a given set of nodes corresponds to a particular target structure, such as a clique or a multi-hop path, within the original training graph. To address this issue, we present novel black-box SMIA attacks that leverage the prediction outputs generated by the target GNN model for inference. Our approach involves training a three-label classifier, which, in combination with shadow training, aids in enabling the inference attack. Our extensive experimental evaluation of three representative GNN models and three real-world graph datasets demonstrates that our proposed attacks consistently outperform three baseline methods, including the one that employs the conventional link membership inference attacks to infer the subgraph structure. Additionally, we design a defense mechanism that introduces perturbations to the node embeddings thus influencing the corresponding prediction outputs by the target model. Our defense selectively perturbs dimensions within the node embeddings that have the least impact on the model's accuracy. Our empirical results demonstrate that the defense effectiveness of our approach is comparable with two established defense techniques that employ differential privacy. Moreover, our method achieves a better trade-off between defense strength and the accuracy of the target model compared to the two existing defense methods.

Simple Contrastive Graph Clustering.

Contrastive learning has recently attracted plenty of attention in deep graph clustering due to its promising performance. However, complicated data augmentations and time-consuming graph convolutional operations undermine the efficiency of these methods. To solve this problem, we propose a simple contrastive graph clustering (SCGC) algorithm to improve the existing methods from the perspectives of network architecture, data augmentation, and objective function. As to the architecture, our network includes two main parts, that is, preprocessing and network backbone. A simple low-pass denoising operation conducts neighbor information aggregation as an independent preprocessing, and only two multilayer perceptrons (MLPs) are included as the backbone. For data augmentation, instead of introducing complex operations over graphs, we construct two augmented views of the same vertex by designing parameter unshared Siamese encoders and perturbing the node embeddings directly. Finally, as to the objective function, to further improve the clustering performance, a novel cross-view structural consistency objective function is designed to enhance the discriminative capability of the learned network. Extensive experimental results on seven benchmark datasets validate our proposed algorithm's effectiveness and superiority. Significantly, our algorithm outperforms the recent contrastive deep clustering competitors with at least seven times speedup on average. The code of SCGC is released at SCGC. Besides, we share a collection of deep graph clustering, including papers, codes, and datasets at ADGC.

HeGCL: Advance Self-Supervised Learning in Heterogeneous Graph-Level Representation.

Representation learning in heterogeneous graphs with massive unlabeled data has aroused great interest. The heterogeneity of graphs not only contains rich information, but also raises difficult barriers to designing unsupervised or self-supervised learning (SSL) strategies. Existing methods such as random walk-based approaches are mainly dependent on the proximity information of neighbors and lack the ability to integrate node features into a higher-level representation. Furthermore, previous self-supervised or unsupervised frameworks are usually designed for node-level tasks, which are commonly short of capturing global graph properties and may not perform well in graph-level tasks. Therefore, a label-free framework that can better capture the global properties of heterogeneous graphs is urgently required. In this article, we propose a self-supervised heterogeneous graph neural network (GNN) based on cross-view contrastive learning (HeGCL). The HeGCL presents two views for encoding heterogeneous graphs: the meta-path view and the outline view. Compared with the meta-path view that provides semantic information, the outline view encodes the complex edge relations and captures graph-level properties by using a nonlocal block. Thus, the HeGCL learns node embeddings through maximizing mutual information (MI) between global and semantic representations coming from the outline and meta-path view, respectively. Experiments on both node-level and graph-level tasks show the superiority of the proposed model over other methods, and further exploration studies also show that the introduction of nonlocal block brings a significant contribution to graph-level tasks.

Vul-LMGNNs: Fusing language models and online-distilled graph neural networks for code vulnerability detection

Code Language Models (codeLMs) and Graph Neural Networks (GNNs) are widely used in code vulnerability detection. However, a critical yet often overlooked issue is that GNNs primarily rely on aggregating information from adjacent nodes, limiting structural information transfer to single-layer updates. In code graphs, nodes and relationships typically require cross-layer information propagation to fully capture complex program logic and potential vulnerability patterns. Furthermore, while some studies utilize codeLMs to supplement GNNs with code semantic information, existing integration methods have not fully explored the potential of their collaborative effects.To address these challenges, we introduce Vul-LMGNNs that integrates pre-trained CodeLMs with GNNs, leveraging knowledge distillation to facilitate cross-layer propagation of both code semantic knowledge and structural information. Specifically, Vul-LMGNNs utilizes Code Property Graphs (CPGs) to incorporate code syntax, control flow, and data dependencies, while employing gated GNNs to extract structural information in the CPG. To achieve cross-layer information transmission, we implement an online knowledge distillation (KD) program that enables a single student GNN to acquire structural information extracted from a simultaneously trained counterpart through an alternating training procedure. Additionally, we leverage pre-trained CodeLMs to extract semantic features from code sequences. Finally, we propose an ”implicit-explicit” joint training framework to better leverage the strengths of both CodeLMs and GNNs. In the implicit phase, we utilize CodeLMs to initialize the node embeddings of each student GNN. Through online knowledge distillation, we facilitate the propagation of both code semantics and structural information across layers. In the explicit phase, we perform linear interpolation between the CodeLM and the distilled GNN to learn a late fusion model. The proposed method, evaluated across four real-world vulnerability datasets, demonstrated superior performance compared to 17 state-of-the-art approaches. Our source code can be accessed via GitHub: https://github.com/Vul-LMGNN/vul-LMGGNN.