BP-MoE: Behavior Pattern-aware Mixture-of-Experts for Temporal Graph Representation Learning

Abstract

Temporal graph representation learning aims to develop low-dimensional embeddings for nodes in a graph that can effectively capture their structural and temporal properties. Prior approaches primarily focus on devising sophisticated neural architectures to encode the historical interactions of nodes, enabling capturing the specific behavior patterns. However, such methods often overlook the fact that different nodes in a graph may exhibit distinct evolutionary preferences, resulting in a failure to adequately model the variations in the node evolution behavior. To address these issues, we propose a novel temporal graph learning framework called Behavior Pattern-aware Mixture-of-Experts (BP-MoE), which leverages multiple expert encoder networks with diverse architectures to comprehensively capture the behavior patterns for nodes. In detail, we first design a multi-perspective graph encoder that employs experts based on the gated recurrent units, graph attention units, and multilayer perceptron units to encode a node’s long-term memory, high-order neighborhood features, and short-term behavior preference, respectively. Subsequently, to achieve the personalized evolution preference learning for each individual node, we incorporate a behavior pattern-aware MoE mechanism to activate the pattern-specific experts for each node via a gating network. Finally, to optimize the training process of expert networks, we introduce two auxiliary losses that prevent the imbalanced training across experts. Extensive experiments conducted on multiple benchmark datasets verify the superiority of our approach over the state-of-the-art baselines. Specifically, BP-MoE consistently outperforms the baseline models for the link prediction and node classification tasks under various experimental settings.