Magnitude estimation is a critical task in seismology, and conventional methods usually require dense seismic station arrays to provide data with sufficient spatiotemporal distribution. In this context, we propose the Earthquake Graph Network (EQGraphNet) to enhance the performance of single-station magnitude estimation. The backbone of the proposed model consists of eleven convolutional neural network layers and ten RCGL modules, where a RCGL combines a Residual Connection and a Graph convolutional Layer capable of mitigating the over-smoothing problem and simultaneously extracting temporal features of seismic signals. Our work uses the STanford EArthquake Dataset for model training and performance testing. Compared with three existing deep learning models, EQGraphNet demonstrates improved accuracy for both local magnitude and duration magnitude scales. To evaluate the robustness, we add natural background noise to the model input and find that EQGraphNet achieves the best results, particularly for signals with lower signal-to-noise ratios. Additionally, by replacing various network components and comparing their estimation performances, we illustrate the contribution of each part of EQGraphNet, validating the rationality of our approach. We also demonstrate the generalization capability of our model across different earthquakes occurring environments, achieving mean errors of ±0.1 units. Furthermore, by demonstrating the effectiveness of deeper architectures, this work encourages further exploration of deeper GNN models for both multi-station and single-station magnitude estimation.