Accurate heat load prediction is the key to stable operation and control of district heating system. However, current heat load forecasting methods tend to treat individual buildings as isolated entities, ignoring the temporal and spatial correlation between buildings. In this paper, a heat load forecasting model is proposed which fully considers spatiotemporal information. Firstly, the spatial and temporal relationship between buildings is analyzed by statistical method. Secondly, synchronous wavelet transform (SWT) is used to denoise the heat load data to reduce the difficulty of prediction. Then, a hybrid model of multi-modal graph attention network (MG) and multi-scale Informer (MI) is constructed to capture spatiotemporal information in the data. Finally, an example of DHS in a campus in Liaoyang is analyzed. The results showed that compared with LSTM, CNN-Informer, GCN-LSTM and GAT-Informer, the MAE of MGMI on 168 h test set was reduced by 71.4%, 61.5%, 58% and 41.6%, respectively. And has the highest computational efficiency. The validity of the model is verified, which provides an important basis for the operation control of the thermal system.