Traditional data-driven methods for modeling and predicting epidemic spreading typically operate in an independent and identically distributed setting. However, epidemic spreading on complex networks exhibits significant heterogeneity across different phases, regions, and viruses, indicating that epidemic time series may not be independent and identically distributed due to temporal and spatial variations. In this article, a novel deep transfer learning method integrating convolutional neural networks (CNNs) and bi-directional long short-term memory (BiLSTM) networks is proposed to model and forecast epidemics with heterogeneous data. The proposed method combines a CNN-based layer for local feature extraction, a BiLSTM-based layer for temporal analysis, and a fully connected layer for prediction, and employs transfer learning to enhance the generalization ability of the CNN-BiLSTM model. To improve prediction performance, hyperparameter tuning is conducted using particle swarm optimization during model training. Finally, we adopt the proposed approach to characterize the spatio-temporal spreading dynamics of COVID-19 and infer the pathological heterogeneity among epidemics of severe acute respiratory syndrome (SARS), influenza A (H1N1), and COVID-19. The comprehensive results demonstrate the effectiveness of the proposed approach in exploring the spatiotemporal variations in the spread of epidemics and characterizing the epidemiological features of different viruses. Moreover, the proposed method can significantly reduce modeling and predicting errors in epidemic spread to some extent.