Deformation directly reflects the behavior of arch dams, and deformation behavior models have been widely used for structural health monitoring (SHM) in accordance with monitoring data. The deformation field brings advantages to the SHM of arch dams in terms of the spatiotemporal scale. In this paper, a spatiotemporal deformation field behavior model based on a convolutional neural network (CNN) was proposed. The temperature field of the dam fusing the climatological data, including air temperature, weather conditions, and solar radiation, was introduced. Furthermore, the water pressure and constraint fields were also input into the CNN-based model. The physical law and structural continuity were considered, and the parameters of the CNN-based model were general and uniform for each predicted location. The accuracy and extrapolation ability of the proposed model were evaluated. The results illustrate that the CNN-based model predicts accurate deformation at the spatiotemporal scale of the arch dam. The accuracy of the prediction of single-point deformation time series from the CNN-based model is close to that from the traditional hydraulic-seasonal-time (HST) model and finite element model (FEM). For the prediction of the spatial deformation field, the mean MAE of the CNN-based model is 2.68 mm, while it is 5.66 mm for the FEM model, showing much higher prediction accuracy of the CNN-based model. The proposed globally uniform CNN-based model can derive a high-precision and high-density deformation field of arch dams with sparse monitoring data. It can also be continuously updated in real-time according to the latest monitoring data without inverse analysis and parameter adjustment like the FEM model.