AbstractAn accurate solar wind (SW) speed model is important for space weather predictions, catastrophic event warnings, and other issues concerning SW—magnetosphere interaction. In this work, we construct a model based on convolutional neural network (CNN) and Potential Field Source Surface (PFSS) magnetic field maps, considering a SW source surface of RSS = 2.5R⊙, aiming to predict the SW speed at the Lagrange‐1 (L1) point of the Sun‐Earth system. The input of our model consists of four PFSS magnetic field maps at RSS, which are three, four, five, and six days before the target epoch. Reduced maps are used to promote the model's efficiency. We use the Global Oscillation Network Group (GONG) photospheric magnetograms and the potential field extrapolation model to generate PFSS magnetic field maps at the source surface. The model provides predictions of the quasi‐continuous test data set, which is generated by randomly assigning 120 data segments that are individually continuous in time, with an averaged correlation coefficient (CC) of 0.53 ± 0.07 and a root mean square error (RMSE) of 80.8 ± 4.8 km/s in an eight‐fold validation training scheme with the time resolution of the data as small as one hour. The model also has the potential to forecast high speed streams of the SW, which can be quantified with a general threat score of 0.39.