The present work proposes a new model based on a convolutional neural network (CNN) to retrieve solar shortwave (SW) irradiance via the estimation of the cloud modification factor (CMF) from daytime sky images captured by all-sky cameras; this model is named CNN-CMF. To this end, a total of 237,669 sky images paired with SW irradiance measurements obtained by using pyranometers were selected at the following three sites: Valladolid and Izaña, Spain, and Lindenberg, Germany. This dataset was randomly split into training and testing sets, with the latter excluded from the training model in order to validate it using the same locations. Subsequently, the test dataset was compared with the corresponding SW irradiance measurements obtained by the pyranometers in scatter density plots. The linear fit shows a high determination coefficient (R2) of 0.99. Statistical analyses based on the mean bias error (MBE) values and the standard deviation (SD) of the SW irradiance differences yield results close to −2% and 9%, respectively. The MBE indicates a slight underestimation of the CNN-CMF model compared to the measurement values. After its validation, model performance was evaluated at the Antarctic station of Marambio (Argentina), a location not used in the training process. A similar comparison between the model-predicted SW irradiance and pyranometer measurements yielded R2=0.95, with an MBE of around 2% and an SD of approximately 26%. Although the precision provided by the SD at the Marambio station is lower, the MBE shows that the model’s accuracy is similar to previous results but with a slight overestimation of the SW irradiance. Finally, the determination coefficient improved to 0.99, and the MBE and SD are about 3% and 11%, respectively, when the CNN-CMF model is used to estimate daily SW irradiation values.