Most existing model-based and learning-based image deblurring methods usually use synthetic blur-sharp training pairs to remove blur. However, these approaches do not perform well in real-world applications as the blur-sharp training pairs are difficult to be obtained and the blur in real-world scenarios is spatial-variant. In this paper, we propose a self-supervised learning-based image deblurring method that can deal with both uniform and spatial-variant blur distributions. Moreover, our method does not need for blur-sharp pairs for training. In our proposed method, we design the Deblurring Network (D-Net) and the Spatial Degradation Network (SD-Net). Specifically, the D-Net is designed for image deblurring while the SD-Net is used to simulate the spatial-variant degradation. Furthermore, the off-the-shelf pre-trained model is employed as the prior of our model, which facilitates image deblurring. Meanwhile, we design a recursive optimization strategy to accelerate the convergence of the model. Extensive experiments demonstrate that our proposed model achieves favorable performance against existing image deblurring methods.