Image blur is a common image degradation phenomenon that occurs in case of camera movement and lens defocusing. In order to deblur the image, point spread function (PSF) model is often used to match the real blur kernel, but currently the matching effect is poor. In this paper, an image restoration technique based on the Gaussian Second Derivative (GSD) fitted PSF is proposed to address this problem. Firstly, a GSD model is proposed. The actual image Line spread function (LSF) extracted by knife-edge method is modeled as a linear combination of a series of Gaussian second derivatives. Then, an improved annihilation filtering algorithm is proposed. The algorithm is used to estimate the parameters of the GSD model, so as to remove the errors in the actual complex scenarios. Finally, the image PSF is obtained by using the LSF curve fitted and corrected by the GSD model, and combined with the existing Richardson-Lucy (R-L) algorithm, the clear image is finally restored. Experimental results show that compared with the existing image restoration strategies based on other PSF models, this method can effectively improve the image restoration accuracy.