Speckle-shearing technology is widely used in defect detection due to its high precision and non-contact characteristics. However, the wrapped-phase recording defect information is often accompanied by a lot of speckle noise, which affects the evaluation of defect information. To solve the problems of traditional denoising algorithms in suppressing speckle noise and preserving the texture features of wrapped phases, this study proposes a speckle denoising algorithm called a speckle denoising convolutional neural network (SDCNN). The proposed method reduces the loss of texture information and the blurring of details in the denoising process by optimizing the loss function. Different from the previous simple assumption that the speckle noise is multiplicative, this study proposes a more realistic wrapped image-simulation method, which has better training results. Compared with representative algorithms such as BM3D, SDCNN can handle a wider range of speckle noise and has a better denoising effect. Simulated and real speckle-noise images are used to evaluate the denoising effect of SDCNN. The results show that SDCNN can effectively reduce the speckle noise of the speckle-shear wrapping phase and retain better texture details.