Image denoising is a fundamental part of image processing. The real electronic speckle pattern interferometry (ESPI) contains a large amount of speckle noise, which affects the image quality and adversely affects subsequent studies. In this paper, a method based on an improved denoising convolutional neural network (CNN) has been proposed, with the goal of reducing noise while maintaining accurate information. The block matching 3D-based adaptive TV denoising CNN can protect the valid information while preventing the information of the original image itself from being corrupted. A two-channel model is used to improve the noise reduction effect of real images. The proposed method is compared with the conventional denoising algorithms and the deep-learning denoising algorithms. Experimental results show that the proposed method can maintain accuracy, integrity, and stability while preserving the details, texture, and edge information of the stripe pattern.