We proposed a method to suppress the speckle noise in scaled phase‐only holographic display. First, the complex Fresnel hologram is calculated by using the scaled diffraction algorithm. Then the phase‐only computer generated hologram (CGH) is calculated by using the error diffusion filter process. Based on this method, both of the amplitude and phase distribution on image plane reconstructed from the phase‐only CGH are optimized. The phase distribution on the image plane is smoothed and uniform so the interference among adjacent pixels is greatly suppressed. The experiment results show a prominent effect in the reconstruction with speckle suppression compared with the conventional iterative based method.