This Letter presents an alternative approach for image refocusing in digital Fresnel holography. In the literature, a large majority of reported focus detection criteria is based on amplitude contrast or phase contrast. We propose a focus detection criterion based on the speckle phase coherence factor. This factor reaches its maximum value at the best focus distance. At any reconstruction distance, estimation of the coherence factor is based on robust noise estimation from anisotropic diffusion. We propose a theoretical relation to estimate the coherence factor from the measured standard deviation. Experimental results for the case of three-color holographic imaging and interferometry are provided. Experimental results show the suitability of the proposed approach for image plane refocusing.