We theoretically develop and experimentally demonstrate a holographic method for imaging cold atoms at the diffraction and photon shot noise limits. Aided by a double point source reference field, a simple iterative algorithm robustly removes the twin image of an 87Rb cold atom sample during the image reconstruction. Shot-noise limited phase shift and absorption images are consistently retrieved at various probe detunings, and during the laser cooling process. We consistently resolve less than 2 mrad phase shift ( attenuation) of the probe light, outperforming shot-noise limited phase-contrast (absorption) imaging by a factor of 4 or more if the same camera is used without pixel saturation. We discuss the possible extension of this work for precise phase imaging of dense atomic gases, and for off-resonant probing of multiple atoms in optical lattices.