Abstract In this paper, we provide a fairly standard derivation of the power spectrum of photocurrent and limited displacement resolution of laser heterodyne detection system. And the phase noise with different oscillation lineshapes of the heterodyne detection system is discussed. Based on the one-dimensional probability distribution model of phase noise, the minimum resolvable displacement and noise-equivalent mean square displacement are established, which represent the minimum detectable amplitude and noise limited resolution respectively. According to the numerical results, we obtain the variation of displacement resolution against the key parameters including the laser wavelength, the detection distance, the laser linewidth and the signal power. It reveals that the phase fluctuation has a significant effect on the power spectrum of photocurrent. Besides, both the probability density distribution of phase noise and displacement resolution have heavy dependence on laser oscillation lineshapes. And the noise-equivalent mean square displacement is inversely proportional to the root-mean-square of a measurable light power. The noise-equivalent mean square displacement can reach 3 . 5 × 1 0 − 15 m ∕ Hz under the given parameters. Our findings provide a quantitative reference for the displacement resolution and engineering application of the heterodyne detection.