Abstract With the increasing demands for detection range and accuracy in heterodyne lidars, precise analyses of system efficiency are of growing importance. An extended system efficiency model is established for the detection of heterodyne lidar returns backscattered by aerosols in turbulent atmosphere; the extended Huygens–Fresnel integral and the Tatarskii spectrum are used in the derivations of system efficiency and effective coherent solid angle(ECSA). The impacts of beam truncation, optical aperture, atmospheric turbulence and laser source coherence on the performance of a monostatic heterodyne lidar are analyzed in detail. Numerical results suggest that the optimal system efficiency and beam truncation change with laser coherence, turbulence and distance for a collimated system. Lidars with larger optical apertures may have worse system efficiencies which can be below 5.5% with the aperture of 160 mm when C n 2 is 1 0 − 14 m − 2 ∕ 3 . Moreover, similar ECSAs can be obtained with both small and large optical apertures. When C n 2 is 1 0 − 18 m − 2 ∕ 3 and 1 0 − 14 m − 2 ∕ 3 , the ECSAs with the laser source coherence factor of 101 have extra attenuations of 16.52dB and 4.83dB within 30 km, which indicates that the partial coherence of laser source should be considered in the performance analyses.