While calculated non-ionizing energy loss (NIEL) generally correlates well to first order with radiation-induced displacement damage rates, it does not account for some well-known differences in damage rates for n- and p-type Si. Here we show that the magnitude of these differences, DeltaKn-p, correlates closely with the fraction of total displacement damage due to low-energy primary knock-on atom (PKA) recoils. The primary products of these displacement damage events, with PKA recoils <~ 2 keV, are close vacancy-interstitial pairs, or Frenkel Pairs (FPs). Based on previous studies of vacancy-dopant complex stabilities in Si, and new parameter-free quantum mechanical calculations of FP properties, details of the stable defect profiles arising from low-energy PKA recoil events are shown to give rise to non-zero values of DeltaKn-p