Physical hardening (PH) significantly affects bitumen's low-temperature performance. This paper investigates the effect of oxidative aging and wax structure on PH using 4-mm dynamic shear rheometer (4-mm DSR) and differential scanning calorimetry (DSC) tests. The results show that the wax chain length has a pronounced effect on the PH. Short-chain waxes, such as C18 and C24 affect the PH while longer-chain n-paraffins like C40 or Sasobit have almost no effect. In addition, the physical hardening index (PHI) is not linearly increasing with wax content but shows a maximum at a particular wax percentage. Wax-doped samples that crystallize before reaching the hardening temperatures typically show a reduced PHI compared to those with crystallization temperatures just below the hardening temperature. When crystallization and melting temperatures are not close to the hardening temperatures, the PHIs are typically reduced. Oxidative aging also influenced the PHI and in most cases the hardening decreased after aging. The highest PHI of almost all investigated samples occurs at 0 °C among the four typical temperatures. This investigation provides insights in the mechanism of PH and in the relations between wax contents, the hardening temperature, and the PHI. It also shows that while long-chain waxes used as warm mix additives, will not cause a PH risk, short-chain waxes present in pyrolyzed waste plastic, for example, should be handled with care. In addition, PH is also crucial for the intermediate service temperatures and not exclusively for thermal cracking at low temperatures.