This article aims to reveal the physical hardening mechanism of polymer-modified asphalt binder (PMA) and accurately assess its impact on the low-temperature fracture properties of asphalt binders. Base asphalt binder (BA), styrene–butadiene–styrene modified asphalt binder (SBSMA), and crumb rubber modified asphalt binder (CRMA) were used as the research objects. Molecular dynamics (MD) simulation and differential scanning calorimetry (DSC) test were used to investigate the physical hardening mechanism of PMA at the microscopic level. The fracture characteristics of PMA under physical hardening were analyzed based on the single-edge notched beam (SENB) test. The results show that under prolonged cold service conditions, the asphalt molecular mobility is weakened, the non-bond energy and the free volume are reduced, and the molecules undergo aggregation state transformation, leading to the physical hardening of asphalt materials. In the process of physical hardening of asphalt molecules in the micro-Brownian motion of the main active force from the aromatic and saturate fractions. Considering the effect of physical hardening, it is recommended that the method used to test the low temperature properties of asphalt binders should ensure a constant temperature for at least 24 h. The crack tolerance (Ct) is recommended as the evaluation index of low-temperature cracking resistance of PMA in cold regions. The Ct of BA, SBSMA, and CRMA are reduced by 52.88 %, 12.04 %, and 35.24 %, respectively, after 24 h of constant low temperature. This indicating that the polymer modifiers can improve the fluidity of asphalt molecules, inhibit molecular aggregation, and reduce the effect of physical hardening on asphalt binders. Thermo-oxidative aging can aggravate the physical hardening phenomenon of asphalt binder, and the higher the degree of thermo-oxidative aging, the more severe the physical hardening of asphalt binder. The related research results provide the theoretical basis and technical support for the accurate design and development of the low-temperature performance of asphalt binders for roads in cold regions.