Due to the harsh climatic conditions in high altitude and cold regions with large temperature differences, asphalt pavement is generally prone to cracking, and the cracks propagate rapidly, which reduces the service life and service level of the road. The factors influencing the fracture characteristics of asphalt mixtures were analyzed in this paper, and the mixtures with different aggregate gradations from various types of asphalt were prepared. The fracture characteristics were explored using the thermal stress restrained specimen test (TSRST) and low-temperature bending test, and the good consistency of the low-temperature fracture performance was identified according to the results of frost-break temperature, flexural strength, and fracture toughness. The frost-break temperature was confirmed as the best indicator of the material crack resistance and could be used as the index to evaluate the performance of asphalt mixtures at low temperatures. The frost-break temperature of matrix asphalt mixture is 8–10 °C higher than that of modified asphalt mixture, and AC asphalt mixture is 2–4 °C higher than that of SMA asphalt mixture. The excellent asphalt performance has a more important influence on the fracture characteristics of asphalt mixture. The asphalt mixture of the same type had similar fracture toughness at varying notch depths, the most deviation is 3.78% which shows that the initial crack depth has little effect on the fracture toughness of asphalt mixture at low temperature. The results of the study can provide a basis for the selection of asphalt pavement surface materials and the optimization of mixtures in high altitude and cold regions with large temperature differences.