Thermal cracking is one of the most annoying distresses in asphalt pavements. The cracking of asphalt pavements caused by extremely low temperature and the damage of thermal fatigue caused by repeated temperature variations are particularly pronounced. The service condition of asphalt pavements has been affected significantly. Laboratory experiments with cyclic loads simulating the actions caused by repeated temperature variations on pavements have been conducted in many previous studies to investigate the thermal fatigue characteristics of asphalt mixtures. In those studies, an equivalent cyclic mechanical stress that can be applied by an actuator was mostly used to replace the cyclic thermal stress induced by a temperature variation. It could simplify the experimental process but could not consider the performances of asphalt mixtures varying as a function of temperature. In the present study, the thermal fatigue properties and cracking behaviors of asphalt mixtures were comprehensively investigated through a modified thermal stress restrained specimen test (TSRST) and the numerical analysis. Various temperature ranges (-20 to −10 °C, −10 to 0 °C), cyclic cooling rates (10 °C/h, 20 °C/h), low-temperature performance cooling rates (10 °C/h, 8 °C/h, 6 °C/h, 4 °C/h, 2 °C/h) and pre-crack dimensions were considered for the study to address the influences of those parameters on the analysis results. Based on the testing and simulation results, it was found that the effects of thermal fatigue on asphalt mixtures under different temperature variations and cooling rates were quantified, and the damage characteristics of the asphalt mixtures were also explored. The results showed that under the low-temperature performance state, the thermal stress of asphalt mixtures increases with a uniform cooling rate and it could be obtained that the faster the cooling rate, the larger the thermal stress generated. In the cyclic thermal stress test, due to the viscoelastic properties of asphalt mixtures, the thermal stress showed significant stress relaxation phenomena under the temperature variations. The magnitude of thermal stress decreased gradually within each cycle. It demonstrates that under the same temperature variation, the thermal stress decreases continuously with the increasing thermal cycles. Through the numerical simulation, it was clear that under different temperature ranges and cooling rates, the magnitude of thermal stress increased with the temperature range decreased, whereas the magnitude of thermal stress increased with an acceleration of the cooling rate. The lower temperature range induced a larger stress intensity factor (KI), which implies a faster cracking evolution. Analyzing the effects of different pre-crack dimensions on the number of temperature cycles, the height of pre-crack showed significant effects on the number of temperature cycles than width.