Thermal cracking is one of the primary distresses of asphalt pavement which is a complex process. Evaluating the Low-temperature Performance (LTP) of asphalt mixtures accurately and conducting material optimization analysis is not only crucial for understanding the thermal cracking mechanism of asphalt mixtures but also serve as the basis for enhancing their thermal cracking resistance performance. However, current research has not sufficiently interpreted the mechanism of thermal cracking. It is segmentary that optimizes asphalt mixtures solely based on their ultimate strength, strain, and so on at certain low-temperature points. Therefore, this study closely integrated the thermal cracking mechanism and proposed a two-parameter optimization method. Firstly, clarifying the decisive role of the contraction and relaxation relationship in determining LTP through the Thermal Stress Restrained Specimen Test (TSRST), thermal contraction test, and low-temperature relaxation test. Secondly, quantizing the impact of the aggregates skeleton and then using a two-parameter method to analyze the comprehensive effects on LTP of asphalt mixtures. Finally, summarizing the material optimization method of asphalt binder-first, aggregates-second based on the multiple linear regression method. This paper simplified the evaluation method and provided a basis for material selection and design for improving the thermal cracking resistance of asphalt mixtures.