High-modulus asphalt mixture (HMAM) is one of the most effective materials to enhance the rutting resistance of asphalt pavement and upgrade pavement sustainability. The objectives of this study are to investigate the modulus properties of different HMAMs and their correlation with the rutting resistance, to propose reasonable modulus evaluation indicators, and to analyze the rutting resistance mechanisms of different materials (hard asphalt, polyethylene, dissolved polyolefin). The effect of three HMAMs and two styrene-butadiene-styrene (SBS) modifiers on asphalt mixtures’ rutting resistance were evaluated by dynamic modulus test and wheel track test, and the results were simulated and further analyzed via ABAQUS. The results indicate that the dynamic modulus of the mixtures showed a gradual increase and decrease with the increase of loading frequency and testing temperature, respectively. The ratio of dynamic modulus in low frequency to that in high frequency correlates well with dynamic stability under high-temperature conditions, and the wider the frequency coverage, the higher the correlation between this ratio and dynamic stability. The rutting resistance of asphalt pavements can be improved by reducing the frequency sensitivity of HMAMs under high temperatures or by increasing the modulus’ absolute value of the pavement structural layer. Therefore, two indicators, the absolute value of the modulus and the ratio of 0.1 Hz dynamic modulus to 25 Hz dynamic modulus at 55 °C, are recommended for the evaluation of rutting resistance of HMAMs. Based on the evaluation indexes proposed in this paper, a comparative analysis of the rutting resistance mechanism of HMAMs prepared with different materials was carried out, and it was concluded that the mixture with high-modulus agents had the best rutting resistance, which is consistent with the test road observations, thus verifying the feasibility of the modulus evaluation indexes recommended in this paper for the evaluation of the rutting resistance of different types of HMAMs.