This study aimed to design and evaluate asphalt mixtures by considering various factors including different binders, aggregate types, and gradation characteristics of the fine portion of mixtures, referred to as the interstitial component (IC). The mixtures were compacted to 93% of maximum theoretical density to prepare specimens for Superpave IDT tests, which included resilient modulus, creep compliance, and fracture tests. Parameters such as failure limit (fracture energy), damage rate (dissipated creep strain energy per cycle), and the relative cracking performance indicator (number of cycles to failure) were assessed for test specimens subjected to the long-term oven aging and the cyclic pore pressure conditioning. The results showed that mixtures with styrene–butadiene–styrene modified binders exhibited slightly higher failure limits but significantly lower damage rates compared to mixtures with unmodified binders, indicating improved cracking performance. Similarly, the limestone mixtures demonstrated equivalent failure limit but much lower damage rate to the granite mixtures, attributed to the porous surface morphology and inherent weakness of limestone aggregates. Notably, even small changes in the IC characteristics of the mixtures yielded substantial differences in fracture properties and performance, a phenomenon believed to be linked to the distribution of asphalt binder and the structure of air voids within the mixture. In conclusion, a thorough examination confirmed the impact of polymer modification and aggregate type on fracture properties and, notably, highlighted the pivotal role of the fine portion's characteristics in determining mixture cracking performance.