This study employs a microstructure-based finite element modeling approach to understand the mechanical behavior of asphalt mixtures across different length scales. Specifically, this work aims to develop a multi-scale modeling approach employing representative volume elements (RVEs) of optimal size; this is a key issue in asphalt modeling for high-fidelity fracture modeling of heterogeneous asphalt mixtures. To determine the optimal RVE size, a convergence analysis of homogenized elastic properties is conducted using two types of RVEs, one made with polydisperse spherical inclusions, and another made with polydisperse truncated cylindrical inclusions, each aligned with the American Association of State Highway and Transportation Official’s maximum density gradation curve for a 12.5 mm Nominal Maximum Aggregate Size (NMAS). The minimum RVE lengths for this NMAS were found to be in the range of 32–34 mm. After the optimal RVE size for each inclusion shape is obtained, computational models of heterogeneous Indirect Tensile Asphalt Cracking Test samples are then generated. These models include the components of viscoelastic mastic, linear elastic aggregates, and cohesive zone modeling to simulate the rate-dependent failure evolution from micro- to macro-cracking. Examination of load-displacement responses at multiple loading rates shows that both heterogeneous models replicate experimentally measured data satisfactorily. Through micro- and macro-level analyses, this study enhances our understanding of the composition-performance relationships in asphalt pavement materials. The procedure proposed in this study allows us to identify the optimal RVE sizes that preserve computational efficiency without significantly compromising their ability to capture the asphalt material behavior under specific operational conditions.