Fatigue cracking is an important pavement distress caused by repeated traffic loading that is known to be very sensitive to aging and climate conditions. A thorough understanding of the relationship between binder rheological and morphological characteristics and cracking resistance parameters of mixtures can contribute to the effective establishment of binder specifications. Therefore, this study aims to evaluate mortar film distribution and fatigue parameters of asphalt binder on fracture resistance of both virgin mixtures and recycled mixtures containing various softening oils and Reclaimed Asphalt Pavement (RAP). The mixtures were subjected to the short-term oven aging (4 h), long-term oven aging (8 h), and extended-term oven aging (16 h) using loose mix aging protocol at 135 °C. The mortar thickness was analyzed by using the 2D image from the sliced asphalt mixture samples. The statistical analysis was conducted to investigate the factors affecting the fracture performance of both asphalt binder and asphalt mixture. The results indicate the CT index of the mixture from the indirect tensile asphalt cracking test (IDEAL-CT) is significantly related to the parameters A and B of asphalt binder from linear amplitude sweep (LAS) test and the thickness of mortar. The mortar film thickness cannot judge the performance of the asphalt mixture alone. It is clearly shown in this study that thicker mortar film thickness within the RAP mixture can delay the weakening of fatigue fracture performance due to aging effect at intermediate temperature. The findings of this study might lead to a better cross-scale understanding of the fracture behavior of RAP mixtures.