Cohesive and adhesive damages play essential roles in the asphalt pavements cracking behaviors; however, due to the microstructural complexity and heterogeneity of asphalt mixtures, it is difficult to effectively characterize these two damage initiations under the realistic service conditions. To address this issue, the presented study developed a multiscale finite element model to quantitatively investigate the distributions of cohesive and adhesive damages as well as their effects on the overall damage performance of asphalt pavement. The microstructure of the asphalt mixture and macrostructure of the asphalt pavement were simultaneously involved into a single finite element (FE) model, and the bilinear cohesive elements were inserted into the fine aggregate matrix (FAM) and the FAM-aggregates interfaces to respectively model the cohesive and adhesive damage initiations. The results showed that the great heterogeneity of the asphalt mixtures caused remarkable stress concentrations, which might be one of the main factors that contribute to the pavement cracking behaviors. In addition, it is illustrated that lower adhesive strengths tended to cause more damage initiations at the pavement surface, which suggested that the adhesive damage could be one of the major factors contributing to the “top-down” cracking initiations.