This study investigates the anti-cracking performance and cracking mechanisms of asphalt mixtures. The results of experimental tests were compared with simulated tests of asphalt mixtures based on the multi-scale finite element method (FEM). The FEM considered the mesostructure of the asphalt mixtures, the internal viscoelasticity of asphalt mortar, and the asphalt mastic-aggregate interface. Coarse aggregates were generated based on the polygonal random aggregate generation and placement method. Numerical cracking models of the asphalt mixtures were established by the multi-scale FEM and fracture tests were simulated according to the experimental test conditions. The reliability of the simulations was verified by comparison with the experimental test results. The influences on cracking behavior were investigated, with an energy-saving and economical alternative mixture—cold recycled mixture (CRM)—used for comparison. The simulations were highly accurate and could effectively simulate the asphalt mixtures’ cracking processes and mechanisms. The cracking behavior was directly related to notch length, test temperature, void content, and mixture type. A significant difference in the macroscopic anti-cracking performance of hot mixture asphalt and CRM was found. According to the simulations of crack development, there is a 'critical point' of crack propagation from the interface to the interior of the mortar. The mesoscale influences on the cracking behavior of asphalt mixtures are discussed and ranked in terms of importance.