In order to quantitatively describe the influence of aggregate distribution on crack development and peak stress of recycled aggregate concrete, a multifractal spectrum theory was proposed to quantitatively characterize aggregate distribution in specimens. A mesomechanical model of reclaimed aggregate concrete mixed with natural aggregate and artificial aggregate was constructed. Numerical simulation tests were conducted on the uniaxial compression mechanical behavior of 25 groups of sample models with the same proportion and different aggregate distribution forms. Based on the box dimension theory, the multiple fractal spectrum method was used to quantitatively characterize the aggregate distribution form, and the key factors affecting cracks were explored based on the gray correlation degree. The research results show that the aggregate distribution in recycled aggregate concrete has multifractal characteristics. The multifractal spectrum was used to effectively characterize the aggregate distribution pattern, which can enlarge local details and provide new ideas for the quantitative analysis of the damage mode of recycled concrete. Secondly, by establishing a statistical model of the correlation between the multifractal spectrum width of the aggregate distribution pattern and the crack distribution box dimension, it was found that there was a positive correlation between the two, that is, the greater the multifractal spectrum width of the aggregate distribution pattern, the greater the crack box dimension, and the more complex the crack distribution. The complexity of aggregate distribution is closely related to the irregularity and complexity of mesoscopic failure crack propagation in recycled concrete specimens. In addition, gray correlation theory was applied to analyze the key factors affecting the formation of cracks in the specimens. The results showed that aggregate distribution had a first-order correlation with crack formation, and changes in aggregate distribution were an important factor affecting the performance of recycled concrete. Secondly, the poor mechanical properties of NAITZ led to obvious material damage, while NCA and MZ had a significant impact on the skeleton effect in the stress–strain process due to their large areas. This study deepens people’s understanding of the damage characteristics and cracking failure modes of recycled concrete. The study verifies the feasibility of the application of recycled aggregates and provides a valuable reference for engineering practice.