Cold milling is a widely used method for rehabilitating asphalt pavement, generating reclaimed asphalt pavement (RAP) chunks. Within this process, aggregates within the asphalt pavement will be crushed, forming RAP agglomerates and aggregate breakdown. However, the mechanism of these phenomena has remained unclear, and a unified evaluation method has yet to be established. In this study, RAP agglomeration and aggregate fragmentation were characterized, five distinct methods were systematically assessed, and the mechanism of RAP agglomeration and breakdown was analyzed by discrete element method (DEM) simulation based on setting different particle contact parameters, then followed by a mechanical analysis, and demonstrated in engineering. The results revealed that both agglomeration and aggregate breakdown occur within RAP particles of various sizes, with the five methods showing similar trends in quantifying these effects. Through DEM simulations and mechanical analyses, the aggregate breakdown predominantly occurs at the cutter's motion trajectory of the cutter and during crack propagation, while agglomeration was mainly related to the sliding surface's area. The milling speed and depth positively impact RAP agglomeration, while negatively affecting aggregate breakdown, and milling drum speed exerts minimal influence on these phenomena. RAP agglomeration varies considerably in different engineering projects, and cold milling parameters should be determined based on the material composition of the asphalt pavement and design requirements to control agglomeration and breakdown rates of RAP.