PurposeThis paper investigates the vibration compaction mechanism and evaluates the impact of vibration frequencies on the stability of coarse-grained soil, aiming to optimize the subgrade filling process.Design/methodology/approachThis study examines the vibratory compaction behavior of coarse-grained soils through indoor vibration tests and discrete element simulations. Focusing on angular gravel (breccias) of varying sizes, the simulations were calibrated using parameters such as Young’s modulus, restitution and friction coefficients. The analysis highlights how particle shape influences compaction, revealing mesoscopic mechanisms that drive macroscopic compaction outcomes.FindingsThis study investigates the influence of vibration frequency on the compaction behavior of coarse-grained soils using discrete element simulation. By analyzing particle contact and motion, the mesoscopic mechanisms driving compaction are explored. The study establishes a positive linear correlation between contact force anisotropy (Cv) and deformation, demonstrating that higher anisotropy leads to greater structural disruption. Additionally, the increase in sliding contact percentage (SCP) at higher frequencies indicates instability in the skeletal structure, driven by uneven contact force distribution. These findings reveal how frequency-induced stress concentration affects the stability and deformation of the soil skeleton.Originality/valueThis research explores the effect of various vibration frequencies on the compaction behavior of coarse-grained soils, examining microscopic interactions to reveal their impact on soil stability and deformation.