Vegetation restoration is an effective and important measure for controlling soil erosion in arid and -arid regions. Both its aboveground and underground parts play a crucial role in controlling surface runoff and soil detachment on slopes. But how much the parts of vegetation contribute to the runoff and sediment reducing benefits of rill erosion on slopes is unclear. We used grassland slopes at four successional stages for simulated scouring experiments to observe how successional vegetation community structures, root characteristics, and soil structures contribute to erosion and sand production. Initial flow production time increased, and total runoff decreased. Under the scour intensities, the 11-year slope had the lowest flood peak and volume and the greatest runoff reduction benefit. The 25-year slope had the lowest sand peak and volume and the greatest sediment reduction benefit. As scour intensity increased, runoff reduction effect of vegetation at the successional stages decreased; the sediment reduction benefit remained high. PLS-PM analysis showed that the indirect effects of the aboveground and underground parts of vegetation on sand production were −0.364 and −0.439, respectively. Aboveground parts mainly embodied the regulation of runoff, in which stem count, humus mass, and biomass were the main factors affecting runoff and sand production. Underground parts mainly reflected their soil structure improvement, in which root volume density, root surface area density, and root mass density are the main explanatory variables. The direct effects of runoff and soil structure on slope rill erosion were 0.330 and −0.616, respectively, suggesting the stability of soil structure is the primary factor affecting the sand production, not erosion energy. The results provide a reference for scientific assessment of the key role of natural vegetation restoration in regional soil erosion control and the development of biological measures for soil and water conservation on the slopes of the Loess Plateau.