Vegetation filter strips (VFSs) have always been an important measure to control agricultural soil erosion, especially in mountainous and hilly areas with more sloping farmland. To investigate the mechanism of the sediment-trapping process by VFSs, a series of tests were conducted with four gradients of flow rate, 7.5–45 L min−1 m−1, and two different sediment concentrations of 40 and 120 g L−1. The whole process of overland flow was monitored, and sediment and particle size samples from the inflow and outflow were collected and measured. The results showed that the changes in sediment concentration did not significantly affect the corresponding coefficients in the power function relationship between overland flow rate and velocity. Using the Reynolds number alone cannot effectively indicate the flow pattern of overland flow on vegetation hillslopes. The peak particle size and linear function were effective in describing the relationship between sediment particle composition and delivery rate during the sediment-trapping process by VFSs. During the sediment-trapping process, the sediment-trapping capacity of VFSs continued to decrease. The increase in sediment discharge was accompanied by a higher proportion of coarse particles. Under the same flow rate conditions, when the sediment concentration was higher, the coarse particles and their proportion also increased faster. Therefore, using only a certain particle size threshold to distinguish suspended and transported sediment may lead to inaccurate estimation of the sediment-trapping performance of VFSs. This study deepened the understanding of the mechanism of water–sediment processes on vegetation hillslopes and promoted the widespread and efficient application of VFSs management technology.