Interactions between precipitation, vegetation, and erosion are crucial and not fully solved issues in the area of earth surface processes. The Qingshui River Basin (QRB), as the main sediment source tributary of the upper reaches of the Yellow River, is characterized by spatial heterogeneity of rainfall, vegetation, and soil erosion. In this study, we investigated the spatiotemporal variations of sediment yields within the QRB and further identified the coupling effects of precipitation and vegetation on soil erosion. We collected annual (1955 to 2016) and daily (2006 to 2016) hydrological and sediment series from six hydrological stations, which subdivided the whole basin into six different sub-basins with heterogeneity in climate and landscape. Variations in parameter a of the sediment rating curves among the six sub-basins continuously declined, showing the continuously increasing effect of vegetation coverage on reducing soil erosion. The unique combination of relationships between precipitation characteristics and vegetation patterns in six sub-basins and these coupling effects resulted in different precipitation–vegetation–erosion patterns in six sub-basins. Sediment yield followed bell-shaped relationships with vegetation and precipitation, with a clear critical threshold at normalized difference vegetation index (NDVI) = 0.36/precipitation = 100 mm at a monthly scale. Based on these thresholds, the non-linear relationships between precipitation, vegetation, and erosion were also explained. We also found that reducing the time lags in which vegetation follows precipitation may be effective in suppressing sediment yield. These findings could provide a quantitative approach to estimating the potential changes in sediment yield associated with proposed ecological rehabilitation schemes in this region.