An improved understanding of the temporal variations in soil erosion rates and sediment sources is necessary to identify the main areas prone to soil erosion and to effectively implement soil and water conservation measures. In this study, we selected two dam-controlled watersheds (Xiaoshilata and Yangjiagou) on the northern Loess Plateau, China, and determined the temporal variations in the sediment yield. The sedimentation upstream of the dams showed that there were 31 and 9 flood couplets in the Xiaoshilata (1958–1972) and Yangjiagou watersheds (2007–2011), respectively, which have trapped 16.5×104t and 3.38×104t of sediment, respectively. The estimated specific sediment yield was approximately 173.6t/ha/a and 106.1t/ha/a in the Xiaoshilata and Yangjiagou watersheds, respectively. A multivariate mixed model was applied to identify the different sediment sources. The results indicated that weathered sandstone contributed approximately 61.52% of the total sediment, and the remaining sediment loss was from bare loess soil (32.54%) and grassland (5.94%) in the Xiaoshilata watershed. In the Yangjiagou watershed, approximately 66.8% of the sediment originated from weathered sandstone, whereas bare loess and grassland accounted for 17.5% and 15.7% of the sediment yield, respectively. Based on these findings, we recommend that comprehensive soil conservation measures are needed to soil erosion control for the permanent steep gullies due to high sediment contribution. The sedimentation behind the check dams provides an indirect method for estimating the sediment yield in ungauged basins.