The spatial distribution of landscape pattern in catchments has a great impact on the potential sediment detachment and transport capacity. In this study, a sediment connectivity model was applied to evaluate the effect of the spatial distribution of landscape pattern on the sediment transfer in a typical catchment on the Loess Plateau in China based on land use and topography data in 1964, 2004, and 2018. Thereupon, the impacts of changes in different land use types on sediment connectivity and those of sediment connectivity changes on sediment yield reduction in the catchment were quantified. The results showed that (i) grassland and slope farmland were the main land use types in the catchment, and a decrease of the slope farmland area and an increase of the vegetation coverage occurred from 1964 to 2018; (ii) compared with 1964, most parts of the catchment in 2004 and 2018 had an overall lower connectivity index (IC); (iii) the distribution of sediment connectivity presented obvious spatial variability. The hill slopes and tributary gullies tended to be relatively well connected, and the main gully was highly disconnected. Scenario simulations showed that the potential sediment reduction effect was 20%−45% for vegetation, and <10% for terraces or dam lands in 1964, 2004, and 2018. The effect of impounding sediment by check dams was not included in IC calculation in this study, and it was found that the sediment impounded by these dams accounted for about 16% of sediment discharge reduction during 1995–2004 and 1% during 2005–2018. The IC reduction was responsible for 84% of the decrease of sediment discharge during 1995–2004 and 99% during 2005–2018, reflecting that vegetation restoration and terrace construction on slopes has become the main drivers for reducing IC value and sediment yield.