Reducing soil erosion (SE) is crucial for achieving harmony between human society and the ecological environment. The cultivated land fragmentation (CLF), directly or indirectly, alters soil structure, diminishes its water-holding capacity, and escalates the risk of SE. Scientific assessment of the effect of CLF on SE can provide new insights into controlling of SE across watersheds in China. However, few studies have quantified the effect of CLF on SE. Therefore, we utilized land use change data in the Yangtze River basin from 2000 to 2020, measuring the levels of CLF and SE using Fragstats and InVEST models. The bivariate spatial autocorrelation model was employed to reveal the spatial relationship between CLF and SE. Additionally, we constructed a spatial Durbin model and introduced the geographically and temporally weighted regression model to analyze the role of CLF on SE. The south bank of the upper and middle reaches of the Yangtze River basin exhibited high CLF and SE. The bivariate spatial autocorrelation results showed a significant positive spatial correlation between CLF and SE. The spatial Durbin model results showed that CLF had a spatial spillover effect and time lag on SE, and the effect of CLF on SE had an inverted “N" curve. The study also confirmed that last SE and neighboring SE areas influenced local SE. Currently, CLF had a negative effect on SE in the Sichuan Basin, Yunnan-Guizhou Plateau, and the middle and lower Yangtze River Plain, and positively in Qinghai, Hunan, and Jiangxi provinces. These findings suggest that the government should enhance cross-regional and cross-sectoral cooperation and monitoring of cultivated land changes to prevent and control SE effectively.