The Pisha sandstone area, situated in the upper and middle reaches of the Yellow River in China, is characterized by severe soil and water erosion, making it one of the most critical regions on the Loess Plateau. The rugged terrain and exposed bedrock complicate management efforts for this area, posing challenges for accurate forecasting using soil erosion models. Through an analysis of terrain, vegetation, and precipitation impacts on soil erosion, this study offers theoretical support for predicting soil erosion within the exposed Pisha sandstone area of the Loess Plateau. This has substantial implications for guiding water and soil conservation measures in this region. Focusing on China’s exposed sandstone area within the Geqiugou watershed, temporal and spatial changes in vegetation cover and land use from 1990 to 2020 were analyzed. The result shows that, from 1990 to 2020, the grassland area has exhibited a consistent downward trend, with successive reductions of 64.86% to 59.46%. The area of low vegetation cover witnessed a significant decline of 59.29% in 2020 compared to that in 1990. The moderate erosion area decreased from 84.52 to 57.17 km2. The significant reduction in soil and water loss can be attributed to the expansion of forest and grassland areas, with the implementation of the Grain for Green project serving as a key policy driver for facilitating this expansion. This study provided a good example of combining rainfall with vegetation coverage to fast estimation soil erosion. A mathematical relationship between the vegetation rainfall coupling index (RV) and soil erosion was established with strong fitting effects, enabling estimation of the soil erosion volume under varying slope conditions within Pisha sandstone areas. The main focus of future soil and water conservation in the Pisha sandstone area should be on effectively managing the channel slope and minimizing exposed bedrock areas through a combination of slope cutting, the application of anticorrosive materials, and the implementation of artificial vegetation planting.