The continuous afforestation has led to significant changes in soil moisture (SM) dynamics on the Loess Plateau, China (LPC). However, evaluating the spatio-temporal distributions and driving factors of the SM changes following afforestation is challenging because of the difficulty of accessing areas and the lack of long-term series records. Here, we evaluated the accuracy of Global Land Evaporation Amsterdam Model (GLEAM) SM product by comparing with in-situ SM data in LPC, and quantified the spatio-temporal variations of SM in different periods from 1982 to 2015 under the ‘Grain for Green’ program (GGP). The results showed that the Normalized Difference Vegetation Index (NDVI) was well matched with the SM variations in more than 46% area of LPC from 1982 to 2015, which mainly located at the vegetated areas. The variations of SM responding to revegetation displayed obvious southeast-negative and northwest-positive patterns, which was the wet region (annual precipitation >450 mm) to be dry and dry region (annual precipitation ≤450 mm) to be wet. Path coefficient value revealed that the positive effect of vegetation activity for grassland on SM was ascribed to its promotion on the occurrence of rainfall (mean path coefficient = 0.278). Precipitation played vital impact on the SM in bareland and sparsely vegetated area, and evapotranspiration played a dominant role in the SM dynamic of the forestland, especially at the early stage of GGP (From 2000 to 2010), while precipitation and NDVI had stronger effects than evapotranspiration on the SM dynamics of grassland. Our study suggests that in arid and semi-arid areas, vegetation achieves the optimal water retention capacity when the vegetation fractions are 1.4%–3.7% and 3.8%–7.2%, respectively. Hence, vegetation should not be further expanded in semi-humid areas, but should be further restored in arid and semi-arid areas with sparse or excessively sparse vegetation cover (especially in desert).