Soil particle size distribution (PSD), bulk density (BD), saturated hydraulic conductivity (Ks), soil water content (SWC) and generalized soil structure index (GSSI) are key physical properties for vegetation growth and are easily affected by natural and anthropogenic factors. However, how these soil physical properties respond to land use, slope aspect, slope position, and their interactions are less clearly understood. Here, an experiment with three land uses (cropland, grassland, and black locust [Robinia pseudoacacia L.] forests after 4, 9, 15, and 19 years of restoration), two slope aspects (south and north slope) and two slope positions (up and down slope) were conducted in the hilly loess region of China. The results showed the following: (1) The variation in different physical properties were driven by different factors of land use, slope aspect, slope position and their interactions. (2) The 9, 15 and 19 year old locust forests significantly decreased the BD by 6.75 %–19.32 % and increased the Ks by 69.96 %–195.97 % compared with grassland, while the cropland had higher SWC and GSSI values than the grassland and locust forests. (3) Soil physical properties were higher on the north slope than on the south slope and were larger on the down slope than on the up slope. (4) The BD, Ks, SWC, and GSSI of locust forests displayed different changes with the restoration year. (5) The key factor affecting forest growth and soil quality improvement varied with forest ages. Overall, 19 year old locust was the most effective at improving the BD and Ks. Furthermore, slope aspect, slope position and restoration time should be considered when evaluating the response of physical properties to land uses.