Agroforestry has great potential for improving the ecological environment and raising the ecosystem productivity. This study is aim to evaluate the soil quality of bamboo forest and identify the main site factor that influencing the yield of Polygonatum tubers following the agroforestry practices. Four intercropping modes with different culm densities were conducted in Anhui, East China. The minimum data set method (MDS) was applied to evaluate soil quality index (SQI). Based on principal component analysis (PCA), microbial biomass carbon (MBC), available potassium (AK), altitude, and pH were selected as the MDS. The results showed that intercropping significantly increased SQI compared with moso bamboo monoculture. The mode M1(1750 culms/ha) was significantly better than other modes in improving soil fertility and increasing the yield of Polygonatum tubers. Results from random forest and structural equation model (SEM) showed the direct path coefficient of density on Polygonatum tubers yield was the largest, which was −0.83. Altitude, soil organic carbon (SOC) and nitrate nitrogen (NO3−-N) had a small impact, with direct path coefficient of 0.16, 0.10, and 0.15 (p < 0.05), respectively. Our findings provide a theoretical basis for managing the moso bamboo and P. cyrtonema agroforestry system in a more effective manner.