The evolution of hyperspectral remote sensing and artificial intelligence technologies has led to a surge in their application for predicting Soil Heavy Metal Concentrations (SHMC). Nevertheless, the preponderance of existing research within this sphere centers around data procured from ground-based and airborne hyperspectral sources. Studies that employ satellite-based methodologies typically rely on medium spatial resolution hyperspectral or multispectral satellite data. The application of high spatial and spectral resolution satellite data, such as that obtained from GaoFen-5 (GF-5), remains conspicuously underexplored. Furthermore, the impact of geographical environmental factors (GEFs) on the accuracy of predictions has been infrequently considered. In the context of this backdrop, the present study introduces stacking models designed to estimate SHMC. This approach integrates reflectance spectral features (SFs) derived from GF-5 hyperspectral imagery and GEFs, including topography and pollution sources. The results demonstrate a notable improvement in the predictive accuracy of SHMC using our Stacking model, as compared to single models. The incorporation of GEFs into the method results in a varying degree of reduction in the Root Mean Square Error (RMSE), along with an enhancement in the R2 on the training set. The predictive performance improvement is most prominent for Cd and As, with the RMSE decreasing by 52% and 48%, respectively. Notably, apart from Pb, there is an improvement in performance for all elements within the test set. This study confirms the effectiveness of integrating GEFs into SHMC prediction models to enhance accuracy. Applying this technique to predict soil pollution at a regional scale and to demarcate heavily polluted areas can yield satisfactory results. In the future, we plan to apply this technique to other research areas or datasets to expand its universality. Furthermore, we aim to delve more deeply into the potential of GEFs to enhance the predictive capacity of SHMC.