Conventional methods for identifying soil heavy metal (HM) pollution sources are limited to area scale, failing to accurately pinpoint sources at specific sites due to the spatial heterogeneity of HMs in mining areas. Furthermore, these methods primarily focus on existing solid waste polluted dumps, defined as “direct pollution sources”, while neglecting existing HM pollution hotspots generated by historical anthropogenic activities (e.g., mineral development, industrial discharges), defined as “potential pollution sources”. Addressing this gap, a novel remote sensing analysis method is proposed to identify both direct and potential pollution sources at site scale, considering source-sink relationships. Direct pollution sources are extracted using a supervised classification algorithm on high-resolution multispectral imagery. Potential pollution sources depend on the spatial distribution of HM pollution, mapped using a machine learning model with hyperspectral imagery. Additionally, a source identification algorithm is developed for gridded pollution source analysis. Validated through a case study in a cadmium (Cd)-polluted mine area, the proposed method successfully extracted 21 solid waste polluted dumps with an overall accuracy approaching 90 % and a Kappa coefficient of 0.80. Simultaneously, 4167 HM pollution hotspots were identified, achieving optimal inversion accuracy for Cd (Rv2 = 0.91, RMSEv = 4.27, and RPDv = 3.02). Notably, the spatial distribution patterns of these identified sources exhibited a high degree of similarity. Further analysis employing the identification algorithm indicated that 3 polluted dumps and 258 pollution hotspots were pollution sources for a selected high-value point of Cd content. This innovative method provides a valuable methodological reference for precise prevention and control of soil HM pollution.