This paper illustrates a spectroscopic analysis of heavy metal concentration in mine soils with the consideration of mineral assemblages originated by weathering and mineralization processes. The mine soils were classified into two groups based on the mineral composition: silicate clay mineral group (Group A) and silicate–carbonate–skarn–clay mineral group (Group B). Both soil groups are contaminated with Cu, Zn, As, and Pb, while the contamination level was higher for Group A. The two groups exhibit different geochemical behaviors with different heavy metal contamination. The spectral variation associated with heavy metal was highly correlated with absorption features of clay and iron oxide minerals for Group A, and the absorption features of skarn minerals, iron oxides, and clay minerals for Group B. It indicates that the geochemical adsorption of heavy metal elements mainly occurs with clay minerals and iron oxides from weathering, and of skarn minerals, iron oxides, and clay minerals from mineralization. Therefore, soils from different secondary mineral production processes should be analyzed with different spectral models. We constructed spectral models for predicting Cu, Zn, As, and Pb in soil group A and Zn and Pb in soil group B using corresponding absorptions. Both models were statistically significant with sufficient accuracy.