Heavy metals (HMs) are often abnormally enriched in soils in geologically high background areas, posing a serious threat to the ecosystem and human health. However, the material origin of HMs in overlying soils is unclear. In this paper, we studied the accumulation control and formation process of HMs (Cd, Hg, As, Pb, Cr, Cu, Ni, and Zn) in soils overlying bedrock from different geological periods in a typical anticline structure area in southwest of Guizhou province. The results revealed that the content of HMs in the bedrock were ranked as Carboniferous carbonate rocks < Permian carbonate rocks < Triassic clastic rocks, while the opposite trend was observed for the HMs content in the overlying soil. The slopes of the EF-Q values of HMs in the soils overlying the Carboniferous, Permian, and Triassic rocks were 0.003, 0.007, and 0.83, respectively, which suggests that HMs enrichment or leaching was more complicated in the process of carbonate weathering soil-forming, whereas it was mainly in situ weathering during the process of clastic weathering soil-forming. The soils overlying the bedrock of the three geological periods experienced similar weathering processes and had the same source of soil-forming matrices, as it was observed that their Fe2O3-Al2O3, Hf-Zr, Nb-Ta, Th/Sc-Zr/Sc and Y/Ho-Y contents were significantly correlated (with R2 values of 0.80, 0.94, 0.72, 0.60, and 0.76, respectively) with very similar REE assignment patterns. The results corroborated the causes of the high background geological heterogeneity of HMs in the overlying soils in karst areas and supported the theory of the source of weathering soil-forming materials from carbonate rocks in karst areas.