Tailings were discharged to the Boroo River from gold mining by amalgamation, resulting in soil contamination near the river. To identify the sources and distribution mechanisms of each metal(loid) in the soil, a total of 184 soil samples were collected near the river and analyzed for As, Cd, Cu, Pb, Zn, and Hg contents. According to the positive matrix factorization result, three factors affected the contamination levels: the application of Hg for gold mining (Factor 1), light minerals containing Cu and Zn (Factor 2), and heavy minerals containing As and Cd (Factor 3). Soil samples were classified into four groups by hierarchical clustering. Groups A and B seemed to be affected by light and heavy minerals discharged in early and later stages of ore processing, respectively. The spatial distribution of the groups suggested differentiation in travel distances by specific gravity. Groups C and D showed high Hg contents implying the effect of Hg mismanagement and spill accidents. The study results show that the distribution of soil contaminants near rivers in mining areas is controlled by the specific gravity of minerals discharged to the environment (e.g., river), ore processing stages, and insufficient recovery and/or spills of Hg, which will help establish restoration measures.