Rapid urbanization has accelerated the accumulation of trace metal(loid)s (TMs) in soils, but the relationship between this accumulation and human activities remains largely unknown. Therefore, based on 775 published literatures (2001–2020), this study aimed to identify the influence of human activities on TM accumulation. Results showed that all soil TM concentrations were higher than their corresponding Chinese soil background values. The pollution risk assessment indicated that the soil TMs in the study area were at moderate levels, and the value of Pollution load index was 2.10. According to the assessment of health risks, the non-carcinogenic risks for adults were at the “Negligible risk” level; while the carcinogenic risk was not negligible for all populations, with children being more susceptible than adults. Meanwhile, six high-risk TMs were identified based on the grading of Contaminating factors (CF ≥ 3) and contribution to health risk (≥ 75%), including four high pollution risk TMs (Cd, Hg, Cu, and Pb) and two high health risk TMs (Cr and As) . In addition, in accordance with the results of the Random forest model, the accumulation of soil high-risk TMs was closely related to influencing factors associated with human activities. The accumulation of Hg and Cr among five major urban agglomerations had the same influencing factors (the number of industrial companies and the amount of industrial wastewater discharge for Hg; the amount of pesticide application and highway mileage for Cr). However, there were significant differences in the factors influencing the accumulation of the other four high-risk TMs (including Cd, As, Cu and Pb), due to the different characteristics of each urban agglomeration. Our results provide new insights into the relationship between human activities and soil TM accumulation.