The contamination of heavy metal has permeated many parts of China, especially in densely populated and industrialized southern China. This study focused on the degree of pollution in farmland soil heavy metals (HMs), and its spatial distribution characteristics and source apportionment. Meanwhile, we conducted an evaluation of the health risks attributed to soil HMs and analyzed the factors that impact them. We found that the distribution of five heavy metals is mainly concentrated in the east-central and southern parts of the study area. Specifically, Cd and Hg have high levels of pollution and present potential ecological risks. The pollution sources of five HMs were analyzed utilizing positive matrix factorization. The results revealed that the contribution of different sources keeps the following order: natural source (42.42%), agricultural activities (29.93%), industrial pollution source (20.49%), and atmospheric deposition pollution (7.16%). The non-carcinogenic risks to residents were acceptable, whereas the carcinogenic risks were relatively high. Children and the elderly are more vulnerable to the negative effects of Cr, As. Using structural equation modeling, we found soil property is a vital factor affecting soil contamination, with the soil organic matter and cation exchange capacity having a relatively greater impact on heavy metals pollution. Our study provides some data reference and guidance for soil ecological protection and restoration.