The historical large mercury slag piles still contain high concentrations of mercury and their impact on the surrounding environment has rarely been reported. In this study, three different agricultural areas [the area with untreated piles (PUT), the area with treated piles (PT), and the background area with no piles (NP)] were selected to investigate mercury slag piles pollution in the Tongren mercury mining area. The mercury concentrations of agricultural soils ranged from 0.42 to 155.00mg/kg, determined by atomic fluorescence spectrometry of 146 soil samples; and mercury concentrations in local crops (rice, maize, pepper, eggplant, tomato and bean) all exceeded the Chinese food safety limits. Soil and crop pollution trends in the three areas were consistent as PUT > PT > NP, indicating that mercury slag piles have exacerbated pollution. Mercury in the slag piles was adsorbed by multiple pathways of transport into soils with high organic matter, which made the ecological risk of agricultural soils appear extremely high. The total hazard quotients for residents from ingesting mercury in these crops were unacceptable in all areas, and children were more likely to be harmed than adults. Compared to the PT area, treatment of slag piles in the PUT area may decrease mercury concentrations in paddy fields and dry fields by 46.02% and 70.36%; further decreasing health risks for adults and children by 47.06% and 79.90%. This study provided a scientific basis for the necessity of treating large slag piles in mercury mining areas.