Mercury (Hg) contamination and accumulation in food crops is a global threat posing potential health risk to humans. However, Hg phytoavailability in soil-pepper system and its influencing factors largely remain unknown. In this study, a greenhouse pot experiment was conducted to grow peppers using 21 Chinese agricultural soils with varied soil properties and aged Hg levels. Mercury concentration in pepper leaves and fruits ranged from 0.021 to 0.057 mg kg−1 and 0.005–0.022 mg kg−1 respectively, while fruit Hg content in three soils (Anhui, Hubei, Beijing) exceeded the safety limit. Fruit Hg concentration was better positively correlated with soil Mg(NO3)2-extractable Hg content (r = 0.7, P < 0.0001) than soil total Hg content (r = 0.45, P < 0.0001). Highest bioconcentration factor (BCF, ratio of Hg plant to Hg soil) yielded in acidic soils, while the lowest BCF occurred in alkaline soils. Path analysis indicated available-Hg (R2 = 0.40) and total-Hg (R2 = 0.40) had direct positive effects on the pepper fruit Hg concentration, while direct negative effects including pH (R2 = −0.86), organic matter (R2 = −0.7), crystalline-Fe (R2 = −0.68). Those agreed with the stepwise multiple linear regression analysis which yielded a regression predictive model (R2 = 0.73, P < 0.0001). Soil available-Hg, total-Hg, pH, organic matter and crystalline-Fe & Mn were the most influencing factors of Hg phytoavailability. These results provide new insights into the phytoavailability of Hg in soil-pepper system, thus facilitating the management of pepper cultivation in Hg-enriched soils.