Surface-enhanced Raman scattering (SERS) is a convenient and fast technique to detect toxic heavy metals such as lead ions (Pb2+). In this study, a silver nanoparticle-decorated hydrogel substrate by a one-pot method is prepared, addressing the instability of traditional sol substrates and the non-tunability of localized surface plasmon resonance (LSPR) of solid substrates, which can achieve in-situ detection of Pb2+ in real systems using a portable Raman spectrometer. Meanwhile, the characteristic SERS peaks attributed to the ternary complex of 4-MBA/H2PO4-/Pb2+ formed by the dihydrogen phosphate, 4-mercaptobenzoic acid, and Pb2+ are used firstly for Pb2+ detection to our knowledge, decreasing the cost of the classical strategy. The method is highly anti-interference, simple, and cost-effective, which exhibits a good linear correlation (y= 21.6x + 2711, R2=0.993) in the range of 3–500 μg/L with a detection limit of 1.6 μg/L that is one order of magnitude lower than that of the EPA regulations. This method is expected to realize the rapid and accurate on-site SERS detection of Pb2+ by the new strategy.