The combination of surface-enhanced Raman scattering (SERS) and optical fiber has led to the development of SERS optical fiber probe, which demonstrates potential for real-time, in-situ, and flexible detection of water contamination. In this paper, a highly sensitive SERS optical fiber probe of gold nano-bipyramids (Au NBPs) was successfully developed for the first time. The Au NBPs were controllably synthesized by a heat-treated seed growth method. Surface modification of the Au NBPs was carried out by adding poly (sodium 4-styrenesulfonate) (PSS) instead of hexadecyl trimethyl ammonium bromide (CTAB), which changed the Zeta potential of the Au NBPs solution from positive to negative. The SERS properties of the Au NBPs can be significantly enhanced by optimizing the parameters of PSS content. The optical fiber end face was pretreated with silane coupling agent to introduce amino groups and exhibit positive charge. As a result, highly sensitive SERS optical fiber probes were prepared through electrostatic adsorption of the Au NBPs. The performance of the SERS optical fiber probe was evaluated. The crystalline violet (CV) limit concentration of 10−10 mol/L was detected by the optimized SERS optical fiber probe with satisfactory reproducibility and stability. Additionally, a limit concentration of 10−5 mol/L for thiram could also be detected by the SERS optical fiber probe. Finally, the SERS enhancement mechanism was further discussed. The finite element method was utilized to analyze the electromagnetic field distribution of Au NBPs, optical fibers, and SERS optical fiber probes, theoretically explaining the performance advantages of Au NBPs SERS optical fiber probes. Based on our results, the Au NBPs SERS optical fiber probe holds significant value for trace detection of water contamination and biomolecules.