Phosphoric acid (PA)-based geopolymers are different from alkali-activated materials. The PA-based geopolymers have good mechanical properties, but the related research is still less, especially the research concerning the durability of geopolymers has received less attention, which has brought some trouble to the engineering application. In this article, to further study the water resistance of PA-based geopolymer prepared using fly ash, the unconfined compressive strength tests were conducted to evaluate the water resistance. Meantime, the Scanning Electron Microscopy (SEM), SEM-EDS, SEM mapping, Mercury Intrusion Porosimetry, X-ray Diffraction, and Infrared Spectroscopy tests were used to reveal the mechanism of strength weakening of the geopolymers after soaking. The results show that the water resistance of the geopolymer first increased and then decreased with the increasing PA concentration. When the PA concentration was 50%, the geopolymer with the optimal PA concentration gave the best water resistance. The compressive strength was decreased significantly after soaking, and the strength first decreased rapidly and then reduced slowly with the soaking time increasing. At 7 days of soaking, the strength decreased rapidly with a decrease of 30–50%. After 7 days of soaking, the strength decreased slightly. After soaking, the at 180 days of soaking, the geopolymer specimens cracked except for the geopolymer with the optimal PA concentration. If the water stability of the acid-based geopolymer is not improved, it is difficult to meet the requirement of engineering application. The microscopic analysis shows that quartz and mullite did not participate in the polymerization, and the crystalline compounds formed in the geopolymer matrix, such as CaPO3(OH)·2H2O, CaHPO4, AlPO4, and CaPO3·OH, basically did not change after soaking, but some calcium-containing crystalline compounds leached after recrystallization after soaking in water, and the Si–O–P unit in the amorphous gel in the geopolymer hydrolyzed after soaking, also leading to the decrease the geopolymer strength.