Water Safety Plans (WSPs) are internationally recommended risk assessment and management strategies for water delivery. However, documented outcomes and impacts from implementing WSPs are lacking, particularly for community-managed supplies. In this research, previously implemented, community-managed WSPs were evaluated in four countries through a mixed-methods protocol assessing: WSP implementation quality, WSP management via key informant interviews (KII) and focus group discussions (FGD), household characteristics, and collection-point and stored household water quality in WSP and non-WSP communities. Overall, 256 KIIs and FGDs, 816 household surveys, and 1,099 water quality samples were completed. The quality of WSP implementations scored 6–13 out of 18 possible points. KIIs and FGDs found WSP communities had improved capacity to manage water supplies and identify key risks to safe water delivery. Fewer non-WSP community households reported paying for water in DRC and Vanuatu (p < 0.001). WSP community water collection-points had more E. coli contamination than non-WSP community collection-points in DRC (p = 0.009), Fiji (p = 0.020), and Vanuatu (p = 0.004); household results varied, although exploratory analysis revealed that non-WSP communities were imperfectly matched to WSP communities. Overall, we found: 1) incomplete WSP implementations; 2) small improvements in water supply operations; and, 3) no documented microbiological water quality improvements from WSPs. These results highlight that WSPs implemented in rural, community-managed supplies are challenging and question their effectiveness; if implemented additional technical and financial resources are necessary to support community-managed WSPs.