Organic fertilizer can improve phosphorus (P) bioavailability, while the mechanism of microbial-driven clay mineral transformation on P retention and utilization with organic fertilization is still unclear. An 8-year organic fertilization experiment was carried out in saline alkaline paddy soil with five treatments: (1) CK, no fertilization; (2) NK, mineral N and K fertilization; (3) NPK, NK with mineral P fertilization; (4) NPKC1, NPK with 450 kg C ha−1 year−1; (5) NPKC2, NPK with 900 kg C ha−1 year−1. The results showed that, compared with NPK treatment, the percentage of illite with more P adsorption sites was increased by 26.8% and 20.7% in NPKC1 and NPKC2 treatments, respectively. Meanwhile, transmission electron microscopy-energy dispersive X-ray spectroscopy analysis further confirmed that there was more P adsorption on silicate clay minerals. Compared with NK treatment, the content of total P and available P was significantly increased with organic fertilization. Besides, the diversity of bacterial community was also increased by organic fertilizer addition, redundancy analysis showed that soil organic carbon was the main factor in the diversity of bacterial community. The relative abundance of Firmicutes and Gemmatimonadetes was promoted, which drove the transformation of clay minerals and improved P availability, respectively. Compared with NPK treatment, P use efficiency in NPKC1 and NPKC2 treatments was increased by 47.2% and 58.2%, respectively, while no significant difference was found between the two treatments. Therefore, appropriate organic fertilization was a better fertilization practice to improve P retention and utilization through bacteria-driven clay mineral transformation in saline-alkaline paddy soil.