The preparation of gypsum-based self-leveling mortar (GSLM) from solid waste-derived sulfoaluminate cement-modified hemihydrate gypsum is an effective path to enhance its mechanical properties. However, the phosphorus impurities in phosphogypsum (PG) still pose a challenge to the hemihydrate gypsum & sulfoaluminate cement composite system. To evaluate the impact of phosphorus impurities and develop targeted removal strategies, this study first assessed the effects of different types of phosphorus impurities on the mechanical properties and hydration characteristics of the GSLM. Then, a method of pretreating PG with carbide slag (CS) was proposed, and the removal efficiency and mechanism of phosphorus were examined by ICP-OES and XPS testing. The results indicated that the soluble phosphorus impurities severely inhibit the hydration of both CŜH0.5 and C4A3Ŝ, leading to a most notable reduced mechanical performance. The sequence of inhibitory action was as follows: H3PO4 >> Ca(H2PO4)2·H2O > CaHPO4·2H2O > Ca3(PO4)2. CS could effectively immobilize the soluble phosphorus in PG. The pretreatment by CS notably enhanced the hydration degree of the GSLM by converting soluble phosphorus into Ca3(PO4)2 beforehand. In addition, the alkaline environment provided by CS promoted the ettringite formation. When the CS content approaches 3%, a PG-based GSLM with good performances meeting the G20 standard requirements of GSLM can be prepared.