To better understand the influence of the impurities in phosphogypsum (PG) on the early-age hydration of calcium sulfoaluminate (CSA) cement manufactured with PG, the study prepared synthetic ye’elimite doped with fluorine (F) and phosphorus (P) and also synthesized ye’elimite with natural gypsum (NG) and PG. The acquired ye’elimite was characterized, and its early-age hydration within 24 h was investigated. Results demonstrate that F contributes to the crystallization and leads to the formation of large crystal synthetic ye’elimite. P and the simultaneous presence of F and P could stabilize C4A3$-c. Without the presence of gypsum, pure ye’elimite is much more active than synthetic ye’elimite doped with F and P, and synthetic ye’elimite prepared with NG is much more active than that prepared with PG. Although the presence of gypsum significantly boosts the hydration activity of synthetic ye’elimite doped with F and P and pared with PG, their hydration activity is still slightly lower. It is suggested that manufacture of alite-calcium sulfoaluminate cement and binary blends of CSA cement and Portland cement is an available way to boost the early-age hydration CSA cement manufactured with PG.