The selection and mixing proportion of cementitious materials in cemented backfills are of great engineering importance to ensure the quality of the backfill. Therefore, based on the mixing proportion test of magnesium slag-phosphogypsum composite cementitious materials, the strength change characteristics and volume change rate of the cemented backfill under different cementitious material proportions were investigated. Combined with indoor tests such as hydration heat test, X-ray diffraction, thermogravimetric-differential thermogravimetric, scanning electron microscopy, and mercury intrusion porosimetry, the hydration mechanism of the cemented backfill under different cementitious material proportions was investigated. The results showed that the highest strength of the cemented backfill was observed with only magnesium slag as the activator. The strength of magnesium slag-phosphogypsum and magnesium slag-desulfurized gypsum cemented backfill was greater than that of cement cemented backfill, yet lower than that of magnesium slag alone cemented backfill. The shrinkage trend and magnitude of magnesium slag alone excites blast furnace slag cemented backfill and cement cemented backfill were similar. With the increasing content of phosphogypsum and desulfurized gypsum, the shrinkage of the backfill tended to increase, especially when the content was 18%. The gel-like products of backfill with magnesium slag-based cementitious materials increased rapidly at 7 days, which strongly bonded the aggregates and filled the backfill pores, resulting in high backfill strength. Although the porosity of backfill with magnesium slag-based cementitious material showed an increasing trend with the increase of age, its most available pore size decreased and harmful pores became less. The results can provide a reference for the application of magnesium slag-phosphogypsum composite cementitious materials.