The effects of the type of raw material, mix proportion, and chemical modifier on the mechanical properties and water resistance of magnesium oxysulfate cement (MOS) have been extensively studied. However, the formation mechanism of hydration products remains obscure. This study aims to clarify the hydration mechanism of MOS through a thermodynamic modeling approach and to explore the effects of mix proportion and type and dosage of modifier on the phase assemblage as well as the mechanical properties of hardened paste. The volume fraction of 5Mg(OH)2⋅MgSO4⋅7H2O (5-1-7) is used as a key parameter to establish the relationship between the mix proportion and the compressive strength of the MOS system. The results demonstrate that the thermodynamic modeling method can predict the phase evolution of MOS with various mix proportions. As the molar ratio of reactive-MgO (α-MgO) to MgSO4 increases, both the 5-1-7 volume fraction and compressive strength of hardened paste initially increase and then decrease, while decreasing as water dosage increases. The volume fraction of 5-1-7 is proportional to the compressive strength of MOS. Compressive strength is greatest at molar ratio of nMgO:nMgSO4 = 4–9 and nH2O:nMgSO4 = 9–20. The efficiency of modifiers is: citric acid (CA) > tartaric acid > EDTA > phosphoric acid > acetic acid. The modifying mechanism of CA involves the initial formation of 3Mg(OH)2⋅MgSO4⋅8H2O(3-1-8), followed by its transformation into 5-1-7.