Phosphogypsum is an industrial waste generated in the production of fertilizers causing environmental problems. Calcined phosphogypsum (CPG) is obtained by low-temperature calcination and dehydration of phosphogypsum. CPG is mainly used to produce building materials. However, mechanical strength properties and water resistance of CPG materials is extremely poor, which leads to its relatively low utilization rate in construction industry. To address this issue, various amount of Ordinary Portland cement (OPC), fly ash (FA), rice husk (RH) and modified admixtures (AD) were mixed with the CPG to obtain CPG-based composite cementitious materials, which enhanced the water resistance and mechanical strength of the corresponding hardened materials. The influence of different contents of cement on the setting time, hydration heat, hydration products, mechanical properties, water absorption, heavy metal leaching and deterioration coefficient of the composite cementitious material is investigated. The results showed that the optimal mixing ratio of CPG: OPC: FA: RH was 57: 20: 10: 13 considering the water resistance and mechanical strength properties of CPG-based composite cementitious materials. On these conditions, the compressive strength, the flexural strength, water absorption and deterioration coefficient of the specimens at 28 days are 6.1 MPa, 4.6 MPa, 29.4% and 0.66 respectively, which were in line with the national standard of phosphorus building gypsum wallboard. It was demonstrated that two skeleton types are present in the samples prepared using CPG-based composite cementitious materials, which had a synergistic support effect on the structure. The first skeleton was formed by gypsum dihydrate crystals (CaSO4·2H2O) and provided strength to the specimens at an early stage, while the second skeleton was formed by ettringite (AFt) and C–S–H gels, which provided strength at a later stage. The skeletons make the hardened materials denser and more waterproof. Moreover, the composite materials exhibited a positive solidification effect on heavy metals present in CPG.