A new-type compound (DT) which contained phosphaphenanthrene and triazine-trione groups was synthesized. DT was served as a reactive flame retardant for unsaturated polyester resin (UP). The thermal degradation, flame-retarded and mechanical properties of UP/DT samples were detected by different tests. According to the results, the addition of DT improved the initial thermal decomposition temperature (T5% and T10%) and the char yields of UP thermosets. Additionally, incorporation of DT resulted in the decrease of flexural and tensile strength of UP samples, and the increase of flexural modulus. The flame-retarded performance of UP/DT samples was greatly improved compared with the neat UP thermoset. For instance, the limited oxygen index (LOI) and vertical burning (UL94) rating of UP/DT-30 sample with 30 wt% DT came up to 29.8% and V-1. In comparison to pure UP thermoset, the average of heat release rate (av-HRR), total heat release (THR) and average of effective heat of combustion (av-EHC) of UP/DT-30 thermoset were decreased by 35.9%, 31.2% and 29.1%, respectively. Phosphaphenanthrene and triazine-trione groups in DT synergistically enhanced flame-retarded capability of UP in both gas phase and condensed phase.