ABSTRACTAmmonium polyphosphate (APP) was modified using a new method, where the resulting modified APP (MAPP) was obtained by mixing APP with unsaturated polyester resin (UPR). MAPP was more effective in improving the flame retardancy of UPR than APP which was due to the improved dispersion of MAPP in UPR composite. Then, the UPR composites were prepared based on dimethyl methylphosphonate, MAPP, montmorillonite, and zinc borate. Finally, the flame‐retardant and mechanical properties of the UPR composites were analyzed using the limited oxygen index (LOI), thermogravimetric analysis, UL‐94 vertical burning test, scanning electron microscopy, cone calorimetry, mechanical tests, and viscosity measurements. The LOI and UL‐94 tests showed that the flame‐retardant properties clearly improved with the addition of fillers in the UPR composites compared to pristine UPR. The synergistic effect of Si‐ and P‐containing flame retardants in this composite resulted in the LOI value increasing from 18.9 to 31.3% and achieved the UL‐94 V‐0 rating. Moreover, the heat release rate was lower than the pristine UPR. © 2018 Wiley Periodicals, Inc. J. Appl. Polym. Sci. 2019, 136, 47180.