It is a great challenge to develop a high-efficiency reactive flame retardant, applied to anhydride-cured epoxy resin (EP) system, simultaneously possessing good compatibility with matrix and mechanical reinforcement. In this respect, we successfully synthesized a novel phosphorus/nitrogen/boron-containing carboxylic acid (TMDB) through the facile esterification and addition reaction among 1,3,5-tris(2-hydroxyethyl)isocyanurate (THEIC), maleic anhydride (MAH), 9,10-dihydro-9-oxa-10-phosphaphenanthrene-10-oxide (DOPO) and boric acid (BA). TMDB was utilized as a co-curing agent for EP/methyltetrahydrophthalic anhydride (MeTHPA) system and finally cured EP behaved great transparency, suggesting excellent compatibility of TMDB with EP. Compared with pure EP, modified EP exhibited comparable thermal stability and heat resistance but higher flame retardance. With only 15.1 wt% TMDB loading, the LOI value of anhydride-cured EP increased to 29.6% from 20.1% of pure EP, and UL-94 V-0 rating was achieved. The peak heat release rate (PHRR), total heat release (THR) and total smoke production (TSP) remarkably decreased by 58.5%, 41.7% and 47.2% compared with that of pure EP, respectively. Besides, different measurements revealed TMDB simultaneously functioned in the condensed and gaseous phase during combustion. Furthermore, after incorporation of TMDB, mechanical properties of cured EP were improved and the maximum increments of flexural and tensile strength can reach 11.8% and 61.4%, respectively.