PurposeBismaleimide (BMI) is a kind of thermosetting resin and its application is usually limited by low toughness. In this paper, two kinds of reinforcement intercalator amino-terminated polyoxypropylene (POP) and octadecyl trimethyl ammonium chloride (OTAC) were designed and synthesized to toughen BMI resin and the toughening effect was compared and analyzed. The purpose of this paper is to toughen BMI resin and analyze the toughening effect of two reinforcements intercalator amino-terminated polyoxypropylene (POP) and octadecyl trimethyl ammonium chloride (OTAC).Design/methodology/approachSodium-based montmorillonite (Na-MMT) was modified by POP and OTAC, and the ion-exchange reaction obtained organic montmorillonite (POP-MMT and OTAC-MMT). The polymer matrix (MBAE) was synthesized, in which 4,4’-diamino diphenyl methane BMI was used as the monomer and 3,3’-diallyl bisphenol A and bisphenol A diallyl ether were used as active diluents. And then, POP-MMT/MBAE and OTAC-MMT/MBAE composites were prepared using MBAE as matrix and POP-MMT or OTAC-MMT as reinforcement. The Fourier-transform infrared, X-ray diffraction and scanning electron microscope (SEM) of the filler and microstructure and mechanical properties of the composite were characterized to the better reinforcement.FindingsPOP-MMT and OTAC-MMT enhanced BMI-cured products’ toughness by generating microcracks in the polymer to absorb more fracture energy. Meanwhile, POP-MMT and OTAC-MMT were the main stress components and the enhancement of the interface interaction was beneficial to transfer the external force from the matrix to the reinforcement and improved the mechanical properties of the composite. Furthermore, with the intercalation rate increasing, the compatibility of the two phases was increased and the performance of MBAE was also elevated.Research limitations/implicationsBMI is generally used as aerospace structural materials, functional materials, impregnating paint and other fields. However, high crosslinking density leads to moulding material’s brittleness and limits a wider range of applications. Therefore, it has become an urgent priority to explore and improve the mechanical properties of BMI resin.Originality/valuePOP and OTAC have successfully intercalated Na-MMT layers to get POP-MMT and OTAC-MMT, and the interplanar crystal spacing and the intercalation rate were calculated, respectively. The results were corresponding with the SEM images of POP-MMT and OTAC-MMT. After that, the morphology of composites illustrated the compatibility was related to the intercalation rate. According to the mechanism of modified MMT toughening epoxy resin, when they were dispersed uniformly in the matrix, the composite’s mechanical properties had been significantly improved. Additionally, OTAC-MMT with a higher intercalation rate had better compatibility and interfacial force with the matrix, so that the mechanical properties of OTAC-MMT/MBAE were the best.