Silica nanoparticles (SNP) and core‐shell‐rubber (CSR) nanoparticles were added to a lightly crosslinked epoxy in an effort to improve fracture toughness. Properties such as the glass transition temperature, tensile modulus, and yield strength were also measured. Toughening mechanisms were studied by inspecting the fracture surfaces with scanning electron microscope and exploring subsurface damage using both transmission optical microscope and transmission electron microscope. Ternary nanocomposities with a fracture energy five times that of the unmodified epoxy resin were obtained with no discernible drop in the tensile modulus. The increase in stiffness due to the presence of SNP and the increase in compliance due to the presence of CSR were accurately predicted using the Mori‐Tanaka micromechanical model. Unfortunately, the use of SNP did not result in the retention of strength in the ternary nanocomposites. Maximum toughness was observed when the SNP content was 4 vol% and the CSR content was 7 vol%. This combination of particles also resulted in the largest damage zone (subsurface), which agrees well with fracture toughness results. The interaction of toughening mechanisms in rubber‐SNP‐epoxy systems was found to be affected by the interparticle distance between rubber particles. The non‐debonding SNPs between rubber particles could act as shear band stoppers and/or ductility suppressors, which suppressed the toughening performance of rubber particles. On the other hand, the SNPs induce more shear bands between SNPs to increase the toughening performance. The total toughening performance depends on the rubber interparticle distance. POLYM. COMPOS., 39:3482–3496, 2018. © 2017 Society of Plastics Engineers