The covalent grafting between the nanoparticles and polymer matrix is considered a useful design factor to control the mechanical properties (e.g., fracture toughness). Meanwhile, in the theoretical viewpoint, the toughening mechanisms of the epoxy nanocomposites with covalent grafting are still unclear. In this study, a new multiscale fracture model is proposed to quantify the toughness enhancement of epoxy nanocomposites induced by the covalent grafting. Especially, it was concentrated on the investigation of the influences of the cohesive failure mode on the fracture toughness and revealed that the failure mode transition (from the adhesive failure mode to cohesive failure mode) near the nanoparticles with increasing grafting ratio is critical factor for the toughening mechanisms of the epoxy nanocomposites with covalent grafting. The theoretically derived multiscale model demonstrated that the cohesive failure mode significantly enhanced the fracture toughness of polymer nanocomposites by dissipating more energy compared to the adhesive failure mode. This study underscores the critical importance of rationally designing nanocomposites with an optimal grafting ratio that properly reflects the cohesive failure mode. It is anticipated that the multiscale analysis approach outlined in this study will provide valuable design guidelines and insights for polymer nanocomposites incorporating covalent grafting.