Modified graphene/muscovite (MGMu) nanocomposite was synthesized with muscovite (Mu) and silane coupling agent modified graphene oxide through a simple hydrothermal method that exhibited excellent dispersion stability in oil. Compared with the base oil sample, the average friction coefficient and wear scar diameter of the MGMu oil sample decreased by 64.4 and 20.0%, respectively, and the microhardness of its wear scar was increased by 16.1%. The MGMu showed better tribological performance than its individual component due to the synergetic effect between the two components. The lubrication mechanism was proposed according to the morphology, chemical composition, and microhardness of the surface of wear scars. MGMu as an oil additive could fill between the friction pairs, cling to some asperities, and occur relative sliding between unit layers, thus playing a role in lubrication. It was found that MGMu would react with the surface of the friction pair during the friction process to generate Fe2O3, SiO2, SiC, and new aluminosilicate, which formed a self-repairing layer with high hardness. This chemically reactive film exhibited a lower shear strength, which made the oil sample containing MGMu have a lower coefficient of friction.