ABSTRACTZnO nanoparticle surface‐modified by diisooctyl phosphate (P204) was prepared by liquid‐phase in situ surface modification technique with zinc acetate as the raw material, P204 as the modifier and anhydrous ethanol as the solvent. The morphology and microstructure of the P204‐ZnO nanoparticle were characterised by transmission electron microscopy, X‐ray diffraction and Fourier transform infrared spectrometry. Its thermal stability was evaluated by thermogravimetric analysis; and its tribological properties as the additive in alkylnaphthalene were evaluated with an SRV‐5 friction and wear tester in reciprocal sliding mode. The results show that the as‐prepared P204‐ZnO nanoparticle has an average particle size of about 4 nm and a P204 content of about 42% (mass fraction); and the surface modifier is grafted onto the surface of ZnO nanoparticle by physical adsorption. With a mass fraction of 0.5% in alkylnaphthalene base oil, P204‐ZnO nano‐additive can mildly reduce the friction coefficient and drastically reduce the wear rate of the steel–steel sliding pair. This is due to the formation of the composite tribofilm via the adsorption and deposition of the nano‐additive on rubbed steel surfaces as well as the tribochemical reactions of the modifier P204 yielding phosphate and of steel substrate yielding iron oxides. The as‐formed composite tribofilm with a thickness of about 20 nm, consisting of phosphate and iron oxides as the binder as well as deposited ZnO nanoparticle as the filling phase, is responsible for the excellent friction‐reducing and antiwear abilities of P204‐ZnO nano‐additive for the steel sliding contact.