An epoxy-based silane coupling agent (KH560) was grafted onto the surface of oleic acid-modified cerium oxide (CeO2-OA) nanoparticles in order to improve the competitive adsorption ability of the anti-wear additives in ester oils and simultaneously hinder the additive desorption owing to thermal disturbance under high-temperature condition. The as-prepared oleic acid-epoxy silane co-modified cerium oxide (CeO2-OA/E) nanoparticles were characterized. The effect of temperature on the adsorption behavior of trityl phosphate (TCP, a commercial high-temperature antifriction agent), CeO2-OA and CeO2-OA/E in PETO was studied using a dissipative quartz microbalance. Their tribological properties as the additives in pentaerythritol oleate (PETO), a polar ester base oil, were evaluated with a four-ball machine and a ball-on-block friction and wear tester; and their tribomechanism was explored with respect to their adsorption behavior on rubbed steel surfaces at elevated temperatures. It was found that the secondary surface-capping of CeO2-OA by the KH560 silane coupling agent resulted in great increases in the surface potential (from 54mV to 396mV) and thermal stability as well (the thermal decomposition temperature rose from 185°C to 254°C). Among the tesetd lubricant additives, CeO2-OA/E exhibited the highest adsorption mass, because of the highest surface potential the chemisorption ascribed to the epoxy group of the silane coupling agent. Particularly, CeO2-OA/E added in PETO exhibited better friction reduction and anti-wear properties at 150°C than CeO2-OA and TCP, because CeO2-OA/E added in PETO formed tribofilm composed of CeO2 and SiO2 with excellent thermal stability as well as friction-reduction and antiwear effects through stable chemical adsorption and tribochemical reaction at elevated temperatures.