Abstract The influence of particle size and surface modifier structure on the adsorption behavior and tribological properties of surface-capped silica nano-additives in rapeseed oil was investigated. With four kinds of SiO2 nanoparticles, methyl-modified DNS-2 (particle size 14 nm) and DNS-3 (particle size 5 nm), vinyl-modified RNS-D (particle size 5 nm), and epoxy-modified RNS-E (particle size 5 nm) as the nano-additives of rapeseed oil, their adsorption behavior in rapeseed oil was studied with a quartz crystal microbalance (QCM-D), and their tribological properties were evaluated with a four-ball friction and wear tester. DNS-2 with a larger particle size can form a high-strength tribo-film on the rubbed surface through shear sintering, thereby reducing the wear scar diameter of the steel ball. DNS-3, RNS-D, and RNS-E with smaller particle sizes can form a viscoelastic adsorption layer on the rubbed steel surface, and their equilibrium adsorption mass thereon increases with the increase of the polarity of the modifiers, which corresponds to the increases in their friction-reducing and anti-wear abilities as well as extreme pressure properties therewith. Particularly, the equilibrium adsorption mass and adsorption rate of RNS-E surface-capped by an epoxy group with the largest polarity are 16 times and 34 times higher than those of DNS-3 surface-capped by methyl. This indicates that encapsulating silica nanoparticles with strongly polar organic species could help to enhance its adsorption on rubbed metal surfaces and improve its tribological properties in rapeseed oil.