Nanoparticles are presently considered to be high-potential filler materials in insulation medium. Surface-modified nanoparticles can not only improve the compatibility between nanoparticles and insulation liquid but also enhance the dielectric properties. Herein, models of soybean oil-based natural ester insulation oil doped with two silane coupling agents (A151, KH550) modified SiO2 were established, respectively. Molecular dynamics and DFT first principles were applied to explore the effect of two nanoparticles on the dielectric properties of liquid-insulation medium. The results showed that two types of nanoparticles can restrict the diffusion of water molecules by forming hydrogen bonds, while the number of hydrogen bonds in KH550–SiO2 is one-third more than that in A151-SiO2, and the interaction energy is lower, demonstrating a stronger adsorption effect on water molecules. Moreover, surface electrostatic potential and frontier molecular orbitals showed that KH550–SiO2 have a higher adsorption on charged particles in natural ester insulation oil, which can further inhibit the development of streamer discharge in the liquid insulation medium, improve its dielectric property and delay the aging and deterioration of natural ester insulation oil. This study extends our knowledge into the influence of silane coupling agents modified SiO2 nanoparticles on the dielectric performance of liquid insulation materials from a microscopic perspective.