A controllable particle size mono-dispersing nanofluid system has been developed to address the challenges of low porosity and low-permeability in low to ultra-low-permeability reservoirs. This system combines high dispersion stability with enhanced oil recovery performance, and its effectiveness in improving recovery rates in low-permeability reservoirs, where conventional chemical flooding is ineffective, has been well demonstrated. Using the in situ method to prepare monodispersed nano-silica particles, the effects of the water concentration, ammonia concentration, and silica precursor concentration on the morphology, particle size, and formation time of the silica spherical particles were analyzed. Building on this foundation, a partially hydrophobic modified nano-silica oil displacement fluid was synthesized in situ. The system’s dispersion stability, ability to reduce oil-water interfacial tension, and capacity to alter rock wettability were evaluated. Core physical models were used to evaluate the oil displacement efficiency and the permeability applicability limits of the self-dispersing nano-silica oil displacement system. The experiments confirmed that the particle size distribution of the self-dispersing nano-silica oil displacement system can be controlled within a range of 10 nm to 300 nm. The nanofluids exhibited excellent stability, effectively altering the rock wettability from oil-wet to water-wet and reducing the oil-water interfacial tension to approximately 10−1 mN/m. The nano-displacement system increased the recovery rate of the low permeability reservoirs by more than 17%. The in situ modification method used to prepare these self-dispersing nanoparticles provides valuable insights for synergistic enhancement of recovery when combined with other systems, such as surfactants and CO2. This approach also opens up new possibilities and drives further development in the field of nano-enhanced oil recovery chemistry.