Chemical enhanced oil recovery often involves the injection of chemicals, such as surfactants, polymers, nanoparticles, and macromolecules. These chemicals are to alter the rock-oil-water interactions and mobilize the oil towards the production well. However, they often pose considerable challenges in the harsh environment of the oil-wet reservoirs, such as chemical instability, and high retention/adsorption of chemicals in the near-wellbore. Herein, we investigated the effect of functionalized silica nanofluids on improving oil recovery in the carbonate reservoir as a function of the oil acidity. To do so, we synthesized two types of silica nanofluids functionalized with sulfonate and carboxylate groups. We used two crude oils with different total acid numbers (TAN) of 0.4 and 5.7 mg/KOH. All experiments were ran at 100 °C and salinity of 67–240 kppm to simulate the harsh reservoir conditions of the Saudi Arabia. Spontaneous imbibition oil recovery studies were conducted to assess their effects on oil recovery. Contact angle and interfacial tension were performed at different conditions to examine the oil recovery mechanism. Further, in order to corroborate the previous research and comprehend the principles underlying the oil recovery, we also examined the electrokinetic zeta potential in parallel. Also, to better understand how oil and nanofluid functional groups interact, we conducted a surface complexation model study. In addition, molecular simulation (density functional theory and molecular dynamics) are employed to gain atomistic insights on the implicated molecular interactions between nanosilica particles with calcite surface in seawater brine. DFT simulations predicted that both functionalized nanosilica can form thermodynamically stable complexes with a dry and hydrated calcite surface. Further, MD simulations reveal that a good amount of both functionalized nanosilica are interacting with calcite surface in the diffuse electric double layer of the calcite-brine interface. However, the carboxylated functionalized nanosilica particles interact with rock stronger than sulfonated ones. Further, the carboxylated nanosilica could form stable complexes with naphthenic acids mediated by calcium ions, which supports the synergy effect observed experimentally. The results shed light on much more complex mechanisms behind the oil recovery rather than the common peripheral understanding. This work provides a good insight into the applicability of nanofluids for enhanced oil recovery for further research and field trials.