Abstract Nanoparticles have gained recognition for significantly improving convective heat transfer efficiency near boundary layer flows. The characteristics of both momentum and thermal boundary layers are significantly influenced by the Prandtl number, which holds a crucial role. In this vein, the current study conducted a detailed computational analysis of the mixed convection flow of $\gamma$Al$_2$O$_3$-H$_2$O and $\gamma$Al$_2$O$_3$-C$_2$H$_6$O$_2$ nanofluids over a stretching surface. This research integrates an effective Prandtl number, utilizing viscosity and thermal conductivity models based on empirical findings. Additionally, a unique double-fractional constitutive model is debuted to accurately evaluate the effective Prandtl number’s function in the boundary layer. The equations were solved using a numerical technique that combined the finite-difference method with the L$_1$ algorithm. This investigation presents numerical findings related to the velocity, temperature distributions, wall shear stress coefficient, and heat transfer coefficient, contrasting scenarios with and without the effective Prandtl number. The research shows that integrating nanoparticles into the base fluids reduces the temperature of the nanofluid with an effective Prandtl number while enhancing the heat transfer rate irrespective of its presence. Nonetheless, the introduction of a fractional parameter reduced the heat transfer efficiency within the system. Notably, the $\gamma$Al$_2$O$_3$-C$_2$H$_6$O$_2$ nanofluid demonstrates superior heat transfer enhancement capabilities compared to its $\gamma$Al$_2$O$_3$-H$_2$O counterpart but also exacerbates the drag coefficient more significantly. Many practical applications of this study include electronics cooling, industrial process heat exchangers, and rotating and stationary gas turbines in power plants, and efficient heat exchangers in aircraft.