Augmentation of the heat transfer in a rectangular channel demands a prominent technique in order to keep the pressure drop penalty at the minimum. The present work focuses on the theme of utilizing the heat transfer enhancement technique using the small height roughness. The rectangular channel, with nanofluid flow inside the duct aids in further enhancing the thermal performance of the system. The effect of geometric parameters of the roughness on heat transfer is analyzed using computational method. The geometric parameters selected for analysis are the Relative rib-groove width ratio (WP/WRG) and the Ratio of rib-groove height (eRG/Dh) along with the Al2O3/H2O nanofluid. The numerical analysis of the heat exchanger tube is carried out for Reynolds number in the range of 4000-21000. The highest increment in the Nusselt number (Nu) is found to be 276 for parameter values of WP/WRG of 3 and eRG/Dh of 0.133 along with the 4 vol% and nanoparticle diameter of 20nm, which fetches the friction factor of 0.113.