This paper discusses the conjoint effect of circular fin turbulators and alumina (Al2O3) nanofluid on the thermal performance of a helically wound coil-in-shell heat exchanger used for waste heat recovery. An experiment was conducted using three different coils: two with circular fin turbulators brazed to the annular coil surface at 45° and 90° orientations and one plain coil without a fin. The coil side fluid was Al2O3 nanofluid (0.05%, 0.1%, and 0.15% by volume concentrations) flowing in the coil under laminar and turbulent (500 ≤ Re ≤ 5500) conditions, while the hot air at different flow velocities from 1 to 3 m/s (i.e., 4500 ≤ Reh ≤ 13,500) was used as the shell-side fluid. The experiment revealed that the Nusselt number increased by 43.96% and 69.66% and the thermal-hydraulic performance increased by 16.92% and 35.71% for the 90° and 45° fin orientations, respectively, at 0.15% concentration of Al2O3 nanofluid and hot air velocity of 3 m/s. Furthermore, the thermal-hydraulic performance showed a nearly two-fold increase with 45° fin orientation/Al2O3 nanofluid combination. Based on the experiment, correlations were developed for the Nusselt number, friction factor, and thermal-hydraulic performance in the laminar and turbulent regimes, which are fairly in agreement with acceptable limits.