AbstractEnhancement in heat transfer of the cold side is vital to amplify the performance of a thermoelectric generator (TEG). With enriched thermophysical properties of nanofluids, significant improvement in heat transfer process can be obtained. The current study concerns the performance comparison of an automobile waste heat recovery system with EG‐water (EG‐W) mixture, ZnO, and SiO2 nanofluid as coolants for the TEG system. The effects on performance parameters, that is, circuit voltage, conversion efficiency, and output power with exhaust inlet temperature, the total area of TEG, Reynolds number, and particle concentration of nanofluids for the TEG system have been investigated. A detailed performance analysis revealed an increase in voltage, power output, and conversion efficiency of the TEG system with SiO 2 nanofluid, followed by ZnO and EG‐W coolants. The electric power and conversion efficiency for SiO 2 nanofluid at an exhaust inlet temperature of 500K were enhanced by 11.80% and 11.39% respectively, in comparison with EG‐W coolants. Moreover, the model speculates that an optimal total area of TEGs exists for the maximum power output of the system. With SiO 2 nanofluid as a coolant, the total area of TEGs can be diminished by up to 34% as compared with EG‐W, which brings significant convenience for the placement of TEGs and reduces the cost of the TEG system.