A computational fluid dynamics (CFD) analysis is conducted to investigate the thermal performance of unsteady free convection of TiO2/water nanofluids. The study process within a coaxial heat exchanger system. The model is based on the mixture Buongiorno approach. Due to the axisymmetric conditions, the study is carried out in two dimensions, and the conservation equations are reformulated in dimensionless form. The obtained PDE system is Discretized using the finite element method. The TiO2 concentration heterogeneity, resulting from thermophoresis and Brownian motion is considered. The presentation of the results encompassed isotherms, nanoparticle volume fraction distributions, and streamlines. The validation of the model shows a good agreement with the experimental results. The impact of the mass fraction’s nanoparticles ( 0.01 ≤ ϕ ≤ 0.04 ) and the Rayleigh number ( 10 3 ≤ Ra ≤ 10 8 ) on the heat transfer performance are analyzed. The parametric study shows that an increase in the Rayleigh number has no effect on heat transfer enhancement. Unlike for Ra = 10 6 where an increase in Nusselt number is observed. It was found also that the effect of nanofluid concentration on the Nusselt number was more noticeable in high Rayleigh cases due to the convection-driven heat transfer.