Researchers have extensively studied convective heat transfer of variable property fluids in tubes because of their different heat transfer characteristics to constant property fluids. However, the heat transfer correlations are often limited to small or large tubes or other limited conditions when the properties of fluids change dramatically, which may be due to inaccuracy definition of the reference temperature in average heat transfer coefficient (HTC) and incorrect understanding of dimensionless parameters in Nusselt correlation.As a typical case, this study numerically investigates the cooling heat transfer of sCO2 in horizontal tubes near the pseudo-critical point. Eight numerical models are compared with experimental results in the literature, and the Realizable k–ε model with enhanced wall treatment is best for predicting cooling HTC. A new definition of reference temperature for average HTC is proposed, which is more accurate than the traditional arithmetical average of inlet and outlet bulk temperature. Sensitivity analysis of HTC is carried out for heat transfer mechanisms. In addition, the definitions of dimensionless numbers in Nusselt correlation are analyzed, and more accurate definitions are proposed. Based on the conclusions, a newly developed Nusselt correlation with concise form shows excellent results with a Root Mean Square (RMS) error of 2.9% for 6.22 and 9.40 mm tubes. It can still maintain high accuracy when it is extrapolated to 4.15 and 14 mm tubes. The numerically derived correlation agrees well with the experimental results for 2–6 mm tubes in literature and can also be extrapolated to constant property cases.