The heat transfer characteristics of CO2 at supercritical pressures during cooling in two fluted tube-in-tube heat exchangers and a smooth tube-in-tube heat exchanger are investigated. Both the overall heat transfer and the local heat transfer coefficients are analyzed at various CO2 mass flow rates, inlet pressures, and flute pitches. The results demonstrate that the smaller hydraulic diameter of a fluted tube has a higher heat transfer coefficient at any pressure. The fluid is more disturbed in the fluted tube with a smaller flute pitch where the temperature gradient of CO2 near the wall is larger, resulting in greater Nusselt number. The overall heat transfer coefficients of the fluted tubes are 2–3 times that of the smooth tube. A new correlation is developed by considering the fluid properties and the fluted tube structure effects for the convective heat transfer calculation of supercritical CO2 in the fluted tube during cooling.