Abstract The application of liquid lithium as a coolant for the forthcoming era of space nuclear reactors exhibits significant potential, and spiral tube heat exchanger components are commonly used in steam generators for reactors. However, the heat transfer characteristics of liquid lithium in spiral tubes are not yet fully understood. This research establishes a non-isothermal heat transfer model incorporating a modified turbulent Prandtl number to analyze the flow of liquid lithium through spiral tubes with varying geometries. Numerical analysis is carried out focusing on the influence of inlet velocity, the distribution of related parameters, and the geometry of spiral tubes. The results demonstrate that in the range of the dimensionless Dean number 8165–13,063, the Nusselt number and the pressure drop present approximately linear relations with the Dean number. For the distribution law of relevant physical quantities, the inner side of the tube displays an eye-anatomy low-flowrate area and a high-temperature area, while a low-pressure area forms on the inner pipe wall. Finally, the pitch and spiral radius are found to be reduced as much as possible to ensure high liquid lithium-based heat transfer performance with a small pressure drop. The optimized design parameters reveal that within the actual design range of non-dimensional pitch of 0.667–10.667 and curvature of 0.0556–0.1667, the non-dimensional pitch and curvature are 0.667–2.667 and 0.1667, respectively. This study offers valuable insights into the heat transfer properties of liquid lithium within heat exchangers of the spiral tube design, promoting its application in space nuclear reactor power supply.