Abstract Background Steam generator (SG) is one of the significant components in the nuclear steam supply system. A variety of SGs have been designed and used in nuclear reactor systems. Every SG has advantages and disadvantages. A brief account of some of the existing SG designs is presented in this study. A high surface to volume ratio of a SG is required in small modular reactors to occupy the least space. In this paper, performance improvement for SGs of integral small modular reactor is proposed. Aims/Methods For this purpose, cross-grooved microfins have been incorporated on the inner surface of the helical tube to enhance heat transfer. The primary objective of this work is to investigate thermal–hydraulic behavior of the proposed improvements through modeling in RELAP5-3D. Results and Conclusions The results are compared with helical-coiled SGs being used in IRIS (International Reactor Innovative and Secure). The results show that the tube length reduces up to 11.56% keeping thermal and hydraulic conditions fixed. In the case of fixed size, the steam outlet temperature increases from 590.1 K to 597.0 K and the capability of power transfer from primary to secondary also increases. However, these advantages are associated with some extra pressure drop, which has to be compensated.