To enhance the heat transfer capability, the helical coil once-through steam generator (HCOTSG) has been widely utilized in the small modular reactor (SMR). There are hundreds of parallel helical channels in the HCOTSG, and the coolant undergoes the phase change from subcooled liquid to superheated vapor in the helical tube. The flow oscillation between parallel channels may be triggered during the phase change. In present study, the two-phase flow instability is investigated with different heat flux profiles under many operating conditions. The thermal–hydraulic model for parallel channels of HCOTSG is established based on our previous work, while the local heat transfer phenomenon such as subcooled boiling, critical heat flux, and wall conduction are considered in detail. The heat flux profile along the tube is ascertained by the steady-state analysis of HCOTSG under different operating conditions and applied to the outside surface of the helical channel. The heat flux value is increased step by step to trigger the out-of-phase oscillation between parallel helical channels. The thermal–hydraulic model and analysis method are validated against experimental results. The characteristics of flow instability in the helical channel are also analyzed. It is observed that the delay effect causing the flow instability is more obvious if the void fraction exceeds 0.7 in the starting stage. Then the boundary of flow instability is compared under different heat flux profiles for various operating conditions of HCOTSG. The results suggest that the distribution of heat flux in the two-phase region plays a more significant role in the occurrence of flow instability in the parallel helical channel system. The thermal–hydraulic characteristics during flow instability are governed by the combined effects of void propagation and delayed feedback in the system. In conclusion, the thermal–hydraulic condition of HCOTSG undergoes the unstable region to achieve the normal operating point, which can be avoided by adjusting the operating parameters.