Thermal stratification is one of the major issues in rectangular regenerative cooling channels because of the asymmetric heating and limited turbulent diffusion of thermal energy of the fuel. As a result, the heat transfer performance is decreased. This paper uses Ansys fluent software to construct a novel cooling structure that combines wavy wall surfaces with cylindrical fins. In addition to this new configuration, this study also examined three other configurations (straight-smooth, cylindrical-ribs, and wavy-smooth) and compared them with each other. This structure aims to efficiently enhance hydrogen coolant flow and heat transfer characteristics while reducing the extent of the non-uniform distribution issue. The results show that the new structure’s Nusselt number and maximum wall temperature are enhanced by 46.89% and 22.71%, respectively, compared to the straight-smooth structure; in contrast, the friction factor increased only by 3.64% over the smooth channel. The novel design has the best thermal performance factor compared to the other three designs, which is improved by 43.33% over the smooth channel. Among all configurations, the new structure is the best fluid temperature distribution configuration, which shows an optimal distribution of the coolant temperatures between layers; thus, the stratification phenomena decreased significantly.