• NSGA-II is used to optimize the self-excited oscillation heat exchange tube. • The RSM models between Nu and f and structural parameters are obtained. • The friction factor of the optimized heat exchanger tube is reduced by 27.37%. • The velocity field and temperature field of the heat exchanger pipe are analyzed. It is a typical multi-objective optimization problem for self-excited oscillation heat exchange tube to enhance heat transfer by limiting pressure drop. In this study, the conflict between heat transfer and pressure drop is solved by multi-objective optimization method. The design parameters are three dimensionless structural parameters, which are collision wall angle (100° ≤ α ≤ 140°), the diameter ratio of chamber outlet to inlet (0.8 ≤ d 2 / d 1 ≤ 1.6) and the length-to-diameter ratio of chamber (0.4 ≤ L T / D T ≤ 0.6). The purpose of optimization is to compromise Nusselt number Nu and friction factor f . The sample points are obtained by CCD. The RSM models of Nu and f are constructed, and variance analysis and sensitivity analysis are carried out for the models. Finally, the Pareto front is obtained by NSGA-II and verified by CFD, the compromise solution is obtained by TOPSIS method. The results show that the most significant factor of Nu and f is d 2 / d 1 . The optimized structural parameters of the compromise solution are α = 125.67°, d 2 / d 1 = 1.2505, L T / D T = 0.4036, and the corresponding Nu and f are 149.8528 and 0.1144, respectively. Compared with the original structure, Nu increases by 1.54% and f decreases by 27.37%, which indicates that the convective heat transfer is enhanced and the flow resistance is significantly reduced.