The application of machine learning based on neural networks (NNs) and genetic algorithm (GA) in multi-objective optimization of heat exchangers is studied. Taking the tube fin heat exchanger (TFHE) as the research object, the inlet air velocity and the ellipticity of tubes are taken as the optimization variables. In order to obtain the optimal heat transfer performance and pressure drop performance, Computational Fluid Dynamics (CFD) simulation is carried out for different Reynolds based on the hydraulic diameter numbers (150–750) and tube ellipticity (0.2–1). Then use simulation data to train the Back-Propagation neural networks and establish the prediction model of heat transfer coefficient and pressure drop. The non-dominated multi-objective genetic algorithm with elitist retention strategy (NSGA-II) is used to optimize two prediction results of NNs. Finally, the optimal heat transfer coefficient and pressure drop are given in the form of Pareto front. The optimization results show that when the Reynolds number is 541 and the ellipticity is 0.34, the pressure drop of the TFHE decreases 20%, and the heat transfer coefficient is basically unchanged, whose j/f is 1.28 times as much as that of the original heat exchanger.