This article reports the optimization of pin fin shape using a genetic algorithm (GA) coupled either to a machine learning (ML) model or a computational fluid dynamics (CFD) model. The ML model evaluates the temperature and pressure induced by the fins within a second and allows us to replace the time-consuming CFD simulations during the design stage. The optimization is conducted for a cooling channel with a uniform heat flux boundary condition (5 W/cm2) in the Reynolds numbers range of 3000 – 12000. The optimization identifies a funnel-shaped fin that enhances the heat transfer coefficient by 20% without an apparent increase of pressure drop as compared to the standard cylindrical pin fins. The funnel-shaped fin outperforms other conventional fins of elliptical, cubic, and drop shapes that induce a similar level of pressure drops. This work demonstrates the potential of ML-based optimization in searching unexplored shapes of heat transfer systems with superior performance.