In this study, a multi-parameter constrained optimization procedure integrating the design of experiments (DOE), full factorial experimental design (FFED), genetic algorithm (GA) and computational fluid dynamics (CFD) is proposed to design two-dimensional wavy channel with nanofluids (Cu/water, Al2O3/water and CuO/water). The elliptical, coupled, steady-state, two-dimensional governing partial differential equations for laminar forced convection of nanofluids are solved numerically using the finite volume approach. Some important parameters for the influences of heat transfer enhancement such as the Reynolds number (250≦Re≦1000), the particle volume concentration (0%≦ϕ≦5%), the wavy channel amplitude (0.1≦α≦0.3) and the wavy numbers (3≦β≦12) on the enhancement of nanofluid heat transfer have been investigated.The numerical results with a single-phase model are first validated with the available data in the literature. The maximum discrepancy is within 8%. Results of a further extension to a two phase model are also validated. The numerical results indicate that the thermal enhancement can achieve 15%, 24% in the wavy channel flow compared with pure fluid, with the particle volume concentration of ϕ=3% and ϕ=5% of Cu/water nanofluids. In addition, after the validation of the numerical results, the numerical optimization of this problem is also presented by using a full factorial experimental design and the genetic algorithm (GA) method. The objective function E which is defined as thermal performance factor has developed a correlation function with three design parameters. The predicting performance factor E (α=0.278, β=3, ϕ=5%) of regression function is closely agreed with those from the CFD computational results within 4.6% difference. The combination of parameters is considered as the optimal solution.