AbstractIn this study, computational fluid dynamic (CFD) simulation of a vertical axis wind turbine (VAWT) geometry based on the Unsteady Reynolds–Averaged Navier–Stokes equations was investigated. In addition, the relationship between the geometric parameters of the VAWT and the two response variables, that is, moment and lift force, was determined using response surface methodology (RSM). Then, the Non‐Dominated Sorting Genetic Algorithm (NSGA‐II) was used to solve the multi‐objective optimization problem. The results obtained from the RSM showed that the lift force of the turbine is more sensitive to the change in the blade chord length, and the output moment of the turbine is more sensitive to the change in the rotor radius. Using the NSGA‐II multi‐objective optimization algorithm and the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) method, it was determined that among the optimal values of the independent variable, the most optimal response occurs in blade chord length = 0.18 m, rotor radius = 0.4 m, blade pitch angle = −3.27° and number of blades = 4. In these optimal values of the independent variables, the values of the dependent variables, which included the turbine's moment and the blades’ lift force, were obtained as 9.58 N m and 57.89 N, respectively.