Optimization is now recognized as essential in the design of modern compact heat exchanger since the augmentation of heat transfer applying longitudinal vortex generator depends on different interacting parameters, such as its streamwise and spanwise locations, and attack and roll angles and shape. For this work, an optimization procedure based on the SIMPLEX method was conducted through a fluid–solid conjugated heat transfer modeling considering a fin-tube compact heat exchanger with two rows of tubes in staggered and inline arrangements, in order to find an optimum configuration of the vortex generators for two objective functions related to Colburn factor (j) and Friction factor (f). Two Reynolds numbers (250 and 650), performing as a function of fin-pitch, were evaluated. Moreover, a simple approach based on trigonometric decomposition to treat the constraints is presented, which allows great flexibility to vortex generators to cover the design space solution. Seven independent input parameters for each vortex generator in optimization procedure were considered, totaling fourteen independent variables. The results indicate that this work-optimized configuration for the vortex generators achieved higher heat transfer augmentation than those in previous works in the open literature, for both objective functions, Reynolds number and tube arrangements, and it is more pronounced for staggered tube arrangement than for inline tube arrangement. Moreover, suitable vortex generator shapes to maximize the objective functions are more similar to rectangular-winglet type than delta-winglet type, and the optimum ratio between the vortex generator height and fin-pitch is 0.6. Although several trends could be defined for the optimum points, optimized configurations of the vortex generators were found to be different for each Reynolds number, tube arrangement and objective functions, indicating strong asymmetry between the vortex generators to achieve higher heat transfer augmentation.