Deciding a gear pair’s macrogeometry parameters requires consideration of the factors such as cost of production, gear strength, and noise contributing parameters. A practical engineering problem is encountered when best possible gear geometry is needed for a fixed center distance to achieve the conflicting objectives. Further, constraints of practical utility on input parameters need to be taken into consideration for deciding the design space. This work represents comprehensive compilation of formulations of these often-conflicting objectives and constraints as non-linear functions and selection of an appropriate optimization technique to achieve a feasible design through a case study of a helical gear pair in high load carrying capacity mesh. For comparison of the quality of results and computational efficiency between gradient based and gradient free algorithms, Sequential Quadratic Programming and Genetic algorithm were used. The differences in the formulations of the objective functions for two algorithms were analyzed and addressed. Quadratic mutation rates and random crossovers were used to prevent premature convergence of Genetic Algorithm. Significant possible improvement in the base design was observed with both the algorithms.