During the operation of supercritical CO (sCO) turbomachineries, large pressure ratios, supersonic conditions and non-ideal gas fluid dynamic phenomenon may happen, which will decrease the whole cycle efficiencies. Hence non-standard designs for the turbomachineries geometry are needed. Successfully simulations in capturing non-ideal gas fluid dynamics and coupled with optimization algorithm are hard and currently not available in any commercial CFD software. Therefore, the problem of optimizing sCO turbomachinery is difficult to solve, which has become a major obstacle to obtain compact and high-efficiency sCO turbomachineries. In this study, a modularized geometry optimizer is developed to obtain the non-standard geometric designs for sCO turbomachineries. Multiple techniques are applied to this optimizer, include the Nelder–Mead algorithm, Mahalanobis distance and stochastic algorithm. The newly developed optimizer can successfully find the optimum satisfying the objective function under given weighting factors. The computational cost can be reduced through a stochastic algorithm. To validate the optimizer, a convergent–divergent nozzle for air with a target Mach number equal to 2.4 is optimized. Different starting points and combinations of weighting factors are used to create a Pareto front. Adjust the weighting factors for different terms of the objective function leading the optimizer to go to different directions in n-dimensional spaces. Three optimized cases, one is Mach number optimized, one is outlet flow uniformity optimized and the other is compromised case, are picked out and analyzed. The results show that the optimizer can successfully find optimized geometry than the reference case and potentially save computational cost. Due to the modularized characteristics, the components of this optimizer can be replaced with any available techniques, which mean the optimizer can be applied to solve different types of optimization problems.