Based on the Rayleigh–Ritz method, this article proposes an indicator to quantify the stiffness of a conical cutter for flank milling of impellers. Its validity is verified by the finite element analysis. A mathematical model is then developed to optimize the geometry of the conical cutter. The objective is to improve the stiffness of the cutter. Three kinds of geometric constraints are considered. First, the ball end of the cutter should be tangential to the hub surface. Second, the cutter should be interference-free with the adjacent blade. Finally, the machining error should satisfy the precision requirements. All these geometric constraints are characterized by the signed point-to-surface distance function. Based on the differential property of the distance function, a sequential linear programming method is applied to obtain the optimal geometry of the cutter along with the tool path. Simulation results confirm the effectiveness of the proposed model and algorithm.