This paper investigates optimization problem of the cutting parameters in milling on NAK80 mold steel with high hardness. In the study, by applying non-dominated sorting genetic algorithm-II, multiple sets of cutting parameters conformed to the requirements are acquired through solving the models, underlied by the design of orthogonal experiment with respect to cutting speed, feed rate and cutting depth, and the statistical model of cutting force and surface roughness by regression method, and the theoretical formula of material cutting rate. The proposed model and algorithm works well in the actual machining process as the experimental results show.