In order to achieve more scientific design of the reduction gear and reduce material waste, pre-stressed modal analysis method was combined with multi-objective optimization algorithm to optimize the structure of the reduction gear basic body. The model was simplified and parameterized and the maximum stress and equivalent stiffness under different parameter size combinations were obtained through finite element analysis. Separately, genetic clustering method, neural network method, and Kriging method were used to construct the response surface function. Through error verification and comparison, it was found that the Kriging method was more suitable for the gear model. In the design of variable extremum search, multi-objective genetic algorithm and sequential quadratic programming were compared and analyzed. The results show that the mass of the gear can be reduced by 39.9 %, while the maximum stress remains unchanged, the equivalent stiffness is not reduced, and a good optimization design effect is achieved.