Abstract In order to address the impact of reduced transmission stability and reliability caused by volume reduction on the quality of gear transmission, this article proposes a multi-objective optimization model for nonlinear dynamic load transmission errors of double helical gears. This study aims to introduce a multi-objective design method for gear transmission, using the volume and smooth reliability of helical gears as objective functions, and establish a multi-objective optimization design mathematical model for helical cylindrical gear transmission. In order to solve this multi-objective optimization problem, we utilized the optimization toolbox in the scientific calculation software MATLAB with examples. The results show that after the joint optimization design of volume and coincidence degree, it is calculated that the volume after the joint optimization design is still 2.2624 × 107 mm3, and the coincidence degree is 5.9908. After rounding, the design result is m n = 3 , Z 1 = 31 , β = 20 ∘ , Ψ d = 1.2 {m}_{{\rm{n}}}=3,{Z}_{1}=31,\beta ={20}^{\circ },{\Psi }_{{\rm{d}}}=1.2 . The optimization design results show that the joint optimization design with the minimum volume and the maximum coincidence as the objective function can reduce the volume and improve the output stability of the helical gear.