This paper aims to realize multi-objective optimization of cycloid-pin gears to improve the positioning accuracy and load-carrying capacity of the rotary vector (RV) reducer, via the consideration of backlash, transmission error, and torsional stiffness. Initially, the analytical models of the RV transmission backlash and transmission error are developed by using both purely geometrical and equivalent model methods individually. Based on the generalized Hooke’s law, a torsion angle model is established to characterize the torsional stiffness of the system, utilizing methods such as Hertzian contact theory and bearing stiffness models. Subsequently, employing the Monte Carlo method, extremum method, and quality loss function, mapping objective functions for dimensional accuracy (tolerance) and transmission performance (backlash, transmission error, and torsional stiffness) are constructed. The geometry dimensions, dimensional accuracy, and modification of the cycloid-pin gear are considered as design variables to create a multi-objective optimization model. The improved Parallel Adaptive Genetic Algorithm using Deferential Evolution (PAGA-DE) is used for multi-objective solutions. Through example calculations, the impact of cycloid-pin gear parameters on transmission performance before and after optimization is determined. The reliability of backlash after optimization within 1.5′ reaches 99.99%, showing an increase of 8.24%. The reliability of transmission error within 1′ reaches 98.52%, demonstrating an increase of 1.35%. The torsional angle is reduced by 8.9% before optimization. The results indicate that the proposed multi-objective optimization design method for cycloid-pin gears can achieve the goal of improving the transmission performance of the RV reducer.