In this study, the analysis method combining multi-body dynamics (MBD) and finite element analysis (FEA) is used to study the transmission failure of the Zero-Backlash High Precision Roller Enveloping Reducer (ZHPRER). A dynamic model of the reducer is first constructed using the MBD method, and the correctness of the model is verified through experiment. The model is then applied to calculate the change in the contact force of the reducer during the meshing period under different speeds and loads. Finally, the maximum value of contact force in the meshing period of different working conditions is used as a boundary condition, and the weak position in the transmission process is analyzed by the FEA. At the same time, the validity of the FEA model is verified by an overload test. The results show that, within the rated load, the failure mode of the transmission is the fracture of the inner ring assembly shaft caused by fatigue damage or the fatigue deformation or cracking of the worm gear assembly hole. The change in speed and load significantly impacts its fatigue life. When the rated load is exceeded, the failure mode of the transmission is manifested by crushing of the outer ring, rolling bodies, and cracking of the worm gear assembly hole.