The torsional vibration problems often occur to industrial robots during start and stop due to the use of flexible wheel in harmonic gear reducer as a transmission pair. Because of the influence of nonlinear stiffness, transmission error, and other factors, the nonlinear torsional vibration of harmonic gear reducer will affect the stability and reliability of transmission system. In this article, based on the structural characteristics of the harmonic gear reducer, considering the nonlinear torsional stiffness, transmission error, meshing damping, and other factors, the nonlinear torsional dynamic model of the harmonic gear reducer was established with the theoretical analysis and the experimental results. Based on this model, the influence of various factors such as rotational speed, moment of inertia, torsional stiffness, and transmission error on the nonlinear torsional vibration of harmonic gear reducer was discussed. The results show that the vibration amplitude of harmonic gear reducer increases with the increase of speed, transmission error, and motor inertia, and decreases with the increase of load inertia and damping. The dynamic model and analysis method established in the study can provide the theoretical guidance for the optimal design and the vibration reduction of harmonic gear reducer.