Since there are backlash friction and time-varying stiffness, the characteristics of gear pair has strong nonlinearity. When a gear has faults such as pitting, spalling and tooth breakage, there are often some nonlinear characteristics as super-harmonics or sub-harmonics along with vibration signal. These characteristics will cause some difficulty for fault diagnostics, so it is necessary to study the mechanism of nonlinear vibration. Fourier expression was used to describe time-varying stiffness, a factor was used to depict gear fault level and nonlinear dynamics model for gear fault level was established. Simulation results show that, with certain friction and rotating speed, the nonlinear vibration has direct relationship with rotating speed and gear friction and has no direct relationship with fault types.