In view of the issue that current gear dynamics model contains no parameters about tooth surface topography, this paper puts forward an improved nonlinear dynamic model of gear pair with tooth surface microscopic features through revision of the backlash equation by W–M function from fractal theory and combination with the tradition gear torsional model. The model sets up a mathematical relationship between gear dynamic characteristics and surface microscopic parameters such as surface roughness and fractal dimension. Results of the numerical simulations indicate that as surface roughness decreases, meshing stiffness increases and viscous damping rises, the gear dynamic performance tends to be better, which is consistent with the existing research reports. Furthermore, it is found that dropping of fractal dimension is good to improve gear dynamic performance, so gear dynamics can be enhanced by decreasing the fractal dimension if surface roughness is set or cannot be decreased anymore. Moreover, it is also shown that initial backlash has little impact on the rule of gear dynamics response but influences the size of start-up or stop shock. Finally, the model is validated by a series of simulations and comparison with experimental data and existing model. The theory here opens up a mathematical methodology to analyze gear dynamics with respect to tooth surface microscopic features, which lays a theoretical basis for design of tooth surface topography to obtain better performance of gear transmission in the future.