In this article, a finite line contact tribo‐dynamics model of helical gears is established by coupling the tooth surface friction, friction moment, and vibration velocity. To obtain convergent results, a cycle iteration strategy is adopted to solve the two submodels, the dynamic model and the thermal elastohydrodynamic lubrication model. The parameters obtained from the dynamic model such as the dynamic load, dynamic transmission error, and instantaneous speed of the teeth surfaces are used as the boundary conditions for solving the lubrication model. Meanwhile, the friction force and friction moment obtained by the lubrication model are used as parametric excitations of the dynamic model. Based on the proposed model, the influence of dynamic effect, thermal effect, rotational speed, and tooth surface roughness on the tribological and dynamic properties of helical gears are discussed. The results show that the tribological and dynamic parameters deviate significantly from the quasistatic values, especially at the characteristic speed. The vibration along the off‐line‐of‐action direction is obviously affected by the friction excitation. As the rotational speed increases, the friction coefficient and friction force decrease as well as the vibration of the gear pair along the off‐line‐of‐action direction weakens. In addition, the surface roughness has the greatest influence on the vibration of the off‐line‐of‐action direction in all directions of gear vibrations.