In the traditional gear dynamic model, the mesh stiffness is obtained under static load. When the mesh stiffness changes greatly with the change of load, the traditional dynamic model is not accurate enough to analyze the gear system. Therefore, this paper proposes a new dynamic model with a more realistic dynamic mesh force based on gear compatibility conditions. The proposed dynamic mesh force model is applicable to the gear system with and without tooth modification and verified by the finite element method. Afterward, the paper compares the differences in the dynamic responses between the proposed model and the traditional model. The results indicate the dynamic responses of the proposed model are generally consistent with those of the traditional model for the standard involute gear system. For the lead crown gear system, compared with the proposed model, the traditional model will lose important nonlinear characteristics in the high-speed area. At most speeds, the amplitudes of vibration and dynamic mesh force of the proposed model are greater than that of the traditional model, especially in the bifurcation, quasi-periodic and chaotic regions.