The synchronous meshing of the gear pair and the screw pair is a typical feature of the planetary roller screw mechanism. In order to fully derive and analyze the nonlinear dynamic characteristics of the system, this paper creatively incorporates the time-varying meshing stiffness of gear pair and the comprehensive transmission error into the research content. Combined with the thread contact force and friction force between the roller and the screw and between the roller and the nut, the nonlinear dynamic model of the planetary roller screw mechanism considering the meshing excitation of the gear pair is established. According to the time domain diagram, frequency domain diagram, phase plane diagram, Poincaré section diagram, three-dimensional spectrum diagram, and spatial phase diagram, the nonlinear behavior of the system is described in detail, and the bifurcation evolution process of the system under the excitation frequency parameters of the external load is revealed. In addition, based on the theory of multi-scale method and considering the variables such as meshing stiffness, meshing damping, and load fluctuation, the stability equation of the primary resonance of the system is derived. The analysis of the stability of the system under different working conditions shows that the parameters are selected within a reasonable range, which effectively reduces the primary common amplitude value and enhances the overall stability of the system. The research content improves the previous system dynamics modeling method and provides a theoretical basis for the nonlinear dynamics modeling method and parameter optimization design of the planetary roller screw mechanism.