In order to facilitate lubrication and avoid the gear stuck due to thermal expansion, there needs to be a gap between the tooth profiles. As a strong nonlinear factor, the backlash will affect the motion state of the planetary gear system. When the gear failures occur, the motion state of the system will accordingly change. In this study, the meshing stiffness of the gear pair with tooth tip chipping fault is calculated by combining the analytic geometry method and the potential energy method. Then, a new nonlinear dynamic model including tooth backlash, time-varying mesh stiffness, and manufacturing error is established to study the dynamic response of the system. The equations of motion are derived by the Lagrangian method and solved by the numerical integration method. Taking the excitation frequency and tooth backlash as the variation parameters, respectively, the dynamic characteristics of the system are analyzed by comparing the global bifurcation diagrams between the health system and the fault system, and the path of the system into chaos is revealed. At the same time, the local characteristics of the system are revealed through the phase diagrams and Poincaré maps. The results show that with the variation of excitation frequency and tooth backlash, the fault system presents a more complex motion state. This study can provide the theoretical support for dynamic design and fault diagnosis of planetary gear transmission systems under the environment of gear fault-prone.