Model-based vibration signal analysis to interpret and understand vibration characteristics can provide prior guidance for the condition monitoring and fault diagnosis of planetary gearboxes (PGs). Transmission path effects due to the rotational motion of planet gears with respect to the accelerometer mounted on the PG housing make understanding its real dynamic behavior challenging. To overcome this challenge, a novel mathematical model of the transmission path effect is proposed to support the construction of more realistic dynamic response of PGs. This model is derived only from the geometric parameters and material properties, and it has higher applicability than traditional models. Subsequently, a phenomenological vibration model is used to simulate all vibration sources, including sun–planet and ring–planet gear meshes. On these bases, a comprehensive vibration signal model is proposed by considering all vibration sources and the corresponding transmission path effects. Moreover, the theoretical spectral structure is deduced via Fourier series analysis. Finally, the vibration characteristics of the vibration signals obtained using the proposed mathematical model are analyzed and validated through simulations and experiments in the time and frequency domains.