High-power gears are widely used in various engineering fields. The vibration and noise produced in the process of gear transmission are a major problem that restricts the development of high-power gears. Particle damping technology has been proven to be effective in steady-state vibration suppression, but it is rarely applied to the centrifugal field due to the nonlinear dynamics. In this study, internal excitation is calculated through stiffness and error excitation generated by system stiffness and comprehensive error. In order to calculate the exciting force, a dumbbell-shaped damper was installed in the lightening hole. Based on the nonlinear energy dissipation mechanism, the damping force was used as the external excitation. And the damping effect of which is discussed in this paper. Using the dynamic coupling method of continuum and non-continuum, the equivalent displacement mapping of contact loads from the discontinuous domain to the continuous element node of the gear was realized and the transformation of the local coordinate to the global coordinate was carried out. The effect of particle diameter on the nonlinear dynamic characteristics of gears with different speeds and loads was explored and verified by the experiment. A diameter of 3 mm for the stainless-steel particles exhibited the best damping effect. Finally, an optimal allocation for high-power gear vibration reduction was sought. It was possible to reduce vibration by a maximum of 47% under a low load and a rotational speed of 1200 rpm.