Dynamic mesh forces are the primary vibration excitations of the planetary gear transmission systems. However, they are difficult to measure directly, and conventional indirect measurement methods are typically only applicable to specific planetary gear sets with special measuring equipment. This study proposes a more widely applicable method for identifying the mesh forces of ring-planet gear pairs by utilizing vibration signals. To this end, a ring gear dynamic model is established based on the elastic theory of the thick ring in the form of a space state system. The state vector of the model is constructed from parameters of ring gear vibration responses, while the input state vector is derived from parameters of dynamic mesh forces. A singular value decomposition (SVD)-based Kalman filter is employed to provide a stable solution for the estimation of state vector and input vector simultaneously. A mesh force indirect measurement method based on tooth fillet strain signals collected by strain gauges is adopted to evaluate the accuracy of identification results. Results of dynamic experiments conducted in a planetary gear set test rig demonstrate that compared with measured results, the proposed method can obtain acceptable estimated mesh forces of ring-planet gear pairs by using vibration signals.