Noncircular face gear drive is a new type of variable transmission ratio mechanism with advantages of light weight, high interchangeability, and convenient installation. Based on its transmission principle, the excitation pattern of the time-varying instantaneous center of the noncircular face gear drive system is investigated. A torsional–lateral–axial coupled dynamic model of the noncircular face gear system is presented under the compound parametric excitations of the time-varying instantaneous center and the mesh stiffness. By using Runge–Kutta numerical integration method, the dynamic responses of the system are derived to analyze the vibration features of the noncircular face gear under compound parametric excitations and the effect of the design parameters on the coupled vibration. The analytical results indicate that there exist multifrequency components in the dynamic responses of the system, including the instantaneous center frequency f1, the meshing one f2, nf1, mf2, and mf2 ± nf1. In addition, the dynamic performance of the noncircular face gear drive can be improved by decreasing the amplitude coefficient of the instantaneous center and the mesh stiffness or increasing their mean values.