Abstract A common issue associated with gas bearing-rotor systems is the tendency to generate self-excited vibrations, leading to instability. To address this problem, the fluid-structure coupling model of an aerostatic bearing-rotor system is established in this paper. Then a hybrid method combining the finite difference method and direct integration method, is employed to solve the bearing lubrication equation and rotor motion equation simultaneously. Furthermore, based on the orbit of rotor center,frequency spectrum diagram, Poincaré map, waterfall diagram and bifurcation diagram, the effects of rotational speed, rotor mass, orifice diameter and nominal clearance on the nonlinear dynamic behaviors of the bearing-rotor system are investigated. The results indicate that the system exhibits rich nonlinear behaviors with increasing rotational speed and rotor mass, including the occurrence of typical half-speed whirl. However, the nonlinear vibrations of the system can be restricted by selecting appropriate bearing structural parameters, providing theoretical guidance for the design of aerostatic bearing-rotor systems.