The gear-bearing system is the most important part of integrally centrifugal compressors. According to statistics, the majority of integrally geared compressor accidents are caused by excessive vibration of the geared rotor. However, its complicated dynamic characteristics and inevitable vibration faults in actual operation present significant challenges throughout the analysis and design stages. In this paper, the coupled self-excited vibration of the gear system characterized by multipoint meshing and oil film bearing supporting is investigated. Firstly, the structure of the gear system in an integrally geared compressor is used as a research object. The modeling approach of meshing excitation, including time-varying mesh stiffness, gear meshing error, and tooth backlash are introduced. However, the variable stiffness and damping coefficient equations of journal bearing and oil film thrust bearing are modeled and utilized to approximate the variable bearing force and simplify the vibration computation under the assumption of Newtonian fluid. Then, a dimensionless modeling method of the gear system considering gyroscopic moment of gear disk, variable meshing force, as well as variable stiffness and damping coefficient is proposed. Based on the dynamic model, the influence of the bearing dynamic coefficients and load on the vibration of the entire gear system is studied. Among which, the vibration displacement and meshing force are examined using frequency-domain and time-domain analysis methods. The results suggest that the flexible support can restrain the system’s nonlinear motion, whilst increasing load on the gear system can improve gear operation stability and reduce load fluctuation.