As sophisticated mechanical equipment, the rotor system of aero-engine is assembled by various parts; bolted flange joints are one of the essential ways of joints. Aiming at the analysis of the nonlinear vibration characteristics of the rotor-bearing system with bolted flange joints, in this paper, a finite element modeling method for a rotor-bearing system with bolted flange joints is proposed, and an incremental harmonic balance method combined with arc length continuation is proposed to solve the dynamic solution of the rotor system. In order to solve the rotor system with rolling bearing nonlinearity, the alternating frequency/time-domain process of the rolling bearing element is deduced. Compared with the conventional harmonic balance method and the time-domain method, this method has the characteristics of fast convergence and high computational efficiency; solving the rotor system with nonlinear bearing force; overcome the shortcoming that the frequency–response curve of the system is too sharp to continue solving. By using this method, the influence of bearing clearance and stiffness on vibration characteristics of the rotor system with bolted flange joints is studied. The evolution law of the state of the rotor system with bolt flange is investigated through numerical simulation and experimental data. The results indicated that the modeling and solving method proposed in this paper could accurately solve the rotor-bearing system with bolted flange joints and analyze its vibration characteristics.