The dynamic characteristics of a flexible foundation would greatly affect the dynamic behavior of the rotor system. In this paper, a hybrid dynamic model method in time domain combines the finite element (FE) method and the subspace-based identification method to model a rotor system with the consideration of a flexible foundation and nonlinear support forces. The state-space model of flexible foundation is identified by the subspace state-space system identification algorithm with measured frequency response functions (FRFs). To solve the calculation problem of the hybrid dynamic model in time domain, a general algorithm named the hybrid model fusion solution (HMFS) method is proposed, which combines the linear and nonlinear nodes separation method used in the FE model and the trapezoidal rule used in the state-space model, improving the calculation efficiency of solving transient response with nonlinear forces. Simulations for the hybrid dynamic model under constant speed and speed-up cases were carried out, resulting that the dynamic characteristics of the entire system can be represented when the nonlinear factors of rolling bearings and gravity were also taken into consideration. Additionally, owe to the consideration of tested dynamic behaviors of flexible base, experimental results in the laboratory test rig are well agreed with the numerical ones under two different test configurations. The investigation provides a complete modeling and highly efficient solution framework for the rotor system with a flexible foundation and nonlinear rolling bearings, with the abilities of further consideration of more nonlinear components such as squeeze film dampers and rubbing forces.