The internal friction of floating spline can cause self-excited vibration of a supercritical flexible rotor system. To address this issue, a high-efficiency dynamic modeling method is proposed to investigate the self-excited vibration behavior and instability evolution of the rotor. Experiments are conducted to validate the theoretical results. The coupled dynamic equations for the rotor system connected with the floating spline are derived through the combination of finite element method and lumped parameter model. A hybrid numerical approach of precise integration and Runge-Kutta method is adopted to examine the effects of the friction coefficient of spline’s tooth surface, torque, and eccentricity on the self-excited vibration of the rotor system. The results show that the spline friction leads to negative damping and inputs energy into the rotor system under supercritical conditions, triggering self-excited vibration when the input energy exceeds a specific level. With the same parameters, the experimentally obtained axial trajectory and primary frequency components are consistent with the theoretical results, verifying the accuracy of the proposed theoretical model. This study can serve as a useful theoretical guide for the dynamic stability design of flexible rotor systems with the floating spline.