To study the nonlinear dynamic behavior and system stability of a rubbing overhung rotor with viscoelastic and memory-effect damping and random uncertain parameters, this paper introduces a fractional-order modeling and stochastic dynamic analysis method for the nonlinear overhung rotor system with frictional impact faults. Firstly, the dynamic equations of the overhung rotor considering friction effect and fractional damping effect are established based on the transfer matrix method and fractional order derivative. Then, the time-domain response of the fractional-order dynamic equations is solved by combining the Runge–Kutta method and the continuous fractional expansion, and the steady-state response characteristics of different fractional damping are analyzed in the deterministic case. Finally, to analyze the response of the system under the effect of stochastic parameters, the sparse grid-based PCE metamodel of the system response is developed. Statistical moments, probability distributions, and sensitivity indices of the response of stochastic systems are revealed. The results of this paper provide a theoretical basis for efficient and accurate prediction of the stochastic response of nonlinear rubbing overhung rotor systems.