In this paper, the nonlinear dynamic characteristics and stability of an energy storage flywheel rotor with shape memory alloys (SMA) damper are studied. A new type of SMA constitutive model is proposed to express SMA's hysteresis properties, and the nonlinear dynamic model of an energy storage flywheel rotor with SMA damper is established. A developed multi-scale method is proposed to obtain the natural frequency of the system with high accuracy in strong nonlinear condition, and the nonlinear dynamic characteristics of the system are analyzed. To a cracked flywheel rotor, a time-varying function is proposed to express the effect of the crack depth on the system's stiffness, and the variation of the system's area with different parameters is discussed. Simulation results show that the nonlinearity and crack will both affect the stable area of the system. To a flywheel rotor subjected to stochastic excitation, a developed analysis method for the stability is proposed to obtain the global stability of the system. Finally, the conditions on stochastic bifurcation are determined. Simulation results show that the stochastic system's motion will change from balance point to periodic orbit, and will also jump from one orbit to another in special cases. The results of this paper are helpful for the design of energy storage flywheel rotor.