High rotating speed milling operation is the most essential mean of implementing high-speed milling. To improve and enhance the dynamic performances of cutting system, a large number of creative spindle-tool systems with high damping performance are designed and applied, in which larger rotational damping (or internal damping) occurs. In this regard, since the system generally operates at supercritical speed, there are two different types of instability, namely rotational damping instability and chatter instability. In this paper, the effect of rotational damping on stability is proposed according to Routh-Hurwitz stability criterion, and comparisons between rotational damping instability and chatter instability are investigated in the subcritical and supercritical speed ranges. Some critical results and comments are analyzed and discussed. And three different types of experiments are performed to verify the numerical results.