Rotor systems are widely used in industrial power generation and propulsion. Once the nonlinear contact stiffness and oil film force are taken into account, the dynamics and stability of rotor systems become quite complex, often accompanied by super-harmonic and chaotic motions. Furthermore, conventional methods face limitations in real-time detection of the bifurcations and complex nonlinear motions. This research investigates the bifurcations and stability induced by nonlinear factors in a rotor-bearing system from an energy perspective. Dynamic equations of a rotor-bearing system considering cubic term stiffness are established, the steady responses are obtained by the fourth-order Runge–Kutta method. The relationship between the bifurcations and energy transfers is analyzed numerically, the proposed stability criterion is validated by comparing the Lyapunov exponents. The bistable phenomenon of period-3[Formula: see text] ([Formula: see text]) motion is discussed in terms of numerical results and experiments which corresponds to the asymmetric jumps of the generalized energy. It is found that bifurcations and unstable motions of the nonlinear system can be captured accurately by detecting the energy transfers, the proposed generalized energy curve exhibits more detailed information than the conventional speed-up curve. These findings provide a new perspective on studying bifurcations and stability of rotating systems, which can be further applied in the condition monitoring, stability prediction as well as the design of nonlinear energy sink, representing significant progress in converting theory into engineering applications for rotor systems.