Health services of rotor-bearing systems are directly related to safe and stable works of rotating machineries in the industrial production. To address the issues of a low impact on vibration characteristics caused by faults in the higher rotational speed region, narrow and weak excitation signals, high dimensionality related to fault induction mechanisms, and difficulty in diagnosing using artificial intelligence technology due to the small number of fault samples in rotor systems, a novel fault detection method of nonlinear rotor systems through the incremental two-dimensional principal component analysis (I2DPCA) of the machine learning on the vibration energy space is proposed. Unlike mainly analyzing dynamic characteristics of rotor-bearing systems in vibration space, the method applies signals of the vibration energy space such as energy tracks, energy-time histories and energy-frequency spectra to have more advantages in vibration amplitudes caused by cracks in the higher rotational speed region and to overcome narrow and weak signals in the early fault stage. Then, the I2DPCA is applied to extract online low dimensional fault features and to process small vibration features of the rotor system. Simulation and experiment results show that performances of between-class margins and within-class clusters of different health conditions on the vibration energy space are more perfect than that on the vibration space. Based on the I2DPCA algorithm and the energy space method, the recognition rate of crack faults within the high rotational speed region is up to 99.6%, and it has good convergence and online learning ability in the case of small samples. The achieved conclusions of the method could provide a novel reference direction for fault diagnoses of rotor systems.