Active control is one of the most important advantages of active magnetic bearing (AMB), and also can be used to suppress the imbalance of AMB rotor system (AMB-RS). This paper aims at dealing with periodic vibration problem caused by unbalanced force existing in AMB-RS as the rotor spins. Firstly, with the advantages of iterative learning control (ILC) over repetitive problems, a novel ILC algorithm based on the system information of the iteration before last iteration is put forward to achieve parallel computing of real-time control, real-time acquisition and signal extraction in AMB digital control system (DCS). Then, two compound control modes are proposed on the basis of AMB closed-loop feedback control system (CFCS), one is a parallel compound control mode (PCCM) that is used to achieve unbalance compensation, the other one is a serial compound control mode (SCCM) that is used to achieve automatic balance, and corresponding compound control systems are designed on the basis of an AMB experimental system. Finally, numerical simulation and experimental research are carried out, and the results show that the control methods proposed in this paper have significant control effect on the periodic vibrations produced by the unbalanced force of AMB-RS. Therefore, the research achievements can provide theoretical references and experimental basis for the further application of AMB in the high-precision and high-speed fields.