Suppressing rotor synchronous current is vital for achieving precise control in active magnetic bearing (AMB) rotor systems. The displacement amplitude inconsistency in orthogonal channels presents new challenges for synchronous current suppression. This paper proposes an improved complex least mean square (CLMS) algorithm with error compensation to effectively mitigate rotor unbalanced vibration. Firstly, a dynamic model of the AMB-rotor system with an unbalanced mass is established, and the displacement amplitude inconsistency within the orthogonal channels of the radial magnetic bearings is analyzed. Subsequently, the rotor displacement signal from the orthogonal channels is converted into a complex plane signal. By calculating the negative sequence weight component, the displacement signal is reconstructed, and an error compensation channel based on the CLMS is introduced to enable real-time tracking and compensation of displacement amplitude inconsistencies in the orthogonal channels. Moreover, system stability across the full speed range is ensured through the application of the phase compensation angle method. Finally, experimental validation on a magnetic levitation molecular pump prototype confirms the effectiveness of the proposed algorithm.