Abstract The variable-speed pumped storage unit (VSPSU) adopts flexible three-phase AC excitation, and the control dominant characteristics of the equipment are significant, which deeply affects the out-of-step (OOS) fault law and stability mechanism of VSPSU. Aiming at the OOS fault of VSPSU, this paper first defines the power angle of VSPSU and analyzes the influence of a phase-locked loop on the power angle stability of the unit. Then, the fault characteristics of the power angle instability of VSPSU are further studied. It is found that the traditional lens impedance protection scheme based on measuring the number of sliding poles has the problem that the phase-locked OOS oscillation cannot be responded to or the response is not timely. Combined with the research on the fault characteristics of power angle instability of VSPSU, an OOS protection method based on a bilateral annular blocker is proposed. The simulation results show that the proposed protection method can quickly and accurately reflect the phase-locked OOS fault of VSPSU, and meet the requirements of VSPSU for the reliability of OOS protection.