Abstract Controlling the vibration levels of turbopump rotor shafts is a key feature for the reliability of space engines. Turbopump components are exposed to high static and dynamic stress levels, and therefore are particularly sensitive to high-cycle fatigues. Among the numerous dynamic excitations that affect the turbopump, self-induced instabilities are the most critical ones because of the exponential growth rate of vibration levels. These flutter-like phenomena may account for rotor shaft instabilities of turbopumps designed with an axial balancing system (ABS). Such a system is necessary to avoid heavy static loads on bearings and is commonly used in high power turbopumps in the space industry. It consists of a fluid cavity located in the back of a centrifugal compressor. Instabilities induced by the ABS have been studied within the framework of a research and technology program using a reduced scale hydrogen turbopump demonstrator called TPtech. This paper focuses on experimental and numerical analysis of rotor instabilities induced by the ABS. A coupled dynamic model of the rotor shaft and the ABS cavity is presented. It shows instabilities of the rigid rotor axial mode, but also of an axisymmetric rotor mode. This result is consistent with the data acquired during TPtech test campaign. The instability mechanism is complex as it involves the rotor modes, the flow in the ABS cavity, and its acoustic modes. Therefore, TPtech tests performed in representative conditions are valuable. They have permitted tool validation and have provided design rules to prevent occurrence of such phenomena.