Abstract Annular gas seals used in turbomachinery are key elements to ensure a high machine efficiency, due to their effect on the leakage flow. As the most common type of annular gas seals, labyrinth seal is widely used as end and center balance drum, interstage, shaft end, and impeller eye seals. However, labyrinth seals have been confirmed to be a major source of destabilizing forces resulting in rotordynamic instability problems. The destabilizing cross-coupling forces, the major mechanism of inducing an instability vibration, are related to swirl velocity in labyrinth seal cavity. Installing swirl brake devices at the seal entrance is efficient to improve the seal stability by suppressing the inlet preswirl velocity. The swirl velocity downstream inlet swirl brakes can be reduced obviously, and it is usually less than the linear velocity of the rotor surface; then, the swirl velocity in seal cavity increases from inlet to outlet as a result of the viscous effect of rotating surface. In this article, to improve the rotordynamic performance of the labyrinth seal, a novel labyrinth seal (design 3), which possesses inlet swirl brakes at the seal entrance and swirl-reversal brakes in the fourth cavity, was designed. The effect of the swirl-reversal brakes on the seal rotordynamic characteristics was numerically investigated at two preswirl velocities and two rotational speeds. A detailed comparison was conducted for the rotordynamic characteristics of this novel labyrinth seal with those of a no-brake configuration (design 1) and an inlet swirl brakes configuration (design 2). The transient computational fluid dynamics (CFD) method based on the multifrequency elliptical whirling orbit model was used to predict frequency-dependent rotordynamic coefficients for these three seal configurations. The accuracy and availability of the present transient numerical method were demonstrated based on the experimental data. In general, for all three seal designs, the rotational speed and the inlet preswirl velocity have a significant effect on the cross-coupling stiffness and effective damping, but a weak influence on the direct stiffness and direct damping, where the effect of the inlet preswirl velocity is more remarkable. For the low inlet preswirl velocity w = 10 m/s case, compared to the original seal design 2, the magnitude of the negative cross-coupling stiffness shows an obvious increase (by ∼314% and ∼218% for the n = 6 krpm and n = 9 krpm case, respectively) for design 3. Compared to the seal design 1 with no swirl brakes, the effective damping increases by ∼33% for design 2 and by ∼130% for design 3 at the rotational speed of 6 krpm. At the high inlet preswirl velocity of w = 106 m/s, the labyrinth seals with no swirl brake (design 1) and with only inlet swirl brake (design 2) both produce the destabilizing positive cross-coupling stiffness and a crossover frequency (175 Hz for the seal design 1, 28 Hz for the seal design 2) below which the effective damping is negative for the n = 9 krpm case; however, design 3 still possesses the stabilizing negative cross-coupling stiffness (below 240 Hz) and the positive effective damping throughout whole frequency range for the n = 9 krpm case. The present novel labyrinth seal (design 3) produces the largest effective damping, followed by the seal design 2 and then the seal Design 1, for all rotational speed and inlet preswirl cases. Therefore, the novel labyrinth seal (design 3) possesses better rotordynamic characteristics than other two seals.