PurposeThe purpose of this paper is to improve the seal performance by proper design of the cavity shape of the damping holes, especially the rotordynamics characteristics of the hole-pattern damped seal (HPDS).Design/methodology/approachA new damping seal structure that comprises a circle-shaped cavity and two directional leaf-shaped cavities with a dovetail-shaped diversion groove is proposed. The comparative study on the sealing characteristics of dovetail-shape, leaf-shape and classical circular HPDSs was carried out using ANSYS CFX.FindingsThe dovetail-shaped HPDS significantly outperformed two other damping seal designs in leakage and rotordynamic performance. At a rotating speed of 7,500 rpm, it showed a 25% reduction in leakage, a 23% increase in average effective damping and a 119% increase in average effective stiffness. The cross-coupled stiffness Kxy shifted from positive to negative, reducing circumferential flow. The dovetail's inclined leaf-shaped grooves create a double vortex that slows jet velocity in the seal clearance and alters spiral flow direction, resulting in a uniform pressure distribution and enhanced rotor stability at low frequencies.Originality/valueThis study proposes a novel HPDS with dovetail-shaped diversion grooves. The seal can realize the simultaneous improvement of rotordynamics and leakage characteristics compared to the current seal structure.Peer reviewThe peer review history for this article is available at: https://publons.com/publon/10.1108/ILT-04-2024-0127/