The packing unit is the key part of a rotating blowout preventer (RBOP) and its sealing performance affects the operation of under-balanced drilling, but sealing failure often happens. This paper studies the influence of various parameters on its sealing performance and discusses the packing unit's failure mechanism. Firstly, the Yeoh constitutive model is chosen through a group of tensile tests of the packing unit material. Then, the finite element model of the packing unit is established, and the influence of well pressure, tripping and drilling speed, friction coefficient and five structural parameters on the contact stress between the packing unit and drill pipe are discussed. Finally, the failure mechanism of the packing unit is analyzed, and the structural parameters of the packing unit are optimized. The results show that during the tripping process, the contact stress is the largest when the drill pipe joint passes through the packing unit, and the contact stress of the packing unit increases with the increase of well pressure and friction coefficient. The tearing at the top of the packing unit is caused by shear stress. The failure mechanism of the packing unit at the inner angle is divided into two types, one is that excessive contact stress causes the tongue piece to fall off, the other is that alternating von Mises stress and shear stress cause lumpy to fall off. Alternating von Mises stress and CSHEAR 2 (Transverse friction shear stress) causes tearing at the inner lower angle. The drill pipe eccentricity causes the Y-shaped cracks at the side of the packing unit. The optimized parameters of the packing unit are as follows: The tripping and drilling speed is 0.15 m/s, the inner diameter of the primary sealing section is 107 mm, the outer angle is 154°, the inner angle is 150°, the inner lower angle is 135°, and the length of the primary sealing section is 107 mm. In this paper, the failure mechanism and various influencing factors of the packing unit were analyzed, and the reliability of the finite element model was verified, which has reference value for failure analysis and structural design of the packing unit.