Investigation of the polarization switching mechanism in ferroelectric hafnium zirconium oxide (HZO) film is of great importance for developing high-quality ferroelectric memory devices. Recently, several works have been reported to describe the switching process of polycrystalline HZO film using the inhomogeneous field mechanism (IFM) model. However, no report has recorded the effect of high pressure annealing (HPA) on the polarization switching process. In this paper, we have carried out a careful investigation on the switching properties of HZO capacitors annealed at 600 °C with various high pressure conditions (1, 50, and 200 atm) using the IFM model. As pressure increases to 200 atm, the ferroelectric properties were enhanced in the HZO films, and, as a result, highest remanent polarization (Pr of 24.5 μC/cm2) was observed when compared with 1 and 50 atm. Similarly, as HPA increases, the HZO capacitors showed a decrement of the coercive field, which significantly improved the switching properties. The time consumed for reversing 80% polarization was 113.1, 105.7, and 66.5 ns for the sample annealed at 1, 50, and 200 atm, respectively. From the IFM model, the smallest active field (2.997 MV/cm) and a uniform distribution of the local electric field (0.304) were observed at 200 atm. Furthermore, the characteristic time constant (τ0) showed a decreasing trend (34.7, 18.1, and 11.7 ps) with increasing HPA. The improved switching properties and detailed findings recorded in this study may be helpful for developing the ferroelectric hafnia based non-volatile memory applications.