This study investigates the operational characteristics of AlGaN/GaN high-electron-mobility transistors (HEMTs) by employing various passivation materials with different dielectric constants and passivation structures. To ensure the simulation reliability, the parameters were calibrated based on the measured data from the fabricated basic Si3N4 passivation structure of the HEMT. The Si3N4 passivation material was replaced with high-k materials, such as Al2O3 and HfO2, to improve the breakdown voltage. The Al2O3 and HfO2 passivation structures achieved breakdown voltage improvements of 6.62% and 17.45%, respectively, compared to the basic Si3N4 passivation structure. However, the increased parasitic capacitances reduced the cut-off frequency. To mitigate this reduction, the operational characteristics of hybrid and partial passivation structures were analyzed. Compared with the HfO2 passivation structure, the HfO2 partial passivation structure exhibited a 7.6% reduction in breakdown voltage but a substantial 82.76% increase in cut-off frequency. In addition, the HfO2 partial passivation structure exhibited the highest Johnson's figure of merit. Consequently, considering the trade-off relationship between breakdown voltage and frequency characteristics, the HfO2 partial passivation structure emerged as a promising candidate for high-power and high-frequency AlGaN/GaN HEMT applications.