In this paper, a novel technology named Pelican-Gaussian process regression machine learning algorithm is proposed for modelling the large-signal characteristics of Gallium Nitride High Electron Mobility Transistors (GaN HEMT). Hyperparameter optimization in traditional Gaussian process regression algorithms tends to fall into local optimums and is overly dependent on the initial values. In order to solve this problem, the Pelican optimization algorithm is introduced to optimize the hyperparameters in Gaussian process regression algorithms in the article. The Pelican optimization algorithm is able to make the global exploration and local search ability of the algorithm be effectively balanced by helping particles to escape from the local optimal position. The I–V characteristics, output power, power gain, power gain efficiency and small-signal S-parameters of GaN HEMT devices are used to verify the effectiveness of the proposed algorithm. The experimental results show that higher fitting accuracy and generalization ability is found in the improved GPR whose hyperparameters are optimized by the Pelican optimization algorithm.