To address the challenge of tracking irregular extended target while adapting to unknown measurement noise, this paper presents a novel tracking algorithm for irregular extended target. Considering the flexibility of the Gaussian process (GP) model for representing shape, in the proposed algorithm, we utilize the GP model to delineate the radial function of the tracked extended target’s contour. Then, we incorporate the GP model into the variational Bayesian technique to formulate the posterior distribution for both the unknown measurement noise and the shape state of the tracked target. Subsequently, the square-root cubature Kalman filter is employed to tackle the nonlinearity filtering in the problem of irregular extended target tracking (ETT), enabling simultaneous estimation of intricate shape details and the unknown measurement noise covariance. Through extensive experiments using both simulated and real-time lidar data, the proposed algorithm demonstrates superior performance compared to the existing ETT algorithms for extended target tracking in both accuracy and robustness.