In this paper, a sparse smooth twin support vector regression (Sparse‐STSVR) model for power amplifier (PA) behavioral modeling is obtained by pruning the kernel matrix based on Cholesky decomposition. Based on the primal smooth twin support vector regression (STSVR) model, the Nystrom approximate matrix of the kernel matrix is found to replace the original kernel matrix, thus simplifying the Newton iterative parameter extraction process of the primal STSVR model and accelerating the convergence of the algorithm. In addition, the new rank approximation kernel matrix has the characteristic of sparse parameters, which further reduces the computational complexity of the feedforward link of the digital predistorter. The 100 MHz 5G New Radio (NR) signal is used for verify the effect of PA modeling and digital predistortion (DPD) experiment. The results show that the proposed method can improve the normalized mean square error (NMSE) by about 2 ~ 3 dB with fewer coefficients compared with the previously proposed machine learning model, and the predistortion linearization effect improves by nearly 3 dB on the adjacent channel power ratio (ACPR), which achieves a good trade‐off between model performance and computational complexity. © 2024 Institute of Electrical Engineers of Japan and Wiley Periodicals LLC.