A hyperspectral image (HSI) is often corrupted by various types of noise during image acquisition, e.g., Gaussian noise, impulse noise, stripes, deadlines, and more. Thus, as a preprocessing step, HSI denoising plays a vital role in many subsequent tasks. Recently, a variety of mixed noise removal approaches have been developed for HSI, and the methods based on spatial–spectral double factor and total variation (DFTV) regularization have achieved comparable performance. Additionally, the nonlocal low-rank tensor model (NLR) is often employed to characterize spatial nonlocal self-similarity (NSS). Generally, fully exploring prior knowledge can improve the denoising performance, but it significantly increases the computational cost when the NSS prior is employed. To solve this problem, this article proposes a novel DFTV-based NLR regularization (DFTVNLR) model for HSI mixed noise removal. The proposed model employs low-rank tensor factorization (LRTF) to characterize the spectral global low-rankness (LR), introduces 2-D and 1-D TV constraints on double-factor to characterize the spatial and spectral local smoothness (LS), respectively. Meanwhile, the NLR is applied to the spatial factor to characterize the NSS. Then, we developed an algorithm based on proximal alternating minimization (PAM) to solve the proposed model effectively. Particularly, we effectively controlled the computational cost from two aspects, namely taking small-sized double factor as regularization object and putting the time-consuming NLR model before the main loop with fewer iterations to solve it independently. Finally, considerable experiments on simulated and real noisy HSI substantiate that the proposed method is superior to the related state-of-the-art methods in balancing the denoising effect and speed.