Recently, infrared (IR) small target detection problem has attracted increasing attention. Tensor component analysis-based techniques have been widely utilized, while they are faced with challenges such as tensor structures, background and target estimation, and real-time performance. In this paper, we propose a 5-D spatial–temporal factor-based completion model (5D-STFC) for IR small target detection. Specifically, a 5-D whitened spatial–temporal patch-tensor is constructed. Then, we devise a spatial–temporal factor-based low-rank background estimation norm and a Moreau envelope-derived sparsity estimation norm based on joint spatial–temporal knowledge. Furthermore, we establish a comprehensive completion model for component analysis. To efficiently solve this model, we design a multi-block alternating direction method of multipliers (multi-block ADMM)-based optimization scheme. Extensive experiments conducted on five real IR sequences demonstrate the superiority of 5D-STFC over nine state-of-the-art competitive methods. It can be concluded that 5D-STFC is excellent and practical in target detectability, background suppressibility, overall performance, and real-time performance.