Detecting small Infrared (IR) targets against the low-altitude complex background is always a challenge for Infrared search and tracking (IRST) system due to limited small target characteristics, the moving background caused by camera motion, and extremely cluttered backgrounds. The existing methods usually cause high false alarm or do not work against the chaotic low-altitude complex background. In this paper, a novel spatial-temporal tensor model with saliency filter regularization (STTM-SFR) is developed to detect small IR targets. Firstly, the small target detection task is transformed into a sparse and low-rank tensor optimization problem by using the spatial-temporal prior knowledge of background and target. The construction of the holistic spatial-temporal tensor model (STTM) can retain the complete spatial-temporal information of the original IR image sequence. Then, the saliency filter regularization term limited between background and foreground aims to promote target saliency learning. That is to say, the saliency filter regularization term can avoid the offset approximation of the low-rank tensor, so as to recover a clean target image from the original IR tensor. Finally, an effective alternating direction method of multipliers (ADMM) algorithm framework is designed to solve the proposed STTM-SFR model. The effectiveness and robustness of STTM-SFR model are verified in six real IR scenes. Experimental results show that our method outperforms other baseline methods. Moreover, the proposed STTM-SFR method is more robust than the existing state-of-the-art STTMs against low-altitude moving backgrounds.