Infrared dim and small target detection has an important role in the infrared thermal imaging seeker, infrared search and tracking system, space-based infrared system and other applications. Inspired by human visual system (HVS), based on the fusion of significant features of targets, the present study proposes an infrared dim and small target detection algorithm for complex backgrounds. Firstly, in order to calculate the target saliency map, the proposed algorithm initially uses the difference of Gaussian (DoG) and the contourlet filters for the preprocessing and fusion, respectively. Then the multi-scale improved local contrast measure (ILCM) method is applied to obtain the interested target area, effectively suppress the background clutter and improve the target signal-to-clutter ratio. Secondly, the optical flow method is used to estimate motion regions in the saliency map, which matches with the interested target region to achieve the initial target screening. Finally, in order to reduce the false alarm rate, forward pipeline filtering and optical flow estimation information are applied to achieve the multi-frame target recognition and achieve continuous detection of dim and small targets in image sequences. Experimental results show that compared with the conventional Tophat (TOP-HAT) and ILCM algorithms, the proposed algorithm can achieve stable, continuous and adaptive target detection for multiple backgrounds. The area under curve (AUC) and the harmonic average measure F1 are used to measure the overall performance and optimal performance of the target detection effect. For sky, sea and ground backgrounds, the test results of proposed algorithm for most sequences are 1. It is concluded that the proposed algorithm significantly improves the detection effect.