Infrared small moving target detection is one of the crucial techniques in infrared search and tracking systems. This paper presents a novel small moving target detection method for infrared image sequence with complicated background. The key points are given as follows: (1) since target detection mainly depends on the incoherence between target and background, the proposed method separate the target from the background according to the morphological feature diversity between target and background; (2) considering the continuity of target motion in time domain, the target trajectory is extracted by the RX filter in random projection. The experiments on various clutter background sequences have validated the detection capability of the proposed method. The experimental results show that the proposed method can robustly provide a higher detection probability and a lower false alarm rate than baseline methods.