The main problem of infrared small target detection in complex background is how to effectively eliminate the edge residue. In this paper, we propose an efficient method named Superpixel Patch Image (SPI) model to handle this challenging task. The SPI model can fit the edges of the background well, thus effectively eliminating edge interference in the process of target detection, and achieving excellent performance. The SPI method consists of three steps: Firstly, an improved Simple Linear Iterative Clustering (ISLIC) algorithm is proposed to generate compact superpixels that perfectly match the background edge. Secondly, setting each superpixel patch as a column, a large patch-image matrix is constructed, and the target foreground image and background image is separated by imprecisely augmented Lagrange multiplication. Finally, based on the comprehensively analysis of the distribution characteristics of the target and the highlighted edge in the foreground image, an adaptive threshold is used to extract the target from the foreground superpixel patch. The experimental results of real infrared scenes show that the presented SPI model achieves the best SCRG, BSF and ROC curves compared with the existing 9 state-of-art algorithms, and can effectively extract small targets under different complex backgrounds.