This paper presents an enhanced multi-level filter algorithm and its Very Large Scale Integration (VLSI) architecture for infrared image processing. The modified multi-level filter algorithm resolves the splitting targets problem using Gaussian pyramid processing. Owning three filtering paths, the proposed VLSI architecture of the filter can simultaneously enhance small targets with different sizes in infrared images. Some design techniques in implementing hardwired multiplication, subsample and asynchronous FIFO have been presented. This VLSI architecture has been implemented using Semiconductor Manufacturing International Corporation (SMIC) 0.35 µm 4-layer CMOS technology. The simulation results show that it not only effectively suppresses background, eliminates noise and enhances small targets in an infrared image comparing with other small target detective methods, but also meets infrared image real-time processing requirements (5 M ∼ 10 M pixels/s). The implemented filter chip consists of 60,284 gates and 8 K Static Random Access Memory (SRAM), operates at 50 MHz.