Micro-polarizer array (MPA) is the core optical component of the Division of Focal-Plane (DoFP) imaging system, and its design is very important to the system's performance. Traditional design methods rely on theoretical analysis and simulation, which is complicated and requires designers to have profound theoretical foundations. In order to simplify the design process and improve efficiency, this paper proposes a 2 × 2 MPA reverse-design strategy based on particle swarm optimization (PSO). This strategy uses intelligent algorithms to automatically explore the design space in order to discover MPA structures with optimal optical properties. In addition, the all-pass filter is introduced to the MPA superpixel unit in the design, which effectively reduces the crosstalk and frequency aliasing between pixels. In this study, two MPA models were designed: a traditional MPA and an MPA with an all-pass filter. The Degree of Linear Polarization (DOLP) image contrast is used as the evaluation standard and compared with the traditional MPA; the results show that the contrast of the newly designed traditional MPA image is increased by 21%, and the MPA image with the all-pass filter is significantly increased by 82%. Therefore, the reverse-design method proposed in this paper not only simplifies the design process but also can design an MPA with enhanced optical performance, which has obvious advantages over the traditional method.