In current biomedical and environmental detection, multi-modal optical imaging technology is playing an increasingly important role. By utilizing information from dimensions such as spectra and polarization, it reflects the detailed characteristics and material properties of the targets. However, as detection system performance becomes more complex, issues such as aberrations introduced by multilayered lenses, signal attenuation, decreased polarization sensitivity, and latency can no longer be ignored. These factors directly affect the assessment of image details, influencing subsequent analyses. In this paper, we propose a method for designing and optimizing spectrum-splitting filters that considers the wavefront aberration and transmittance of the multi-modal optical imaging system. The method of optimizing coating phases to minimize scalar phase aberrations while maximizing system transmission leads to substantially improved imaging performance. Simulation and experimental results demonstrate that the method can improve the imaging performance. The proposed approach has potential applications in fields such as biomedical field, multi-spectral, remote sensing and microscopy.