The influence of the orientations of both polarizer and analyzer on modulation depth of spatially distributed interferograms for static polarization interference imaging spectrometer (SPIIS) is analyzed. A generally, theoretical relationship to determine the modulation depth of a SPIIS is derived. The special cases of maximum modulation depth ( V=1) and the minimum modulation depth ( V=0) are examined. Our results will provide a theoretical and practical guide for studying, developing and engineering polarization interference imaging spectrometers.