We reported a wide-band single walled carbon nanotube (SWCNT) saturable absorber (SA) fabricated in the one-dimensional nanochannels of ZnAPO4-5 (AFI) zeolite single crystals. The as-synthesized SA could provide Q-switched operation at both 1038.8 and 1856.5 nm in the ytterbium-, and thulium-doped fiber lasers. Owing to the polarization state in the cavities remained unchanged by the all polarization maintaining (PM) optical paths, the modulation was only related to the nonlinearity of the material. The modulation depths were characterized by the balanced twin-detector system at different wavelengths, showing excellent anisotropic optical properties, which were 5.5 % and 13.0 %, respectively. These results demonstrated that the SWCNT@AFI crystal could act as a specific polarization sensitive SA for some certain scenarios requiring seismic resistance and environmental adaptability, and was an appropriate device for short-duration pulsed fiber laser with a broad spectral coverage.