To continuously monitor, identify, and classify a wide range of areas all day and satisfy the hyperspectral remote sensing requirements in disaster reduction, environment, agriculture, forestry, marine, and resource areas, the authors participated in a pre-research program for full spectrum hyperspectral detection in geostationary orbit. As part of the program, the authors designed a cryogenic infrared spectrometer working at the diffraction limit. Such spectrometer complied with prism dispersion, exhibiting a 120 mm long slit, 2.5-5 µm band range, and 50 nm minimum spectral resolution. The spectrometer should overtake a temperature variation of 143 K for its assembly temperature at 293 K and the working temperature at 150 K. Low-temperature invar and carbon fiber were adopted as the framework material. The spectrometer was composed of two reflective Zerodur mirrors and one CaF2 Fery prism. Compensation mounts were developed for the reflective mirrors, while a spring-loaded autocentering cryogenic lens mount was designed for a CaF2 prism. CaF2 material exhibits a large linear expansion coefficient, making its mount difficult to design. The alignment requirements of the system were described, and the calculations that ensure the lenses undergo both appropriate stresses and temperature differences were presented. Structural thermal optical performance analysis was also conducted to assess the degradation in optical performance caused by temperature variation to verify the overall optomechanical design.