A traditional flat-panel spectrometer does not allow high-resolution observation and miniaturization simultaneously. In this study, a compact, high-resolution cross-dispersion spectrometer was designed based on the theoretical basis of echelle grating for recording an infrared spectrum. To meet the high-resolution observation and miniaturization design requirements, a reflective immersion grating was used as the primary spectroscopic device. To compress the beam aperture of the imaging system, the order-separation device of the spectrometer adopted toroidal uniform line grating, which had both imaging and dispersion functions in the spectrometer. The aberration balance condition of the toroidal uniform line grating was analyzed based on the optical path difference function of the concave grating, and dispersion characteristics of the immersed grating and thermal design of the infrared lens were discussed based on the echelle grating. An immersion echelle spectrometer optical system consisting of a culmination system, an immersed echelle grating, and a converged system was used. The spectrometer was based on the asymmetrical Czerny-Turner and Littrow mount designs, and it was equipped with a 320 × 256 pixel detector array. The designed wavelength range was 3.7–4.8 μm, the F-number was 4, and the central wavelength resolution was approximately 30,000. An infrared cooling detector was used. The design results showed that, in the operating band range, the root implied that the square diameter of the spectrometer spot diagram was less than 30 μm, the energy was concentrated in a pixel size range, and the spectrometer system design met the requirements.