We propose the implementation of an imaging telescope in combination with an inter-frequency calibrator to measure the spectral shape of the microwave sky by exploiting the differences in the sky intensity between multiple pairs of frequency channels. By jointly sampling the cosmological and foreground parameters in a Bayesian framework for $600$ instrument configurations, we determine the minimum calibration accuracy required in order to obtain measurements of spectral distortions and simultaneously measure spectral and spatial fluctuations of the CMB. We demonstrate the feasibility of this technique for a CMB mission like PICO (Probe of Inflation and Cosmic Origins), and show that a $10$-$\sigma$ measurement of the $y$-distortion along with a two orders of magnitude improvement on the FIRAS (Far-Infrared Absolute Spectrophotometer) $\mu$-distortion limit is feasible from this technique. We argue that longer term applications may be envisaged at even higher sensitivity, capable of attaining the $\mu$-distortion that provide a robust prediction of the $\Lambda$CDM model and even primordial recombination lines of hydrogen and helium, in the context of the ESA (European Space Agency) Voyage 2035-2050 program.