Colour Fidelity in Spectral Reconstruction from RGB Images

Abstract

Spectral reconstruction (SR) aims to recover high resolution spectra from RGB images. Recent developments - leading by Convolutional Neural Networks (CNN) - can already solve this problem with low errors. However, those leading methods do not explicitly ensure the predicted spectra will re-integrate (with the underlying camera response functions) into the same RGB colours as the ones they are recovered from, namely the 'colour fidelity' problem. The purpose of this paper is to show, visually and quantitatively, how well (or bad) the existing SR models maintain colour fidelity. Three main approaches are evaluated - regression, sparse coding and CNN. Furthermore, aiming for a more realistic setting, the evaluations are done on real RGB images and the 'end-of-pipe' images (i.e.rendered images shown to the end users) are provided for visual comparisons. It is shown that the state-of-the-art CNN-based model, despite of the superior performance in spectral recovery, introduces significant colour shifts in the final images. Interestingly, the leading sparse coding and the simple linear regression model, both of which are based on linear mapping, best preserve the colour fidelity in SR.