Spectral super-resolution meets deep learning: Achievements and challenges

Abstract

Spectral super-resolution (sSR) is a very important technique to obtain hyperspectral images from only RGB images, which can effectively overcome the high acquisition cost and low spatial resolution of hyperspectral imaging. From linear interpolation to sparse recovery, spectral super-resolution have gained rapid development. In the past five years, as deep learning has taken off in various computer vision tasks, spectral super-resolution algorithms based on deep learning have also exploded. From residual learning to physical modeling, deep learning-based models used in spectral super-resolution is multifarious. This paper has collected almost all deep learning-based sSR algorithms and reviewed them according to their main contributions, involving network architecture, feature extraction, and physical modeling. This paper proposed a benchmark about deep learning-based spectral super-resolution algorithms: https://github.com/JiangHe96/DL4sSR, and besides spectral recovery, their potential in colorization and spectral compressive imaging is also systematically discussed. Furthermore, we presented our views about challenges and possible further trends of deep learning-based sSR. Light-weight model architecture with generalization is crucial to in-camera processing. Model robustness should be considered carefully to manage data with various degradation. Finally, multi-task sSR meets the multiple needs of humans and meanwhile achieves inter-task mutual improvement, including low-level with low-level, low-level with high-level, and data reconstruction with parameter inversion.