Efficient Transfer Learning for Spectral Image Reconstruction from RGB Images

Abstract

Spectral image reconstruction from RGB images has emerged as a hot topic in the computer vision community due to easy-access and low-cost acquisition of the latter. The goal is to learn a non-linear mapping from 3-RGB bands to L spectral bands. With the growth of the available spectral datasets, this mapping has been learned using deep convolutional representations. However, these methods demand a large number of spectral images to train the net to obtain a good recovery. In contrast, the proposed process consists of a pre-training step where the weights of a convolutional neural network fit with a large amount of available RGB datasets without spectral mapping, taking into account the RGB system acquisition as a layer. Then, some layers of this pre-trained network are frozen to retrain it with the available spectral dataset to generate a spectral image with L bands. The proposed training scheme can be used with any pre-existing deep network that maps RGB to spectral images and it is here evaluated with a “U-net” architecture, and the RGB sensing is based on the Bayer filter pattern. The simulated and experimental data demonstrate the effectiveness of the proposed approach compared to training without transfer learning, showing a gain of up to 4 dB, with less spectral data.