Learning a physics-based filter attachment for hyperspectral imaging with RGB cameras

Abstract

Countless RGB cameras are ubiquitously distributed in our daily lives, serving to perceive and depict the diverse colors of the world. Reconstructing hyperspectral images (HSI) from these trichromatic cameras emerges as a promising solution to address the limitations of existing, costly hyperspectral imaging systems. The performance of HSI reconstruction relies heavily on the camera spectral response (CSR). Thus, designing a better CSR and putting it into practice is the critical issue for RGB-based HSI reconstruction. However, the CSR curves designed in the existing works are overly random, making them challenging to manufacture directly. Additionally, the designed CSR curves require modifications to the camera hardware, resulting in the loss of RGB imaging functionality. In this paper, we propose a hyperspectral imaging system, which involves enhancing the CSR curve of existing RGB cameras and preserving RGB imaging functionality by adding a learnable physics-based spectral filter. Specifically, we first parameterize the spectral filter transmittance as a function of the filter thicknesses, based on the physical constraints of the multilayer interference principle. Then, we propose a joint optimization framework in which the thicknesses of the filter and the hyperspectral reconstruction network are optimized. In this manner, the thicknesses of the filter are obtained and used to manufacture the filter directly. Finally, we construct a prototype and verify the benefits of our spectral filter design method through experiments including both synthetic data and real images.