Deep Unfolding Network for Spatiospectral Image Super-Resolution

Abstract

In this paper, we explore the spatiospectral image super-resolution (SSSR) task, <i>i.e.</i>, joint spatial and spectral super-resolution, which aims to generate a high spatial resolution hyperspectral image (HR-HSI) from a low spatial resolution multispectral image (LR-MSI). To tackle such a severely ill-posed problem, one straightforward but inefficient way is to sequentially perform a single image super-resolution (SISR) network followed by a spectral super-resolution (SSR) network in a two-stage manner or reverse order. In this paper, we propose a model-based deep learning network for SSSR task, named unfolding spatiospectral super-resolution network (US3RN), which not only uses closed-form solutions to solve SISR subproblem and SSR subproblem, but also has extremely small parameters (only 295 K). In specific, we reformulate the image degradation and incorporate the spatiospectral super-resolution (SSSR) model, which takes the observation models of SISR and SSR into consideration. Then we solve the model-based energy function via the alternative direction multiplier method (ADMM) technique. Finally, we unfold the iterative process of the ADMM algorithm into a multistage network. Therefore, US3RN combines the merits of interpretability and generality of model-based methods with the advantages of learning-based methods. The experimental results show that, compared with the two-step method, US3RN achieves better results both quantitatively and qualitatively, while sharply reducing the number of parameters and FLOPs. Source code will be available at <uri>https://github.com/junjun-jiang/US3RN</uri>.