Light field image super-resolution based on raw data with transformers.

Abstract

Light field (LF) image super-resolution (SR) can improve the limited spatial resolution of LF images by using complementary information from different perspectives. However, current LF image SR methods only use the RGB data to implicitly exploit the information among different perspectives, without paying attention to the information loss from raw data to RGB data and the explicit structure information utilization. To address the first issue, a data generation pipeline is developed to collect LF raw data for LF image SR. In addition, to make full use of the multiview information, an end-to-end convolutional neural network architecture (namely, LF-RawSR) is proposed for LF image SR. Specifically, an aggregated module is first used to fuse the angular information based on a volume transformer with plane sweep volume. Then the aggregated feature is warped to all LF views using a cross-view transformer for nonlocal dependencies utilization. The experimental results demonstrate that our method outperforms existing state-of-the-art methods with a comparative computational cost, and fine details and clear structures can be restored.