Efficient image super-resolution based on transformer with bidirectional interaction

Abstract

In single-image super-resolution (SISR) tasks, many methods benefit from the local and global contexts of the image. Despite that, no methods use the bidirectional interaction between these two contexts. So, we were inspired by the fully adaptive Transformer for high-level vision. We propose a fully adaptive Transformer super-resolution (FATSRN) for SISR. The model uses local and global information and their bidirectional interaction in a context-aware manner. The model is based on fully adaptive self-attention (FASA) as the main block, which uses self-modulated convolutions to extract local representation adaptively. Also, the FASA uses self-attention in down-sampled space to extract global representation. In addition, this FASA uses a bidirectional adaptation process between local and global representation to model the interaction. Moreover, a fine-grained downsampling strategy is used to improve the down-sampled self-attention mechanism. Based on the FASA, we built a fully adaptive self-attention block (FASAB) as the main block of our model. Then, the fully adaptive self-attention group (FASAG) is used as the backbone for our FATSRN. Extensive experiments are done to show the efficiency of the model against the state-of-the-art methods. For example, our model improved the PSNR from 27.69 to 27.73 compared to the SwinIR-light for the B100 dataset at the scale of × 4. In addition, our model achieved 0.04 dB better PSNR compared to the state-of-the-art STSN model for the Set5 dataset at the scale of × 2 with 64% and 48% fewer parameters and Mult-adds.