Single Image Super-resolution Task Research Articles

Non-local sparse attention based swin transformer V2 for image super-resolution

In single image super resolution tasks, distortion measurement (such as PSNR, SSIM) and perceptual quality (such as PI, NIQE) are contradictory, and methods that perform well in perceptual quality often perform poorly in distortion measurement, and vice versa. In this article, we propose a method of balancing the two, which is divided into three stages. Firstly, this article proposes an image super-resolution model NLSAV2 that focuses on PSNR and SSIM metrics. The entire NLSAV2 consists of three modules: shallow feature extraction, deep feature extraction, and high-quality image reconstruction. In the shallow feature extraction module, non-local sparse attention is used to identify the most abundant feature information in mapping input from low-dimensional space to high-dimensional space, and the deep feature extraction module mainly consists of residual Swin Transformer V2 Block. Then, NLSAV2 is used as a generator and a relative discriminator is introduced to further train the model, which is called NLSAV2-GAN. The experimental results indicate that NLSAV2 and NLSAV2-GAN exhibit advantages in distortion measurement and perceptual quality respectively. Finally, network interpolation and image interpolation strategies are used to continuously adjust the reconstruction style and smoothness to achieve a balance between the two.

Local Means Binary Networks for Image Super-Resolution.

The success of modern single image super-resolution (SISR) algorithms is inspired by the development of deep convolutional neural networks (CNNs). However, these CNN-based methods require considerable computation and complexity, making it impossible for these methods to perform real-time calculations in edge devices. Thus, lightweight model design has become a development trend in the super-resolution field, including pruning, quantization, and other methods. The 1-bit quantization is an extreme lightweight method which can reduce the calculation amount of the model in an extreme manner and is friendly to hardware such as edge devices. Most existing binary quantization approaches lead to a large information loss during forward propagation, especially in detailed color information (e.g., edge, texture, and contrast). The loss of color information makes modern binary methods unsuitable for SISR tasks. We think the loss occurs because these methods typically utilize a uniform threshold to quantize the weights and activations. Thus, in this article, we thoroughly analyze the difference between normal classification tasks and SISR tasks, and present a binarization scheme based on local means. The proposed method can maintain more detailed information in feature maps using dynamic thresholds during quantization. Specifically, each value in the full precision activations has a corresponding threshold during the quantization process, and those thresholds are determined by the full precision values of the surroundings. In addition, a gradient approximator is introduced to adaptively optimize the gradient for updating binary weights. We then verify the effectiveness of our method for training binary networks on several SISR benchmarks including VDSR and SRResNet. Experimental results show that the proposed method can outperform the state-of-the-art algorithms to obtain binary networks for image super-resolution with better peak signal-to-noise ratio (PSNR) values and visual quality.

Dual contrastive attention-guided deformable convolutional network for single image super-resolution

With its powerful ability to model geometric transformations, the deformable convolutional network brings great improvements for single image super-resolution (SISR). Nevertheless, its location-variant sampling method leads to an escalation in spatial variance as the deformable convolutional layers are stacked, consequently resulting in limited performance. Hence, we propose a novel and effective approach called dual contrastive attention-guided deformable convolutional network (DCADCN) for SISR modeling. Specifically, we propose an attention-guided deformable convolutional module with joint inner and external attention mechanisms to fully exploit the correspondences between input and deformation features and preserve spatial characteristics to the extent possible. Additionally, we propose a dual mixed feature extractor consisting of two parallel sub-paths. This design allows for the learning of diverse and complementary spatial features. Furthermore, contrastive learning is applied to further amplify the role of key features and mitigate the interference of noisy features. Extensive experimental results demonstrate that DCADCN is capable of effectively handling classic SISR, SISR with blind noise, and real-world SISR tasks. Moreover, our method achieves comparable or even better performance with lower computational cost compared to state-of-the-art methods.

Hierarchical Similarity Learning for Aliasing Suppression Image Super-Resolution.

As a highly ill-posed issue, single-image super-resolution (SISR) has been widely investigated in recent years. The main task of SISR is to recover the information loss caused by the degradation procedure. According to the Nyquist sampling theory, the degradation leads to the aliasing effect and makes it hard to restore the correct textures from low-resolution (LR) images. In practice, there are correlations and self-similarities among the adjacent patches in the natural images. This article considers the self-similarity and proposes a hierarchical image super-resolution network (HSRNet) to suppress the influence of aliasing. We consider the SISR issue in the optimization perspective and propose an iterative solution pattern based on the half-quadratic splitting (HQS) method. To explore the texture with local image prior, we design a hierarchical exploration block (HEB) and progressive increase the receptive field. Furthermore, multilevel spatial attention (MSA) is devised to obtain the relations of adjacent feature and enhance the high-frequency information, which acts as a crucial role for visual experience. The experimental result shows that HSRNet achieves better quantitative and visual performance than other works and remits the aliasing more effectively.

Multi-branch-feature fusion super-resolution network

Deep network attracts extensive attention in the single image super-resolution field. Most of the deep network-based super-resolution methods usually employ a feature extraction stream to generate the input for the next extraction module during extracting deep features. Whereas, many experiments show that the feature maps yielded by different feature extraction streams generally contain different but complementary detailed information. Based on this observation, we propose a multi-branch feature fusion super-resolution network termed as MBFSR to solve the single image super-resolution task. MBFSR progressively deploys multiple feature extraction modules to extract the deep feature. Specifically, the deployed feature extraction modules first generate different feature maps through three well-designed feature extraction branches, and then fuse them by the developed three-branch fusion module to generate a high-resolution feature map. Finally, the output features of all modules are directly used to generate the desired high-resolution image. Experimental results demonstrate the availability and superiority of MBFSR over other popular methods.

Cross-scale collaborative network for single image super resolution

Convolution-based networks show impressive performance in Single Image Super-Resolution (SISR) task by making full use of local patterns. However, it requires deeper structure to capture the long-range dependencies which is inefficient. Hence, the Non-Local Attention (NLA) is introduced to search similar patterns globally and effectively exchange information globally. Nevertheless, the naïve NLA only exchanges information in original spatial scale and has quadratic computational complexity with spatial size of input image. Besides, the global fusion of all similar patterns may lead to blurring, which deviates from the goal of SISR to accurately recover image details. Hence, the naïve NLA is not suit for SISR. To solve this problem, we propose a novel Cross-Scale Collaborative Attention (CSCA) which is designed for SISR. Due to similar patterns tend to occur at different locations and scales, we explore self-similarity and cross-scale similarity simultaneously on multi-scale feature pyramid to utilize more rich details for improving the performance of recovery. To guarantee the recovered details are accurate, we aggregate features in each layer of pyramid into groups and only exchange information between the most relevant groups across multi-scale. Hence, CSCA is both efficient and can accurately recover details using the rich information contained in multi-scale. Furthermore, we propose a Cross-Scale Collaborative Network (CSCN) by inserting a few CSCA modules into a simple backbone network. Extensive experiments show the effectivity and efficiency of CSCA, and state-of-the-art performance of CSCN in SISR task.

Hybrid knowledge distillation from intermediate layers for efficient Single Image Super-Resolution

Convolutional and Transformer models have achieved remarkable results for Single Image Super-Resolution (SISR). However, the tremendous memory and computation consumption of these models restricts their usage in resource-limited scenarios. Knowledge distillation, as an effective model compression technique, has received great research focus on the SISR task. In this paper, we propose a novel efficient SISR method via hybrid knowledge distillation from intermediate layers, termed HKDSR, which leverages the knowledge from frequency information into that RGB information. To accomplish this, we first pre-train the teacher with multiple intermediate upsampling layers to generate the intermediate SR outputs. We then construct two kinds of intermediate knowledge from the Frequency Similarity Matrix (FSM) and Adaptive Channel Fusion (ACF). FSM aims to mine the relationship of frequency similarity between the Ground-truth (GT) HR image, and the intermediate SR outputs of teacher and student by Discrete Wavelet Transformation. ACF merges the intermediate SR output of the teacher and GT HR image in a channel dimension to adaptively align the intermediate SR output of the student. Finally, we leverage the knowledge from FSM and ACF into reconstruction loss to effectively improve student performance. Extensive experiments demonstrate the effectiveness of HKDSR on different benchmark datasets and network architectures.

Single-Image Super-Resolution Challenges: A Brief Review

Single-image super-resolution (SISR) is an important task in image processing, aiming to achieve enhanced image resolution. With the development of deep learning, SISR based on convolutional neural networks has also gained great progress, but as the network deepens and the task of SISR becomes more complex, SISR networks become difficult to train, which hinders SISR from achieving greater success. Therefore, to further promote SISR, many challenges have emerged in recent years. In this review, we briefly review the SISR challenges organized from 2017 to 2022 and focus on the in-depth classification of these challenges, the datasets employed, the evaluation methods used, and the powerful network architectures proposed or accepted by the winners. First, depending on the tasks of the challenges, the SISR challenges can be broadly classified into four categories: classic SISR, efficient SISR, perceptual extreme SISR, and real-world SISR. Second, we introduce the datasets commonly used in the challenges in recent years and describe their characteristics. Third, we present the image evaluation methods commonly used in SISR challenges in recent years. Fourth, we introduce the network architectures used by the winners, mainly to explore in depth where the advantages of their network architectures lie and to compare the results of previous years’ winners. Finally, we summarize the methods that have been widely used in SISR in recent years and suggest several possible promising directions for future SISR.

Learning knowledge representation with meta knowledge distillation for single image super-resolution

Although the deep CNN-based super-resolution methods have achieved outstanding performance, their memory cost and computational complexity severely limit their practical employment. Knowledge distillation (KD), which can efficiently transfer knowledge from a cumbersome network (teacher) to a compact network (student), has demonstrated its advantages in some computer vision applications. The representation of knowledge is vital for knowledge transferring and student learning, which is generally defined in hand-crafted manners or uses the intermediate features directly. In this paper, we propose a model-agnostic meta knowledge distillation method under the teacher–student architecture for the single image super-resolution task. It provides a more flexible and accurate way to help teachers transmit knowledge in accordance with the abilities of students via knowledge representation networks (KRNets) with learnable parameters. Specifically, the texture-aware dynamic kernels are generated from local information to decompose the distillation problem into texture-wise supervision for further promoting the recovery quality of high-frequency details. In addition, the KRNets are optimized in a meta-learning manner to ensure the knowledge transferring and the student learning are beneficial to improving the reconstructed quality of the student. Experiments conducted on various single image super-resolution datasets demonstrate that our proposed method outperforms existing defined knowledge representation-related distillation methods and can help super-resolution algorithms achieve better reconstruction quality without introducing any extra inference complexity.

Investigation into Perceptual-Aware Optimization for Single-Image Super-Resolution in Embedded Systems

Deep learning has been introduced to single-image super-resolution (SISR) in the last decade. These techniques have taken over the benchmarks of SISR tasks. Nevertheless, most architectural designs necessitate substantial computational resources, leading to a prolonged inference time on embedded systems or rendering them infeasible for deployment. This paper presents a comprehensive survey of plausible solutions and optimization methods to address this problem. Then, we propose a pipeline that aggregates the latter in order to enhance the inference time without significantly compromising the perceptual quality. We investigate the effectiveness of the proposed method on a lightweight Generative Adversarial Network (GAN)-based perceptual-oriented model as a case study. The experimental results show that our proposed method leads to significant improvement in the inference time on both Desktop and Jetson Xavier NX, especially for higher resolution input sizes on the latter, thereby making it deployable in practice.

Fast Non-Local Attention network for light super-resolution

Although convolutional neural network-based methods have achieved significant performance improvement for Single Image Super-Resolution (SISR), their vast computational cost hinders real-world environment application. Thus, the interest in light networks for SISR is rising. Since existing SISR light models mainly focus on extracting fine local features using convolution operation, they have a limitation in that networks hardly capture global information. To capture the long-range dependency, Non-Local (NL) attention and Transformers have been explored in the SISR task. However, they are still suffering from a balancing problem between performance and computational cost. In this paper, we propose Fast Non-Local attention NETwork (FNLNET) for a super light SISR, which can capture the global representation. To acquire global information, we propose The Fast Non-Local Attention (FNLA) module that has low computational complexity while capturing global representation that reflects long-distance relationships between patches. Then, FNLA requires only 16 times lower computational cost than conventional NL networks while improving performance. In addition, we propose a powerful module called Global Self-Intension Mining (GSIM) that fuses the multi-information resources such as local, and global representation. Our FNLNET shows outstanding performance with fewer parameters and computational costs in the experiments on the benchmark datasets against state-of-the-art light SISR models.

Differentiable Neural Architecture Search for Extremely Lightweight Image Super-Resolution

Single Image Super-Resolution (SISR) tasks have achieved significant performance with deep neural networks. However, the large number of parameters in CNN-based methods for SISR tasks require heavy computations. Although several efficient SISR models have been recently proposed, most are handcrafted and thus lack flexibility. In this work, we propose a novel differentiable Neural Architecture Search (NAS) approach on both the cell-level and network-level to search for lightweight SISR models. Specifically, the cell-level search space is designed based on an information distillation mechanism, focusing on the combinations of lightweight operations and aiming to build a more lightweight and accurate SR structure. The network-level search space is designed to consider the feature connections among the cells and aims to find which information flow benefits the cell most to boost the performance. Unlike the existing Reinforcement Learning (RL) or Evolutionary Algorithm (EA) based NAS methods for SISR tasks, our search pipeline is fully differentiable, and the lightweight SISR models can be efficiently searched on both the cell-level and network-level jointly on a single GPU. Experiments show that our methods can achieve state-of-the-art performance on the benchmark datasets in terms of PSNR, SSIM, and model complexity with merely 68G Multi-Adds for ×2 and 18G Multi-Adds for ×4 SR tasks.

Enhancing Single-Image Super-Resolution using Patch-Mosaic Data Augmentation on Lightweight Bimodal Network

With the advancement of deep learning, single-image super-resolution (SISR) has made significant strides. However, most current SISR methods are challenging to employ in real-world applications because they are doubtlessly employed by substantial computational and memory costs caused by complex operations. Furthermore, an efficient dataset is a key factor for bettering model training. The hybrid models of CNN and Vision Transformer can be more efficient in the SISR task. Nevertheless, they require substantial or extremely high-quality datasets for training that could be unavailable from time to time. To tackle these issues, a solution combined by applying a Lightweight Bimodal Network (LBNet) and Patch-Mosaic data augmentation method which is the enhancement of CutMix and YOCO is proposed in this research. With patch-oriented Mosaic data augmentation, an efficient Symmetric CNN is utilized for local feature extraction and coarse image restoration. Plus, a Recursive Transformer aids in fully grasping the long-term dependence of images, enabling the global information to be fully used to refine texture details. Extensive experiments have shown that LBNet with the proposed data augmentation with zero-free additional parameters method outperforms the original LBNet and other state-of-the-art techniques in which image-level data augmentation is applied.

Single-image super-resolution via dynamic multi-task learning and multi-level mutual feature fusion

In recent years, deep neural networks have exhibited promising performance in the single-image super-resolution (SISR) task by designing complex network structures from the low-resolution (LR) images to the high-resolution (HR) images and learning the mapping function. However, the SISR task is a widely known ill-posed problem. Since it may map one LR image input to multiple HR images, there are always undesired structural distortions in the recovered SR images. We creatively propose a SISR network with dynamic multi-task learning and multi-level mutual feature fusion (DMMFSR), alleviating structural distortion problems while maintaining more edge structure. Specifically, we utilize a DML block (DMLB) to train low-frequency and high-frequency image information extraction networks so the two networks can mutually promote each other. In addition, we also use a multi-level mutual feature fusion block to fuse image and edge features at different levels. Extensive experiments demonstrate the effectiveness of our proposed DMMFSR. Meanwhile, our model achieves comparable peak signal-to-noise ratio and structural similarity scores on multiple SR benchmarks compared with state-of-the-art SR methods.

Improving trajectory calculations by FLEXPART 10.4+ using single-image super-resolution

Abstract. Lagrangian trajectory or particle dispersion models as well as semi-Lagrangian advection schemes require meteorological data such as wind, temperature and geopotential at the exact spatiotemporal locations of the particles that move independently from a regular grid. Traditionally, these high-resolution data have been obtained by interpolating the meteorological parameters from the gridded data of a meteorological model or reanalysis, e.g., using linear interpolation in space and time. However, interpolation is a large source of error for these models. Reducing them requires meteorological input fields with high space and time resolution, which may not always be available and can cause severe data storage and transfer problems. Here, we interpret this problem as a single-image super-resolution task. That is, we interpret meteorological fields available at their native resolution as low-resolution images and train deep neural networks to upscale them to a higher resolution, thereby providing more accurate data for Lagrangian models. We train various versions of the state-of-the-art enhanced deep residual networks for super-resolution (EDSR) on low-resolution ERA5 reanalysis data with the goal to upscale these data to an arbitrary spatial resolution. We show that the resulting upscaled wind fields have root-mean-squared errors half the size of the winds obtained with linear spatial interpolation at acceptable computational inference costs. In a test setup using the Lagrangian particle dispersion model FLEXPART and reduced-resolution wind fields, we find that absolute horizontal transport deviations of calculated trajectories from “true” trajectories calculated with un-degraded 0.5∘ × 0.5∘ winds are reduced by at least 49.5 % (21.8 %) after 48 h relative to trajectories using linear interpolation of the wind data when training on 2∘ × 2∘ to 1∘ × 1∘ (4∘ × 4∘ to 2∘ × 2∘) resolution data.

Global Learnable Attention for Single Image Super-Resolution.

Self-similarity is valuable to the exploration of non-local textures in single image super-resolution (SISR). Researchers usually assume that the importance of non-local textures is positively related to their similarity scores. In this paper, we surprisingly found that when repairing severely damaged query textures, some non-local textures with low-similarity which are closer to the target can provide more accurate and richer details than the high-similarity ones. In these cases, low-similarity does not mean inferior but is usually caused by different scales or orientations. Utilizing this finding, we proposed a Global Learnable Attention (GLA) to adaptively modify similarity scores of non-local textures during training instead of only using a fixed similarity scoring function such as the dot product. The proposed GLA can explore non-local textures with low-similarity but more accurate details to repair severely damaged textures. Furthermore, we propose to adopt Super-Bit Locality-Sensitive Hashing (SB-LSH) as a preprocessing method for our GLA. With the SB-LSH, the computational complexity of our GLA is reduced from quadratic to asymptotic linear with respect to the image size. In addition, the proposed GLA can be integrated into existing deep SISR models as an efficient general building block. Based on the GLA, we constructed a Deep Learnable Similarity Network (DLSN), which achieves state-of-the-art performance for SISR tasks of different degradation types (e.g., blur and noise). Our code and a pre-trained DLSN have been uploaded to GitHub† for validation.

Lightweight image super-resolution with a feature-refined network

In recent years, deep Convolutional Neural Networks (CNNs) have achieved impressive successes on the Single Image Super-Resolution task (SISR). However, it remains difficult to apply these CNN-based SISR methods in embedded devices due to their high memory and computational requirements. To alleviate this issue, we focus on lightweight SISR methods. The observed similarity between the feature maps in CNNs serves as inspiration to explore the design of a cost-efficient module to obtain feature maps whose representation ability is roughly equivalent to that of a conventional convolutional layer. We thus propose a shadow module applying simple linear transformations with a lower cost to generate similar feature maps. Based on this module, we design a Feature-Refined Block (FRB) to learn more representative features. Besides, we propose a Global Dense Feature Fusion (GDFF) structure to construct a Feature-Refined Network (FRN) with such FRBs for lightweight SISR. Extensive experimental results demonstrate the superior performance of the proposed FRN in comparison with the state-of-the-art lightweight SISR methods, while consuming relatively low memory and computation resources.

Single image super‐resolution based on progressive fusion of orientation‐aware features

Single image super-resolution (SISR) is an active research topic in the fields of image processing, computer vision and pattern recognition, restoring high-frequency details and textures based on the low-resolution input image. In this paper, we aim to build more accurate and faster SISR models via developing better-performing feature extraction and fusion techniques. Firstly, we proposed a novel Orientation-Aware feature extraction/selection Module (OAM), which contains a mixture of 1D and 2D convolutional kernels (i.e., 3×1, 1×3, and 3×3) for extracting orientation-aware features. The channel attention mechanism is deployed within each OAM, performing scene-specific selection of informative outputs of the orientation-dependent kernels (e.g., horizontal, vertical, and diagonal). Secondly, we present an effective fusion architecture to progressively integrate multi-scale features extracted in different convolutional stages. Instead of directly combining low-level and high-level features, similar outputs of adjacent feature extraction modules are grouped and further compressed to generate a more concise representation of a specific convolutional stage for high-accuracy SISR task. Based on the above two important improvements, we present a compact but effective CNN-based model for high-quality SISR via Progressive Fusion of Orientation-Aware features (SISR-PF-OA). Extensive experimental results verify the superiority of the proposed SISR-PF-OA model, performing favorably against the state-of-the-art models in terms of both restoration accuracy and computational efficiency (e.g., SISR-PF-OA outperforms RCAN model, achieving higher PSNR 31.25 dB vs. 31.21 dB and using fewer FLOPs 764.41 G vs. 1020.28 G on the Manga109 dataset for scale factor ×4 SISR task.). The source codes will be made publicly available.

Adaptive feature denoising based deep convolutional network for single image super-resolution

Recently, the feature map recalibration (FMR) mechanism has been widely explored in single image super-resolution (SISR) and obtained remarkable performances. However, the existing FMR-based SISR methods directly incorporate the attention module into a deeper network structure (e.g. EDSR), while neglecting the differences between the low-level and high-level vision problems. In this paper, we design a low-level specific FMR mechanism for SISR task based on a new observation by examining current SISR methods, which all demonstrate a solid correlation between the SISR performance and the convolutional feature noise. Inspired by this, we extend the classic soft thresholding technique in the way of deep network, and develop an Adaptive Soft Thresholding (AST) module for feature noise suppression. Comparing to existing attention modules, AST is light-weighted and can be taken as an easy plug-in module in any SISR networks. To this end, we construct a adaptive Feature Denoising Super-Resolution (FDSR) network by combining the baseline EDSR and the proposed AST. Extensive experimental results show that the proposed FDSR network could achieve the state-of-the-art performances on SISR benchmarks, and significantly reduce the parameter (28.8% for EDSR, 74.6% for RCAN,s 76.0% for SAN) with respect to FMR module.

Best-Buddy GANs for Highly Detailed Image Super-resolution

We consider the single image super-resolution (SISR) problem, where a high-resolution (HR) image is generated based on a low-resolution (LR) input. Recently, generative adversarial networks (GANs) become popular to hallucinate details. Most methods along this line rely on a predefined single-LR-single-HR mapping, which is not flexible enough for the ill-posed SISR task. Also, GAN-generated fake details may often undermine the realism of the whole image. We address these issues by proposing best-buddy GANs (Beby-GAN) for rich-detail SISR. Relaxing the rigid one-to-one constraint, we allow the estimated patches to dynamically seek trustworthy surrogates of supervision during training, which is beneficial to producing more reasonable details. Besides, we propose a region-aware adversarial learning strategy that directs our model to focus on generating details for textured areas adaptively. Extensive experiments justify the effectiveness of our method. An ultra-high-resolution 4K dataset is also constructed to facilitate future super-resolution research.