Network For Single Image Super-resolution Research Articles

GlobalSR: Global context network for single image super-resolution via deformable convolution attention and fast Fourier convolution

Vision Transformer have achieved impressive performance in image super-resolution. However, they suffer from low inference speed mainly because of the quadratic complexity of multi-head self-attention (MHSA), which is the key to learning long-range dependencies. On the contrary, most CNN-based methods neglect the important effect of global contextual information, resulting in inaccurate and blurring details. If one can make the best of both Transformers and CNNs, it will achieve a better trade-off between image quality and inference speed. Based on this observation, firstly assume that the main factor affecting the performance in the Transformer-based SR models is the general architecture design, not the specific MHSA component. To verify this, some ablation studies are made by replacing MHSA with large kernel convolutions, alongside other essential module replacements. Surprisingly, the derived models achieve competitive performance. Therefore, a general architecture design GlobalSR is extracted by not specifying the core modules including blocks and domain embeddings of Transformer-based SR models. It also contains three practical guidelines for designing a lightweight SR network utilizing image-level global contextual information to reconstruct SR images. Following the guidelines, the blocks and domain embeddings of GlobalSR are instantiated via Deformable Convolution Attention Block (DCAB) and Fast Fourier Convolution Domain Embedding (FCDE), respectively. The instantiation of general architecture, termed GlobalSR-DF, proposes a DCA to extract the global contextual feature by utilizing Deformable Convolution and a Hadamard product as the attention map at the block level. Meanwhile, the FCDE utilizes the Fast Fourier to transform the input spatial feature into frequency space and then extract image-level global information from it by convolutions. Extensive experiments demonstrate that GlobalSR is the key part in achieving a superior trade-off between SR quality and efficiency. Specifically, our proposed GlobalSR-DF outperforms state-of-the-art CNN-based and ViT-based SISR models regarding accuracy-speed trade-offs with sharp and natural details.

Rectified Binary Network for Single-Image Super-Resolution.

Binary neural network (BNN) is an effective approach to reduce the memory usage and the computational complexity of full-precision convolutional neural networks (CNNs), which has been widely used in the field of deep learning. However, there are different properties between BNNs and real-valued models, making it difficult to draw on the experience of CNN composition to develop BNN. In this article, we study the application of binary network to the single-image super-resolution (SISR) task in which the network is trained for restoring original high-resolution (HR) images. Generally, the distribution of features in the network for SISR is more complex than those in recognition models for preserving the abundant image information, e.g., texture, color, and details. To enhance the representation ability of BNN, we explore a novel activation-rectified inference (ARI) module that achieves a more complete representation of features by combining observations from different quantitative perspectives. The activations are divided into several parts with different quantification intervals and are inferred independently. This allows the binary activations to retain more image detail and yield finer inference. In addition, we further propose an adaptive approximation estimator (AAE) for gradually learning the accurate gradient estimation interval in each layer to alleviate the optimization difficulty. Experiments conducted on several benchmarks show that our approach is able to learn a binary SISR model with superior performance over the state-of-the-art methods. The code will be released at https://github.com/jwxintt/Rectified-BSR.

FDE-Net: A memory-efficiency densely connected network inspired from fractional-order differential equations for single image super-resolution

Dense connection has proved an effective way to take full advantage of hierarchical features extracted from low-resolution images using deep neural networks (DNNs) for single image super-resolution (SISR). However, existing densely connected networks are often manually designed and overly depended on practical experience, thus leading to suboptimal performance. Moreover, due to their complicated connections, they are memory-consuming and lack of interpretability. To address all these problems with one stone, we propose to construct a new densely connected network for SISR from a dynamic system perspective. Following this idea, we cast the hierarchical transformation of DNNs into the state evolution of a fractional-order dynamic system, which empowers us to automatically construct two interdependent densely connected modules based on the system solution rather manual design. They are a prediction module that controls the system to iteratively predict the next state, and a correction module that iteratively refines the predicted state to improve the prediction accuracy. With these two modules as backbone, we establish a Fractional-order Differential Equations-based network (FDE-Net) for SISR. Since the iterative computational manner requires both densely connected modules to be of the recurrent structure, FDE-Net is memory-efficiency and good interpretability. In addition, we analyze the existence and uniqueness of the solution for FDE to theoretically guarantee the feasibility of FDE-Net. Experiments on four SISR benchmark datasets demonstrate the superiority of FDE-Net over existing densely connected networks and other baselines in terms of generalization capacity, especially with limited memory.

Enhanced pyramidal residual networks for single image super-resolution

Several super-resolution (SR) techniques are introduced in the literature, including traditional and machine learning-based algorithms. Especially, deep learning-based SR approaches emerge with demands for better quality images providing deeper subpixel enhancement. Dealing with the image enhancement task in the satellite images domain, a new SR method for single image SR, namely Enhanced Deep Pyramidal Residual Networks, is introduced in this study. The proposed method overcomes the potential instability problem of Enhanced Deep Residual Networks for Single Image Super-Resolution (EDSR) approach by gradually increasing the feature maps depending upon Pyramidal Residual Networks architecture. The EDSR itself is a good algorithm in the SR domain. However, it has a strict structure for increasing the block size. To overcome this problem with the aim of increasing the algorithm’s performance, the pyramidal residual networks gradually increasing hypothesis is utilized in the proposed approach, which is the main contribution and novelty of this study. Besides, by using the pyramidal residual networks gradually increasing hypothesis in the proposed approach, the parameter size of the models is also reduced, which affects the computational time. Two different models are proposed by considering addition and multiplication manners, and the proposed models are evaluated using well-known remote sensing datasets NWPU-RESISC45 and UC Merced. The results obtained by the proposed model are compared with the results of traditional image enhancement algorithms together with the EDSR itself, EDSR with deeper structure, Super-Resolution Generative Adversarial Networks approach, and Residual Local Feature Networks approach in terms of peak signal to noise ratio (PSNR) and structural similarity index measure (SSIM) metrics and showed that the proposed models present better quality images. Moreover, considering the computational time and complexity, it is shown that some proposed models achieve approximately 27% less output parameter having similar PSNR and SSIM values and computational time for EDSR itself and 65% less output parameter having better PSNR and SSIM values and 16% lower computational time for EDSR with deeper structure.

Lightweight blueprint residual network for single image super-resolution

The application of deep convolutional neural networks (CNNs) makes the lightweight single image super-resolution (SISR) task develop rapidly in recent years. However, existing lightweight SISR networks for the terminals with limited computing resources are still not efficient enough. In order to further alleviate the model complexity while keeping remarkable reconstruction performance, in this paper, we propose a lightweight blueprint residual network (LBRN), which is a powerful CNN-based SR model while simultaneously maintaining extremely lightweight. Specifically, we design a novel lightweight blueprint residual block (LBRB) to learn the high-frequency information efficiently for lightweight SISR. The LBRB with fewer parameters consists of more efficient convolution operations to extract high-frequency information which helps recover more visual details. In addition, we propose an effective feature fusion residual block (EFFRB) which is composed of three LBRBs with convolutional kernels of different sizes. The EFFRB, which is designed with effective multi-scale receptive fields, can availably extract and fuse multi-scale high-frequency information to obtain discriminative feature, and it further improves model performance. Extensive experiments show that our LBRN achieves superior reconstruction performance quantitatively and visually compared with other lightweight state-of-the-art SR methods.

Dual-space high-frequency learning for transformer-based MRI super-resolution

Background and ObjectiveMagnetic resonance imaging (MRI) can provide rich and detailed high-contrast information of soft tissues, while the scanning of MRI is time-consuming. To accelerate MR imaging, a variety of Transformer-based single image super-resolution methods are proposed in recent years, achieving promising results thanks to their superior capability of capturing long-range dependencies. Nevertheless, most existing works prioritize the design of transformer attention blocks to capture global information. The local high-frequency details, which are pivotal to faithful MRI restoration, are unfortunately neglected. MethodsIn this work, we propose a high-frequency enhanced learning scheme to effectively improve the awareness of high frequency information in current Transformer-based MRI single image super-resolution methods. Specifically, we present two entirely plug-and-play modules designed to equip Transformer-based networks with the ability to recover high-frequency details from dual spaces: 1) in the feature space, we design a high-frequency block (Hi-Fe block) paralleled with Transformer-based attention layers to extract rich high-frequency features; while 2) in the image intensity space, we tailor a high-frequency amplification module (HFA) to further refine the high-frequency details. By fully exploiting the merits of the two modules, our framework can recover abundant and diverse high-frequency information, rendering faithful MRI super-resolved results with fine details. ResultsWe integrated our modules with six Transformer-based models and conducted experiments across three datasets. The results indicate that our plug-and-play modules can enhance the super-resolution performance of all foundational models to varying degrees, surpassing the capabilities of existing state-of-the-art single image super-resolution networks. ConclusionComprehensive comparison of super-resolution images and high-frequency maps from various methods, clearly demonstrating that our module possesses the capability to restore high-frequency information, showing huge potential in clinical practice for accelerated MRI reconstruction.

RTEN-SR: A reference-based texture enhancement network for single image super-resolution

Most current super-resolution (SR) reconstruction methods suffer from edge blurring and insufficient detail reconstruction. To avoid these problems, this paper proposes a reference-based texture enhancement network for single image SR (RTEN-SR). Firstly, a preliminary reconstruction module (PRM) is constructed to learn the initial reconstructed high-resolution (HR) image features. Then, a multi-scale texture enhancement module (MTEM) based on the idea of texture transfer is designed to further supplement and enhance the texture details for the initial reconstructed HR images. To improve the feature learning ability of network, an efficient mixed attention module (EMAM) is constructed by combining channel attention and spatial attention. Meanwhile, the EMAM is integrated into the above two modules to enhance the important image features in the channel and spatial dimensions. Experimental results demonstrate that the proposed method can achieve better performance than state-of-the-art methods, effectively improving the objective indexes and visual results of reference-based image super-resolution (RefSR). For example, the proposed method outperforms the best RefSR method among the comparison methods by 0.38 dB in PSNR metric and 0.128 in SSIM metric.

Post-trained Convolution Networks for Single Image Super-resolution (Abstract Reprint)

A new method is proposed to increase the accuracy of the state-of-the-art single image super-resolution (SISR) using novel training procedure. The proposed method, named post-trained convolutional neural network (CNN), is carried out stochastic dual simplex algorithm (SDSA) in the last reconstruction layer. The method utilizes contextual information to update the last reconstruction layer of CNN. The extracted contextual information is projected to the last reconstructed layer by optimized weights and the bias is managed through SDSA. Post-trained CNN is applied to the very deep super-resolution (VDSR) method to show its performance. The quantitative and visual results demonstrate that the proposed post-trained VDSR (PTVDSR) exhibits excellent and competitive performance when compared with the VDSR and other super-resolution methods.

Dual contrastive attention-guided deformable convolutional network for single image super-resolution

With its powerful ability to model geometric transformations, the deformable convolutional network brings great improvements for single image super-resolution (SISR). Nevertheless, its location-variant sampling method leads to an escalation in spatial variance as the deformable convolutional layers are stacked, consequently resulting in limited performance. Hence, we propose a novel and effective approach called dual contrastive attention-guided deformable convolutional network (DCADCN) for SISR modeling. Specifically, we propose an attention-guided deformable convolutional module with joint inner and external attention mechanisms to fully exploit the correspondences between input and deformation features and preserve spatial characteristics to the extent possible. Additionally, we propose a dual mixed feature extractor consisting of two parallel sub-paths. This design allows for the learning of diverse and complementary spatial features. Furthermore, contrastive learning is applied to further amplify the role of key features and mitigate the interference of noisy features. Extensive experimental results demonstrate that DCADCN is capable of effectively handling classic SISR, SISR with blind noise, and real-world SISR tasks. Moreover, our method achieves comparable or even better performance with lower computational cost compared to state-of-the-art methods.

A Channel-Wise Multi-Scale Network for Single Image Super-Resolution

A Channel-Wise Multi-Scale Network for Single Image Super-Resolution

A Two-Branch Multiscale Residual Attention Network for Single Image Super-Resolution in Remote Sensing Imagery

A Two-Branch Multiscale Residual Attention Network for Single Image Super-Resolution in Remote Sensing Imagery

RefQSR: Reference-Based Quantization for Image Super-Resolution Networks.

Single image super-resolution (SISR) aims to reconstruct a high-resolution image from its low-resolution observation. Recent deep learning-based SISR models show high performance at the expense of increased computational costs, limiting their use in resource-constrained environments. As a promising solution for computationally efficient network design, network quantization has been extensively studied. However, existing quantization methods developed for SISR have yet to effectively exploit image self-similarity, which is a new direction for exploration in this study. We introduce a novel method called reference-based quantization for image super-resolution (RefQSR) that applies high-bit quantization to several representative patches and uses them as references for low-bit quantization of the rest of the patches in an image. To this end, we design dedicated patch clustering and reference-based quantization modules and integrate them into existing SISR network quantization methods. The experimental results demonstrate the effectiveness of RefQSR on various SISR networks and quantization methods.

Attention for Adaptive Convolution in Convolutional Neural Networks for Single Image Super-Resolution

Attention for Adaptive Convolution in Convolutional Neural Networks for Single Image Super-Resolution

CVANet: Cascaded visual attention network for single image super-resolution

Deep convolutional neural networks (DCNNs) have exhibited excellent feature extraction and detail reconstruction capabilities for single image super-resolution (SISR). Nevertheless, most previous DCNN-based methods do not fully utilize the complementary strengths between feature maps, channels, and pixels. Therefore, it hinders the ability of DCNNs to represent abundant features. To tackle the aforementioned issues, we present a Cascaded Visual Attention Network for SISR called CVANet, which simulates the visual attention mechanism of the human eyes to focus on the reconstruction process of details. Specifically, we first designed a trainable feature attention module (FAM) for feature-level attention learning. Afterward, we introduce a channel attention module (CAM) to reinforce feature maps under channel-level attention learning. Meanwhile, we propose a pixel attention module (PAM) that adaptively selects representative features from the previous layers, which are utilized to generate a high-resolution image. Satisfactory, our CVANet can effectively improve the resolution of images by exploring the feature representation capabilities of different modules and the visual perception properties of the human eyes. Extensive experiments with different methods on four benchmarks demonstrate that our CVANet outperforms the state-of-the-art (SOTA) methods in subjective visual perception, PSNR, and SSIM.The code will be made available https://github.com/WilyZhao8/CVANet.

CRUN: a super lightweight and efficient network for single-image super resolution

CRUN: a super lightweight and efficient network for single-image super resolution

MFFN: Multi‐path feedback fusion network for lightweight image super resolution

AbstractRecently, convolutional neural network (CNN) has shown great power in single image super resolution (SISR) reconstruction, and achieving significant improvements over traditional methods. Despite the great success of these CNN‐based methods, direct application of these methods to some edge devices is impractical due to the large computational overhead required. To address this problem, a novel, lightweight SISR network focusing on speed and accuracy, called the multi‐path feedback fusion network (MFFN), has been designed in this paper. Specifically, in order to extract features more effectively, the authors propose a novel fusion attention feedback block (FAFB) as the main building block of MFFN. The FAFB consists of a backbone branch and several hierarchical branches. The backbone branch is composed of stacked enhanced pixel attentional blocks (EPAB), which are responsible for incremental deep feature learning on the feature map. And the hierarchical branches are responsible for extracting feature maps with different sizes of receptive fields and fusing these feature maps with the features extracted from the trunk branches to achieve multi‐scale feature learning, which the authors refer to this design as the multi‐scale fusion block (MSFB). Extra enhancement information (EIE) is added to each EPAB input, which enables the backbone branch to learn more effectively. On the other hand, the outputs of the cascade branches are further complemented by an additional feedback fusion enhancement block (FFEB) before being fused with the output of the trunk branches to achieve more comprehensive and accurate feature learning. Numerous experiments have shown that MFFN achieves higher accuracy than other state‐of‐the‐art methods on benchmark test sets.

Deep recurrent residual channel attention network for single image super-resolution

Deep recurrent residual channel attention network for single image super-resolution

Recurrent Multi-scale Approximation-Guided Network for Single Image Super-Resolution

Single-image super-resolution (SISR)is an essential topic in computer vision applications. However, most CNN-based SISR approaches directly learn the relationship between low- and high-resolution images while ignoring the contextual texture and detail fidelity to explore super-resolution; thus, they hinder the representational power of CNNs and lead to the unrealistic, distorted reconstruction of edges and textures in the images. In this study, we propose a novel recurrent structure preservation mechanism with the integration and innovative use of multi-scale wavelet transform,Recurrent Multiscale Approximation-guided Network (RMANet), to recursively process the low-frequency and high-frequency sub-networks at each level separately. Unlike traditional wavelet transform, we propose a novelApproximation Level Preservation (ALP)architecture to import and learn the low-frequency sub-networks at each level. Through proposedApproximation level fusion (ALF)and inverse wavelet transform, rich image structures of low frequency at each level can be recursively restored and greatly preserved with the combination of ALP at each level. In addition, a novel low-frequency to high-frequencydetail enhancement (DE)mechanism is also proposed to solve the problem of detail distortion in high-frequency networks by transmitting low-frequency information to the high-frequency network. Finally, a joint loss function is used to balance low-frequency and high-frequency information with different degrees of fusion. In addition to correct restoration, image details are further enhanced by tuning different hyperparameters during training. Compared with the state-of-the-art approaches, the experimental results on synthetic and real datasets demonstrate that the proposed RMANet achieves better performance in visual presentation, especially in image edges and texture details.

Multi-scale convolutional attention network for lightweight image super-resolution

Convolutional neural network (CNN) based methods have recently achieved extraordinary performance in single image super-resolution (SISR) tasks. However, most existing CNN-based approaches increase the model’s depth by stacking massive kernel convolutions, bringing expensive computational costs and limiting their application in mobile devices with limited resources. Furthermore, large kernel convolutions are rarely used in lightweight super-resolution designs. To alleviate the above problems, we propose a multi-scale convolutional attention network (MCAN), a lightweight and efficient network for SISR. Specifically, a multi-scale convolutional attention (MCA) is designed to aggregate the spatial information of different large receptive fields. Since the contextual information of the image has a strong local correlation, we design a local feature enhancement unit (LFEU) to further enhance the local feature extraction. Extensive experimental results illustrate that our proposed MCAN can achieve better performance with lower model complexity compared with other state-of-the-art lightweight methods.

Single-Image Super-Resolution Challenges: A Brief Review

Single-image super-resolution (SISR) is an important task in image processing, aiming to achieve enhanced image resolution. With the development of deep learning, SISR based on convolutional neural networks has also gained great progress, but as the network deepens and the task of SISR becomes more complex, SISR networks become difficult to train, which hinders SISR from achieving greater success. Therefore, to further promote SISR, many challenges have emerged in recent years. In this review, we briefly review the SISR challenges organized from 2017 to 2022 and focus on the in-depth classification of these challenges, the datasets employed, the evaluation methods used, and the powerful network architectures proposed or accepted by the winners. First, depending on the tasks of the challenges, the SISR challenges can be broadly classified into four categories: classic SISR, efficient SISR, perceptual extreme SISR, and real-world SISR. Second, we introduce the datasets commonly used in the challenges in recent years and describe their characteristics. Third, we present the image evaluation methods commonly used in SISR challenges in recent years. Fourth, we introduce the network architectures used by the winners, mainly to explore in depth where the advantages of their network architectures lie and to compare the results of previous years’ winners. Finally, we summarize the methods that have been widely used in SISR in recent years and suggest several possible promising directions for future SISR.