Performance Of Single Image Super-resolution Research Articles

Non-local degradation modeling for spatially adaptive single image super-resolution

Existing methods for single image super-resolution (SISR) model the blur kernel as spatially invariant across the entire image, and are susceptible to the adverse effects of textureless patches. To achieve improved results, adaptive estimation of the degradation kernel is necessary. We explore the synergy of joint global and local degradation modeling for spatially adaptive blind SISR. Our model, named spatially adaptive network for blind super-resolution (SASR), employs a simple encoder to estimate global degradation representations and a decoder to extract local degradation. These two representations are fused with a cross-attention mechanism and applied using spatially adaptive filtering to enhance the local image detail. Specifically, SASR contains two novel features: (1) a non-local degradation modeling with contrastive learning to learn global and local degradation representations, and (2) a non-local spatially adaptive filtering module (SAFM) that incorporates the global degradation and spatial-detail factors to preserve and enhance local details. We demonstrate that SASR can efficiently estimate degradation representations and handle multiple types of degradation. The local representations avoid the detrimental effect of estimating the entire super-resolved image with only one kernel through locally adaptive adjustments. Extensive experiments are performed to quantitatively and qualitatively demonstrate that SASR not only performs favorably for degradation estimation but also leads to state-of-the-art blind SISR performance when compared to alternative approaches.

Cross-Receptive Focused Inference Network for Lightweight Image Super-Resolution

Recently, Transformer-based methods have shown impressive performance in single image super-resolution (SISR) tasks due to the ability of global feature extraction. However, the capabilities of Transformers that need to incorporate contextual information to extract features dynamically are neglected. To address this issue, we propose a lightweight Cross-receptive Focused Inference Network (CFIN) that consists of a cascade of CT Blocks mixed with CNN and Transformer. Specifically, in the CT block, we first propose a CNN-based Cross-Scale Information Aggregation Module (CIAM) to enable the model to better focus on potentially helpful information to improve the efficiency of the Transformer phase. Then, we design a novel Cross-receptive Field Guided Transformer (CFGT) to enable the selection of contextual information required for reconstruction by using a modulated convolutional kernel that understands the current semantic information and exploits the information interaction within different self-attention. Extensive experiments have shown that our proposed CFIN can effectively reconstruct images using contextual information, and it can strike a good balance between computational cost and model performance as an efficient model. Source codes will be available at <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><uri>https://github.com/IVIPLab/CFIN</uri></i> .

Cross-scale collaborative network for single image super resolution

Convolution-based networks show impressive performance in Single Image Super-Resolution (SISR) task by making full use of local patterns. However, it requires deeper structure to capture the long-range dependencies which is inefficient. Hence, the Non-Local Attention (NLA) is introduced to search similar patterns globally and effectively exchange information globally. Nevertheless, the naïve NLA only exchanges information in original spatial scale and has quadratic computational complexity with spatial size of input image. Besides, the global fusion of all similar patterns may lead to blurring, which deviates from the goal of SISR to accurately recover image details. Hence, the naïve NLA is not suit for SISR. To solve this problem, we propose a novel Cross-Scale Collaborative Attention (CSCA) which is designed for SISR. Due to similar patterns tend to occur at different locations and scales, we explore self-similarity and cross-scale similarity simultaneously on multi-scale feature pyramid to utilize more rich details for improving the performance of recovery. To guarantee the recovered details are accurate, we aggregate features in each layer of pyramid into groups and only exchange information between the most relevant groups across multi-scale. Hence, CSCA is both efficient and can accurately recover details using the rich information contained in multi-scale. Furthermore, we propose a Cross-Scale Collaborative Network (CSCN) by inserting a few CSCA modules into a simple backbone network. Extensive experiments show the effectivity and efficiency of CSCA, and state-of-the-art performance of CSCN in SISR task.

Multi-scale convolutional attention network for lightweight image super-resolution

Convolutional neural network (CNN) based methods have recently achieved extraordinary performance in single image super-resolution (SISR) tasks. However, most existing CNN-based approaches increase the model’s depth by stacking massive kernel convolutions, bringing expensive computational costs and limiting their application in mobile devices with limited resources. Furthermore, large kernel convolutions are rarely used in lightweight super-resolution designs. To alleviate the above problems, we propose a multi-scale convolutional attention network (MCAN), a lightweight and efficient network for SISR. Specifically, a multi-scale convolutional attention (MCA) is designed to aggregate the spatial information of different large receptive fields. Since the contextual information of the image has a strong local correlation, we design a local feature enhancement unit (LFEU) to further enhance the local feature extraction. Extensive experimental results illustrate that our proposed MCAN can achieve better performance with lower model complexity compared with other state-of-the-art lightweight methods.

A very lightweight and efficient image super-resolution network

Deep convolutional neural networks significantly improve the performance of single image super-resolution (SISR). Generally, larger networks (i.e., deeper and wider) have better performance. However, larger networks require higher computing and storage costs, which limit their application on resource-constrained devices. Lightweight SISR networks with fewer parameters and smaller computational workloads are highly desirable. The key challenge is to obtain a better balance of model complexity and performance. In this paper, we propose a very lightweight and efficient SISR network. Our main contributions include: (1) Propose a frequency grouping fusion block (FGFB), which can better fuse high-/low-frequency feature information; (2) Propose a multi-way attention block (MWAB), which can exploit the multiple different clues of the feature information; (3) Propose a lightweight residual concatenation block (LRCB), which can combine the advantages of the residual connection and the channel concatenation; (4) Propose a lightweight convolutional block (LConv) for image super-resolution, which can significantly reduce the number of parameters; (5) Propose a progressive interactive group convolution (PIGC), which is more effective than the conventional group convolution. Extensive experimental results demonstrate that our method is significantly superior to other state-of-the-art methods currently available, with a better balance between model complexity and performance.

Adaptive feature denoising based deep convolutional network for single image super-resolution

Recently, the feature map recalibration (FMR) mechanism has been widely explored in single image super-resolution (SISR) and obtained remarkable performances. However, the existing FMR-based SISR methods directly incorporate the attention module into a deeper network structure (e.g. EDSR), while neglecting the differences between the low-level and high-level vision problems. In this paper, we design a low-level specific FMR mechanism for SISR task based on a new observation by examining current SISR methods, which all demonstrate a solid correlation between the SISR performance and the convolutional feature noise. Inspired by this, we extend the classic soft thresholding technique in the way of deep network, and develop an Adaptive Soft Thresholding (AST) module for feature noise suppression. Comparing to existing attention modules, AST is light-weighted and can be taken as an easy plug-in module in any SISR networks. To this end, we construct a adaptive Feature Denoising Super-Resolution (FDSR) network by combining the baseline EDSR and the proposed AST. Extensive experimental results show that the proposed FDSR network could achieve the state-of-the-art performances on SISR benchmarks, and significantly reduce the parameter (28.8% for EDSR, 74.6% for RCAN,s 76.0% for SAN) with respect to FMR module.

Enhanced implicit function-based network for arbitrary-scale image super-resolution

With the great development of deep learning, the performance of single image super-resolution (SR) has achieved tremendous progress. As an emerging and promising branch of the SR task, the arbitrary-scale SR task is receiving increasing attention from researchers due to its efficiency and practicality. Although the recent work learning implicit image function opened a solution for arbitrary-scale image SR, its reconstructed images contained structural distortions caused by defective prediction of high-frequency textures. To overcome this problem and further improve the performance of arbitrary-scale image SR, we propose an effective arbitrary-scale SR network, namely, enhanced arbitrary-scale super-resolution, which achieves the arbitrary-scale SR task in a single model by introducing a local-global encoder and enhanced implicit image function. Unlike conventional SR methods, which only stack up convolutional blocks to extract the local feature, the local-global encoder has two branches in parallel, including the local feature branch and the global prior branch. The former effectively extracts the local feature from a low-resolution image and the latter extracts the global prior to assist in the high-resolution image reconstruction. Next, we redesigned an enhanced implicit image function in the form of a dual modulation multiplayer perceptron (MLP) by replacing the implicit image function with a vanilla MLP. Moreover, we introduce the spatial encoding to further reduce structural distortions of reconstructed images. Extensive experiments were conducted to evaluate the performance and demonstrate the superiority of our proposed model.

Image super-resolution with an enhanced group convolutional neural network

CNNs with strong learning abilities are widely chosen to resolve super-resolution problem. However, CNNs depend on deeper network architectures to improve performance of image super-resolution, which may increase computational cost in general. In this paper, we present an enhanced super-resolution group CNN (ESRGCNN) with a shallow architecture by fully fusing deep and wide channel features to extract more accurate low-frequency information in terms of correlations of different channels in single image super-resolution (SISR). Also, a signal enhancement operation in the ESRGCNN is useful to inherit more long-distance contextual information for resolving long-term dependency. An adaptive up-sampling operation is gathered into a CNN to obtain an image super-resolution model with low-resolution images of different sizes. Extensive experiments report that our ESRGCNN surpasses the state-of-the-arts in terms of SISR performance, complexity, execution speed, image quality evaluation and visual effect in SISR. Code is found at https://github.com/hellloxiaotian/ESRGCNN.

APPLICATION OF PRE-TRAINED REAL-WORLD SUPER RESOLUTION MODELS TO OPTICAL SATELLITE IMAGE

Abstract. The single image super-resolution (SISR) technique refers to improving the resolution over the original image. In recent years, we use deep learning-based convolutional neural networks to improve the spatial resolution of images more reasonably. To train such deep learning models, we use training samples consisting of the original HR images and LR images obtained by bicubic downsampling. However, this method of training data using downsampling has a negative impact when applying the trained model to real images. That is because the downsampling function that occurs in the real image is unknown, and the hypothetically created LR image does not represent the resolution degradation that can realistically occur. Therefore, SISR methods that use realistic degradation called real-world super-resolution (RWSR) have been proposed. In this paper, we investigate how such RWSR methods using realistic degradation affect the SISR performance of satellite images. The results of applying the trained model to optical satellite images show that the RWSR method is not the most effective way to handle optical satellite images when compared to the deep learning method without modeling the degradation. In particular, we showed that the effect of RWSR with not only upsampling but also noise and blur removal is significant in the visibility of optical satellite images. Moreover, pre-trained RWSR models can be an aid in visually deciphering ground objects.

Multi-scale receptive field fusion network for lightweight image super-resolution

In recent years, with the rapid development of deep learning, quantities of convolutional neural network (CNN) based methods are applied to the field of single image super-resolution (SISR), which have excellent performance compared with traditional ones. However, these methods have a large amount of calculation and memory consumption, limiting their application in edge devices. To address this problem, we propose a more lightweight multi-scale receptive field fusion network (MRFN) for fast and accurate SISR. Specifically, we first propose a multi-scale receptive field fusion block to fuse different scales of spatial information and enlarge the overall receptive field simultaneously. Secondly, based on the non-local sparse attention, we introduce a hash-learnable cheap non-local attention to capture long-range dependencies with much fewer parameters and calculations. Moreover, channel attention is also very important for SISR, we further combine directional second-order information and spatial coordinate information into it to enhance its capability, which is called second-order coordinate attention. The cooperation of these components leads to the success of our MRFN. Extensive experimental results show that our method achieves a better trade-off against other state-of-the-art methods in terms of SISR performance and model complexity.

Infrared Image Super-Resolution via Progressive Compact Distillation Network

Deep convolutional neural networks are capable of achieving remarkable performance in single-image super-resolution (SISR). However, due to the weak availability of infrared images, heavy network architectures for insufficient infrared images are confronted by excessive parameters and computational complexity. To address these issues, we propose a lightweight progressive compact distillation network (PCDN) with a transfer learning strategy to achieve infrared image super-resolution reconstruction with a few samples. We design a progressive feature residual distillation (PFDB) block to efficiently refine hierarchical features, and parallel dilation convolutions are utilized to expand PFDB’s receptive field, thereby maximizing the characterization power of marginal features and minimizing the network parameters. Moreover, the bil-global connection mechanism and the difference calculation algorithm between two adjacent PFDBs are proposed to accelerate the network convergence and extract the high-frequency information, respectively. Furthermore, we introduce transfer learning to fine-tune network weights with few-shot infrared images to obtain infrared image mapping information. Experimental results suggest the effectiveness and superiority of the proposed framework with low computational load in infrared image super-resolution. Notably, our PCDN outperforms existing methods on two public datasets for both ×2 and ×4 with parameters less than 240 k, proving its efficient and excellent reconstruction performance.

Image super-resolution based on adaptive cascading attention network

Single image super-resolution (SISR) refers to the task restoring a high-resolution (HR) image from its low-resolution (LR) counterpart. Deep convolutional neural networks (CNNs) have significantly improved SISR performance. However, the improvements usually come at the cost of the increased network size, which is not practical for the resource constrained mobile devices. In this paper, we propose a lightweight adaptive cascading attention network (ACAN) for SISR. Our contributions are threefold. First, ACAN can assign different weights to each pixel on each channel, so as to select the important information to reconstruct a high quality HR image. Second, ACAN can adaptively combine hierarchical features and effectively reuse the features. Third, ACAN can adaptively reconstruct an HR image using multi-scale global features. Compared with the other state of the art methods, ACAN achieves a better balance between performance and computational cost.

Deep Stereoscopic Image Super-Resolution via Interaction Module

Deep learning-based methods have achieved remarkable performance in single image super-resolution. However, these methods cannot be effectively applied in stereoscopic image super-resolution without considering the characteristics of stereoscopic images. In this article, an interaction module-based stereoscopic image super-resolution network (IMSSRnet) is proposed to effectively utilize the correlation information in stereoscopic images. The key insight of the network lies with how to explore the complementary information of one view to help the reconstruction of another view. Thus, an interaction module is designed to acquire the enhanced features by utilizing complementary information between different views. Specifically, the interaction module is composed of a series of interaction units with a residual structure. In addition, the single image features of left and right views are obtained by a spatial feature extraction module, which can be realized by any existing single image super-resolution models. In order to obtain high-quality stereoscopic images, a gradient loss is introduced to preserve the texture details in a view, and a disparity loss is developed to constrain the disparity relationship between different views. Experimental results demonstrate that the proposed method achieves a promising performance and outperforms the state-of-the-art methods.

A lightweight multi-scale channel attention network for image super-resolution

In recent years, deep learning techniques have significantly improved the performance of single image super-resolution (SISR). However, this improvement is often achieved at the cost of introducing a large amount of parameters, which limits the real-world applications for SISR. In this paper, we propose a lightweight SISR network called Multi-scale Channel Attention Network for Image Super-Resolution (MCSN). Our contributions are threefold. First of all, the multi-scale feature fusion block (MSFFB) can extract multi-scale features by filters with different receptive fields. Secondly, the channel shuffle attention mechanism (CSAM) encourages the flow of the information across feature channels and enhances the ability of feature selection. Thirdly, the global feature fusion connection (GFFC) can effectively improve feature utilization. Extensive experiments demonstrate that the parameter amount of our method is reduced by 3/4 compared with the current state-of-the-art MSRN method, while both the subjective visual effect and objective quality of the reconstructed high-resolution images are significantly better.

A Lightweight Dense Connected Approach with Attention on Single Image Super-Resolution

In recent years, neural networks for single image super-resolution (SISR) have applied more profound and deeper network structures to extract extra image details, which brings difficulties in model training. To deal with deep model training problems, researchers utilize dense skip connections to promote the model’s feature representation ability by reusing deep features of different receptive fields. Benefiting from the dense connection block, SRDensenet has achieved excellent performance in SISR. Despite the fact that the dense connected structure can provide rich information, it will also introduce redundant and useless information. To tackle this problem, in this paper, we propose a Lightweight Dense Connected Approach with Attention for Single Image Super-Resolution (LDCASR), which employs the attention mechanism to extract useful information in channel dimension. Particularly, we propose the recursive dense group (RDG), consisting of Dense Attention Blocks (DABs), which can obtain more significant representations by extracting deep features with the aid of both dense connections and the attention module, making our whole network attach importance to learning more advanced feature information. Additionally, we introduce the group convolution in DABs, which can reduce the number of parameters to 0.6 M. Extensive experiments on benchmark datasets demonstrate the superiority of our proposed method over five chosen SISR methods.

Attention Mechanisms in CNN-Based Single Image Super-Resolution: A Brief Review and a New Perspective

With the advance of deep learning, the performance of single image super-resolution (SR) has been notably improved by convolution neural network (CNN)-based methods. However, the increasing depth of CNNs makes them more difficult to train, which hinders the SR networks from achieving greater success. To overcome this, a wide range of related mechanisms has been introduced into the SR networks recently, with the aim of helping them converge more quickly and perform better. This has resulted in many research papers that incorporated a variety of attention mechanisms into the above SR baseline from different perspectives. Thus, this survey focuses on this topic and provides a review of these recently published works by grouping them into three major categories: channel attention, spatial attention, and non-local attention. For each of the groups in the taxonomy, the basic concepts are first explained, and then we delve deep into the detailed insights and contributions. Finally, we conclude this review by highlighting the bottlenecks of the current SR attention mechanisms, and propose a new perspective that can be viewed as a potential way to make a breakthrough.

A lightweight network with bidirectional constraints for single image super-resolution

The existing methods based on the convolutional neural network (CNN) show excellent performance in single image super-resolution (SISR). Massive convolutional layers can enhance these CNN-based methods, but they increase memory consumption due to a large number of parameters. In addition, the unilateral constraint on SISR only focuses on the up-sampling process and ignores image degradation, limiting the accuracy of convergence in network training. In response to these problems, we propose a lightweight network with bidirectional constraints (LNBC) for SISR. We present an extended layer named enhanced cycle residual block (CRB), then develop a lightweight network with CRB as the feature inference layer. CRB improves feature expression ability by alternating and multiplexing convolutional layers without increasing parameters. In addition, unlike existing methods that only constrain the up-sampling process, we propose bidirectional constraints on SISR named cycle consistency verification (CCV). In CCV, the introduced degradation network simulates image degradation, and provides constraints on image degradation for the up-sampling network in joint training. Image up-sampling and degradation constitute bidirectional constraints to tighten the convergence of the up-sampling network. Experiments show that LNBC advances comparison methods in the compromise of image super-resolution performance, memory overhead and running time.

Image super-resolution using multi-granularity perception and pyramid attention networks

Recently, single image super-resolution (SISR) has been widely applied in the fields of multimedia and computer vision communities and obtained remarkable performance. However, most current methods ignore to utilize multi-granularity features of low-resolution (LR) image to further improve the SISR performance. And the channel and spatial features obtained from original LR images are treated equally, resulting in unnecessary computations for abundant uninformative features, thereby hindering the representational ability of super-resolution (SR) models. In this paper, we present a novel Multi-Granularity Pyramid Attention Network (MGPAN) which fully exploits the multi-granularity perception and attention mechanisms to improve the quality of reconstructed images. We design a multi-branch dilated convolution layer with varied kernels corresponding to receptive fields of different sizes to modulate multi-granularity features for adaptively capturing more important information. Moreover, a novel spatial pyramid pooling attention (SPPA) module is constructed to integrate the channel-wise and multi-scale spatial information, which is beneficial to compute the response values from the multi-scale regions of each neuron, and then establish the accurate mapping from low to high dimensional solution space. Besides, for long-short-term information preservation and information flow enhancement, we adopt the short, long, and global skip connection structures to concatenate and fuse the states of each module, which can improve the SR network performance effectively. Extensive experiments on several standard benchmark datasets show that the proposed MGPAN can provide state-of-the-art or even better performance in both quantitative and qualitative measurements.

Gradient information distillation network for real-time single-image super-resolution

In recent years, deep convolutional neural networks have played an increasingly important role in single-image super-resolution (SR). However, with the increase of the depth and width of networks, the super-resolution methods based on convolution neural networks are facing training difficulties, memory consumption, running slowness and other problems. Furthermore, most of the methods do not make full use of the image gradient information which leads to the loss of geometric structure information of the image. To solve these problems, we propose a gradient information distillation network in this paper. On the one hand, the advantages of fast and lightweight are maintained through information distillation. On the other hand, the SR performance is improved by gradient information. Our network has two branches named gradient information distillation branch (GIDB) and image information distillation branch. To combine features in both branches, we also introduce a residual feature transfer mechanism (RFT). Under the function of GIDB and RFT, our network can retain the rich geometric structure information which can make the edge details of the reconstructed image sharper. The experimental results show that our method is superior to the existing methods while well limits the parameters, computation and running time of the model. It provides the possibility for real-time image processing and mobile applications.

Embarrassingly Simple Binarization for Deep Single Imagery Super-Resolution Networks.

Deep convolutional neural networks (DCCNs) have shown pleasing performance in single image super-resolution (SISR). To deploy them onto real devices with limited storage and computational resources, a promising solution is to binarize the network, i.e., quantize each float-point weight and activation into 1 bit. However, existing works on binarizing DCNNs still suffer from severe performance degradation in SISR. To mitigate this problem, we argue that the performance degradation mainly comes from no appropriate constraint on the network weights, which causes it difficult to sensitively reverse the binarization results of these weights using the backpropagated gradient during training and thus limits the flexibility of network in respect of fitting extensive training samples. Inspired by this, we present an embarrassingly simple but effective binarization scheme for SISR, which can obviously relieve the performance degeneration resulted from network binarization and is applicable to different DCNN architectures. Specifically, we force each weight to follow a compact uniform prior, with which the weight will be given a very small absolute value close to zero and its binarization result can be straightforwardly reversed even by a small backpropagated gradient. By doing this, the flexibility and the generalization performance of the binarized network can be improved. Moreover, such a prior performs much better when introducing real identity shortcuts into the network. In addition, to avoid falling into bad local minima during training, we employ a pixel-wise curriculum learning strategy to learn the constrained weights in an easy-to-hard manner. Experiments on four SISR benchmark datasets demonstrate the effectiveness of the proposed binarization method in terms of binarizing different SISR network architectures, e.g., it even achieves performance comparable to the baseline with 5 quantization bits.