Image Super-resolution Task Research Articles

Differentiable Neural Architecture Search for Extremely Lightweight Image Super-Resolution

Single Image Super-Resolution (SISR) tasks have achieved significant performance with deep neural networks. However, the large number of parameters in CNN-based methods for SISR tasks require heavy computations. Although several efficient SISR models have been recently proposed, most are handcrafted and thus lack flexibility. In this work, we propose a novel differentiable Neural Architecture Search (NAS) approach on both the cell-level and network-level to search for lightweight SISR models. Specifically, the cell-level search space is designed based on an information distillation mechanism, focusing on the combinations of lightweight operations and aiming to build a more lightweight and accurate SR structure. The network-level search space is designed to consider the feature connections among the cells and aims to find which information flow benefits the cell most to boost the performance. Unlike the existing Reinforcement Learning (RL) or Evolutionary Algorithm (EA) based NAS methods for SISR tasks, our search pipeline is fully differentiable, and the lightweight SISR models can be efficiently searched on both the cell-level and network-level jointly on a single GPU. Experiments show that our methods can achieve state-of-the-art performance on the benchmark datasets in terms of PSNR, SSIM, and model complexity with merely 68G Multi-Adds for ×2 and 18G Multi-Adds for ×4 SR tasks.

A Dynamic Fusion of Local and Non-Local Features-Based Feedback Network on Super-Resolution

Many Symmetry blocks were proposed in the Single Image Super-Resolution (SISR) task. The Attention-based block is powerful but costly on non-local features, while the Convolutional-based block is good at efficiently handling the local features. However, assembling two different Symmetry blocks will generate an Asymmetry block, making the classic Symmetry-block-based Super-Resolution (SR) architecture fail to deal with these Asymmetry blocks. In this paper, we proposed a new Dynamic fusion of Local and Non-local features-based Feedback Network (DLNFN) for SR, which focus on optimizing the traditional Symmetry-block-based SR architecture to hold two Symmetry blocks in parallel, making two Symmetry-blocks working on what they do best. (1) We introduce the Convolutional-based block for the local features and Attention-based network block for non-local features and propose the Delivery–Adjust–Fusion framework to hold these blocks. (2) we propose a Dynamic Weight block (DW block) which can generate different weight values to fuse the outputs on different feedback iterations. (3) We introduce the MAConv layer to optimize the In block, which is critical for our two blocks-based feedback algorithm. Experiments show our proposed DLNFN can take full advantage of two different blocks and outperform other state-of-the-art algorithms.

SR-FEINR: Continuous Remote Sensing Image Super-Resolution Using Feature-Enhanced Implicit Neural Representation

Remote sensing images often have limited resolution, which can hinder their effectiveness in various applications. Super-resolution techniques can enhance the resolution of remote sensing images, and arbitrary resolution super-resolution techniques provide additional flexibility in choosing appropriate image resolutions for different tasks. However, for subsequent processing, such as detection and classification, the resolution of the input image may vary greatly for different methods. In this paper, we propose a method for continuous remote sensing image super-resolution using feature-enhanced implicit neural representation (SR-FEINR). Continuous remote sensing image super-resolution means users can scale a low-resolution image into an image with arbitrary resolution. Our algorithm is composed of three main components: a low-resolution image feature extraction module, a positional encoding module, and a feature-enhanced multi-layer perceptron module. We are the first to apply implicit neural representation in a continuous remote sensing image super-resolution task. Through extensive experiments on two popular remote sensing image datasets, we have shown that our SR-FEINR outperforms the state-of-the-art algorithms in terms of accuracy. Our algorithm showed an average improvement of 0.05 dB over the existing method on ×30 across three datasets.

RSHAN: Image super-resolution network based on residual separation hybrid attention module

Transformer has become one of the main architectures in deep learning, showing impressive performance in various vision tasks, especially for image super-resolution (SR). However, due to the usage of high-resolution input images, most current Transformer-based image super-resolution models have a large number of parameters and high computational complexity. Moreover, some components employed in the Transformer may be redundant for SR tasks, which may limit the SR performance. In this work, we propose an efficient and concise model for image super-resolution tasks termed Residual Separation Hybrid Attention Network (RSHAN), which aims to solve the problems of redundant components and insufficient ability to extract high-frequency information of Transformer. Specifically, we present the Residual Separation Hybrid Attention Module (RSHAM) which fuses the local features extracted by the convolutional neural network (CNN) branch and the long-range dependencies extracted by Transformers to improve the performance of RSHAN. Extensive experiments on numerous benchmark datasets show that the proposed method outperforms state-of-the-art SR methods by up to 0.11 dB in peak signal-to-noise ratio (PSNR) metric, while the computational complexity and the inference time is reduced by 5% and 10%, respectively.

Color Polarization Image Super-Resolution Reconstruction via a Cross-Branch Supervised Learning Strategy

The division-of-focal-plane polarimeter, as a real-time polarization imaging technology, suffers from missing spatial resolution. To address this challenge, we propose a novel two-branch network for color polarization image super-resolution (CPSRNet), which provides a low-cost solution. Specifically, CPSRNet utilizes the degree of linear polarization (DoLP) image and intensity image (S0) as inputs instead of a conventional intensity image. The DoLP branch, as the high-frequency signal stream, supervises cross-branch feature activation in the shallow architecture and restores the polarization characteristic. CPSRNet, which mainly consists of a cross-branch activation module (CBAM) and a related-supervised feature residual fusion module (RSRFM), tends to improve the learning of high-frequency features both locally and globally. The proposed CBAM utilizes deeper features in the DoLP branch to activate the low-level high-frequency features in the intensity branch. Meanwhile, RSRFM introduces adjusted cosine similarity (ACOS) loss to fuse two-branch feature maps in a supervised-learning manner. ACOS loss exploits the intensity stream to encourage the fused feature stream to flow in the desired direction. In addition, we build a new dataset for the color polarization image super-resolution task. All the color polarization images are captured by a division-of-focal-plane polarization camera in real scenarios. We perform extensive experiments, including polarization image super-resolution and demosaicking, to confirm the superiority of our CPSRNet. The source code is available at https://github.com/yudadabing/CPSRNet.

AESR3D: 3D overcomplete autoencoder for trabecular computed tomography super resolution

AbstractOsteoporosis is a major cause of bone fracture and can be characterised by both mass loss and microstructure deterioration of the bone. The modern way of osteoporosis assessment is through the measurement of bone mineral density, which is not able to unveil the pathological condition from the mesoscale aspect. To obtain mesoscale information from computed tomography (CT), the super‐resolution (SR) approach for volumetric imaging data is required. A deep learning model AESR3D is proposed to recover high‐resolution (HR) Micro‐CT from low‐resolution Micro‐CT and implement an unsupervised segmentation for better trabecular observation and measurement. A new regularisation overcomplete autoencoder framework for the SR task is proposed and theoretically analysed. The best performance is achieved on structural similarity measure of trabecular CT SR task compared with the state‐of‐the‐art models in both natural and medical image SR tasks. The HR and SR images show a high correlation (r = 0.996, intraclass correlation coefficients = 0.917) on trabecular bone morphological indicators. The results also prove the effectiveness of our regularisation framework when training a large capacity model.

Lightweight image super-resolution based on deep learning: State-of-the-art and future directions

Recently, super-resolution (SR) techniques based on deep learning have taken more and more attention, aiming to improve the images and videos resolutions. Most of the SR methods are related to other fields of computer vision such as image classification, image segmentation, and object detection. Based on the success of the image SR task, many image SR surveys are introduced to summarize the recent work in the image SR domains. However, there is no survey to summarize the SR models for the lightweight image SR domain. In this paper, we present a comprehensive survey of the state-of-the-art lightweight SR models based on deep learning. The SR techniques are grouped into six major categories: include convolution, residual, dense, distillation, attention, and extremely lightweight based models. Also, we cover some other issues related to the SR task, such as benchmark datasets and metrics for performance evaluation. Finally, we discuss some future directions and open problems, that may help other community researchers in the future.

Dynamical Fusion Model With Joint Variational and Deep Priors for Hyperspectral Image Super-Resolution

In this paper, we propose a novel dynamic fusion model (DFM) with joint variational and deep priors for the task of hyperspectral image super-resolution (HISR). The given model can benefit from both the advantages of traditional modeling and deep learning methods, thus achieving significant improvements based on existing deep pre-trained models. Specifically, the given model mainly contains two new designed terms, i.e., the weighted spatial fidelity (WSF) term and the deep fusion (DF) term. The WSF term focuses on the spatial recovery of the low-resolution hyperspectral image through the high-resolution multispectral image without the knowledge of the spectral response matrix, thus the proposed DFM can be viewed as a semi-blind model for HISR. Moreover, the DF term relied upon deep fusion with a designed adaptive weight matrix, which can effectively inject the deep priors into the traditional minimization model. Besides, the proposed DFM can be quickly and effectively solved using the alternating direction method of multipliers. Experimental results on widely used datasets demonstrate the superiority of our approach compared with state-of-the-art HISR methods.

SCTANet: A Spatial Attention-Guided CNN-Transformer Aggregation Network for Deep Face Image Super-Resolution

Numerous CNN-based algorithms have been proposed to reconstruct high-quality face images. However, the inability of convolution operation to model long-distance relationships limits the performance of the CNN-based methods. Moreover, in the high-resolution (HR) image reconstruction stage, with the well decoded feature representations, more efficient architecture design can be explored to synthesize pixel-level image details. In this work, we propose a spatial attention-guided CNN-Transformer aggregation network (SCTANet) for face image super-resolution (FSR) tasks. The core component in the deep feature extraction stage is the Hybrid Attention Aggregation (HAA) block. The HAA block has two parallel paths, one for the Residual Spatial Attention (RSA) block, the other for the Multi-scale Patch embedding and Spatial-attention Masked Transformer (MPSMT) block. The HAA block combines the strengths of CNN and transformer to effectively exploit both local and global information. For the reconstruction stage, we propose to use the Sub-pixel MLP-based Upsampling (SMU) module instead of the conventional CNN architecture. The SMU module promotes the reconstruction of pixel-level image details and reduces computational complexity. Extensive experiments on both synthetic and real-world face datasets demonstrate the superiority of our proposed SCTANet over state-of-the-art methods.

Super-resolution reconstruction of binocular image based on multi-level fusion attention network

Super-resolution reconstruction of binocular image based on multi-level fusion attention network

Enhanced Autoencoders With Attention-Embedded Degradation Learning for Unsupervised Hyperspectral Image Super-Resolution

Recently, unmixing-based networks have shown significant potential in unsupervised multispectral-aided hyperspectral image super-resolution task (MS-aided HS-SR). Nevertheless, the representation ability of unsupervised networks and the design of loss functions still have not been fully explored, leaving large room for further improvement. To this end, we propose an enhanced unmixing-inspired unsupervised network with attention-embedded degradation learning, EU2ADL for short, to realize MS-aided HS-SR. First, two coupled autoencoders serve as the backbone of EU2ADL network to simultaneously decompose input modalities into abundances and corresponding endmembers, whose encoder part is composed of a spatial-spectral two-stream subnetwork for modality-salient representation learning and a parameter-shared one-stream subnetwork for modality-interacted representation enhancement. More importantly, a hybrid model-constrained loss containing a perceptual abundance term and a degradation-guided term is introduced to further eliminate the latent distortions. Since the hybrid loss is built on the degradation model, we additionally present an attention-embedded degradation learning network to adaptively estimate the unknown degradation parameters. Extensive experimental results on four datasets demonstrate the effectiveness of our proposed methods when compared with state-of-the-arts.

Learning Dynamic Generative Attention for Single Image Super-Resolution

Attention mechanisms have achieved great success for image super-resolution as they can effectively improve the feature representation ability. However, most attention-based methods produce the static attention weights, which are applied identically for all input samples. This popular attention strategy is difficult to automatically adapt the content variations of each individual input, hence hindering further improvements of the magnification performance. To explore towards resolving this challenge, we propose a variational hybrid network with newly dynamic attention mechanisms for image super-resolution tasks. Specifically, we design a multi-scale variational encoder network to transform the curvature map of an input image into the latent space. This is made possible for randomly generated latent variables to reflect the valuable high-frequency information and recalibrate the main network. We utilize these latent variables to further generate controllable attention weights, which modulate not only frequency parameters of convolutional kernels but also spatial characteristics of feature maps for boosting representation power. Moreover, a curvature-domain loss is designed to help the main network to concentrate more on high-frequency geometric structures. Experimental results have revealed that our method can generate more realistic and visually pleasing high-resolution images in comparison to state-of-the-art methods.

Lightweight image super-resolution with a feature-refined network

In recent years, deep Convolutional Neural Networks (CNNs) have achieved impressive successes on the Single Image Super-Resolution task (SISR). However, it remains difficult to apply these CNN-based SISR methods in embedded devices due to their high memory and computational requirements. To alleviate this issue, we focus on lightweight SISR methods. The observed similarity between the feature maps in CNNs serves as inspiration to explore the design of a cost-efficient module to obtain feature maps whose representation ability is roughly equivalent to that of a conventional convolutional layer. We thus propose a shadow module applying simple linear transformations with a lower cost to generate similar feature maps. Based on this module, we design a Feature-Refined Block (FRB) to learn more representative features. Besides, we propose a Global Dense Feature Fusion (GDFF) structure to construct a Feature-Refined Network (FRN) with such FRBs for lightweight SISR. Extensive experimental results demonstrate the superior performance of the proposed FRN in comparison with the state-of-the-art lightweight SISR methods, while consuming relatively low memory and computation resources.

Medical image super-resolution reconstruction via multi-scale information distillation network under multi-scale geometric transform domain

High resolution (HR) magnetic resonance images (MRI) or computed tomography (CT) images can provide clearer anatomical details of human body, which facilitates early diagnosis of the diseases. However, due to the imaging system, imaging environment and human factors, it is difficult to obtain clear high-resolution images. In this paper, we proposed a novel medical image super resolution (SR) reconstruction method via multi-scale information distillation (MSID) network in the non-subsampled shearlet transform (NSST) domain, namely NSST-MSID network. We first proposed a MSID network that mainly consisted of a series of stacked MSID blocks to fully exploit features from images and effectively restore the low resolution (LR) images to HR images. In addition, most previous methods predict the HR images in the spatial domain, producing over-smoothed outputs while losing texture details. Thus, we viewed the medical image SR task as the prediction of NSST coefficients, which make further MSID network keep richer structure details than that in spatial domain. Finally, the experimental results on our constructed medical image datasets demonstrated that the proposed method was capable of obtaining better peak signal to noise ratio (PSNR), structural similarity (SSIM) and root mean square error (RMSE) values and keeping global topological structure and local texture detail better than other outstanding methods, which achieves good medical image reconstruction effect.

A Transformer-Based Model for Super-Resolution of Anime Image.

Image super-resolution (ISR) technology aims to enhance resolution and improve image quality. It is widely applied to various real-world applications related to image processing, especially in medical images, while relatively little appliedto anime image production. Furthermore, contemporary ISR tools are often based on convolutional neural networks (CNNs), while few methods attempt to use transformers that perform well in other advanced vision tasks. We propose a so-called anime image super-resolution (AISR) method based on the Swin Transformer in this work. The work was carried out in several stages. First, a shallow feature extraction approach was employed to facilitate the features map of the input image's low-frequency information, which mainly approximates the distribution of detailed information in a spatial structure (shallow feature). Next, we applied deep feature extraction to extract the image semantic information (deep feature). Finally, the image reconstruction method combines shallow and deep features to upsample the feature size and performs sub-pixel convolution to obtain many feature map channels. The novelty of the proposal is the enhancement of the low-frequency information using a Gaussian filter and the introduction of different window sizes to replace the patch merging operations in the Swin Transformer. A high-quality anime dataset was constructed to curb the effects of the model robustness on the online regime. We trained our model on this dataset and tested the model quality. We implement anime image super-resolution tasks at different magnifications (2×, 4×, 8×). The results were compared numerically and graphically with those delivered by conventional convolutional neural network-based and transformer-based methods. We demonstrate the experiments numerically using standard peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), respectively. The series of experiments and ablation study showcase that our proposal outperforms others.

Multi-scale feature learning network with channel self-attention for remote sensing single-image super-resolution

ABSTRACT The performance of the remote sensing image super-resolution (SR) task has been significantly improved by applying deep learning. However, previously reported methods usually require sufficient synthetic datasets obtained by bicubic downsampling for long-term training, which can get better results when the input of the test phase is still the bicubic downsampling low-resolution (LR) image. However, when the input does not meet this bicubic downsampling situation, the performance of the model will be significantly degraded, and the generated SR image will be blurred. In order to better adapt to remote sensing image SR in different situations, this paper proposes a multi-scale feature learning network with channel self-attention for remote sensing single-image super-resolution (MSFLCSA). The proposed MSFLCSA does not need any extra synthetic paired dataset but one LR input image for training. Further, MSFLCSA uses a well-designed channel self-attention multi-scale feature learning network to fully learn the repetitive multi-scale features of the input image to realize remote sensing image SR tasks. Specifically, the proposed MSFLCSA extracts features of different levels through multi-column convolutions with different receptive fields to better learn the multi-scale features inside remote sensing images. In addition, the channel self-attention module constructs the channel dependence relationship between channels, selectively emphasizes the interdependent channel features, and further improves the learned multi-scale feature representation. Various types of qualitative and quantitative experiments have validated the effectiveness of MSFLCSA. Compared with advanced technologies, MSFLCSA has achieved superior performance.

Lightweight frequency-based attention network for image super-resolution

The advent of convolutional neural networks has been driving the rapid development of image super-resolution (SR) tasks. Existing works, however, tend to devise deeper and wider networks to boost accuracy, leading to huge model sizes and computation costs. In addition, they also ignore the effect of frequency domain information on image restoration. To address these challenges, we propose a simple and effective frequency-based attention network, comprising a series of frequency-domain enhancement modules (FDEMs), for accurate image SR. Each FDEM integrates a two-dimensional discrete wavelet transform, progressive frequency enhancement module (PFEM), and frequency aggregation module (FAM). Specifically, DWT first decomposes the features into high-frequency and low-frequency parts, and then the low-frequency branch is processed and gradually added to the high-frequency branch to realize the interaction of frequency information in PFEM. Finally, frequency representations from the PFEM are upsampled to the FAM to further achieve information enhancement in a specific space. Extensive experiments indicate that our method is efficient in reconstruction accuracy with less model capacity, exceeding most existing lightweight SR networks.

MS2Net: Multi-Scale and Multi-Stage Feature Fusion for Blurred Image Super-Resolution

At present, most mainstream algorithms for single image super-resolution (SISR) assume the image degradation process as an ideal degradation process (e.g. bicubic downscaling), which violates the actual degeneration conditions. In real-world image capturing, objects often move in a dynamic environment, and camera shake also often occurs, which results in serious blurs. Our work focuses on the task of image super-resolution with heavy motion blur, for which we adopt a network with two branches: one branch for image deblurring and the other one for super-resolution. Since the features obtained by the deblurring are rich in details, we apply their features as supplementary information to the super-resolution branch. Based on the adopted dual-branch framework, our major technical novelties lie in two novel modules: Multi-Scale Feature Fusion (MSFF1) module which fuses features of different scale from the deblurring branch to get local and global information, and Multi-Stage Feature Fusion (MSFF2) module which further filters useful information with attention. We evaluate the proposed method under various blur scenarios on the benchmark datasets, demonstrating competitive performance against existing methods.

BAM: a balanced attention mechanism to optimize single image super-resolution

Recovering texture information from the aliasing regions has always been a major challenge for single image super-resolution (SISR) task. These regions are often submerged in noise so that we have to restore texture details while suppressing noise. To address this issue, we propose an efficient Balanced Attention Mechanism (BAM), which consists of Avgpool Channel Attention Module (ACAM) and Maxpool Spatial Attention Module (MSAM) in parallel. ACAM is designed to suppress extreme noise in the large-scale feature maps, while MSAM preserves high-frequency texture details. Thanks to the parallel structure, these two modules not only conduct self-optimization, but also mutual optimization to obtain the balance of noise reduction and high-frequency texture restoration during the back propagation process, and the parallel structure makes the inference faster. To verify the effectiveness and robustness of BAM, we applied it to 10 state-of-the-art SISR networks. The results demonstrate that BAM can efficiently improve the networks' performance, and for those originally with attention mechanism, the substitution with BAM further reduces the amount of parameters and increases the inference speed. Information multi-distillation network (IMDN), a representative lightweight SISR network with attention, when the input image size is 200 × 200, the FPS of proposed IMDN-BAM precedes IMDN {8.1%, 8.7%, 8.8%} under the three SR magnifications of × 2, × 3, × 4, respectively. Densely residual Laplacian network (DRLN), a representative heavyweight SISR network with attention, when the scale is 60 × 60, the proposed DRLN-BAM is {11.0%, 8.8%, 10.1%} faster than DRLN under × 2, × 3, × 4. Moreover, we present a dataset with rich texture aliasing regions in real scenes, named realSR7. Experiments prove that BAM achieves better super-resolution results on the aliasing area.

CNN-Enhanced graph attention network for hyperspectral image super-resolution using non-local self-similarity

ABSTRACT The small-sample problem that widely existed in the hyperspectral image (HSI) super-resolution task will lead to insufficient feature extraction in network training. Therefore, it is necessary to design an effective network to extract the feature of HSIs fully. In addition, existing HSI super-resolution (SR) networks usually capture multiple receptive fields by staking massive convolutions, which will inevitably produce many parameters. In this paper, we propose a novel HSI SR network based on the convolution neural network enhanced graph attention network (CEGATSR), which can fully capture different features by using a graph attention block (GAB) and a depthwise separable convolution block (DSCB). Moreover, the graph attention block can also capture different receptive fields by using relatively few layers. Specifically, we first divide the whole spectral bands into several groups and extract the features separately for each group to reduce the parameters. Second, we design a parallel feature extraction unit to extract non-local and local features by combining the graph attention block (GAB) and the depthwise separable convolution block (DSCB). The graph attention block makes full use of the non-local self-similarity strategy not only to self-learn the effective information but also to capture the multiple receptive fields by using relatively few parameters. The depthwise separable convolution block is designed to extract the local feature information with few parameters. Third, we design a spatial-channel attention block (SCAB) to capture the global spatial-spectral features and to distinguish the importance of different channels. A large number of experiments on three hyperspectral datasets show that the proposed CEGATSR performs better than the state-of-the-art SR methods. The source code is available at [Online]. Available: https://github.com/Dongyx1128/CEGATSR.