Single Image Super-resolution Methods Research Articles

Efficient blind super-resolution imaging via adaptive degradation-aware estimation

Single image super-resolution (SISR) has achieved prominent success based on deep learning. However, most SISR methods based on the specific degradation pattern, e.g., bicubic interpolation with a bulky network structure, are unsuitable for real-world images reconstructed on edge devices. To mitigate the above limitations, unlike the majority of blind super-resolution methods, which consist of a separate complex degradation kernel projection network and a SR reconstruction network, we propose an end-to-end lightweight multi-degradation oriented network that not only can estimate degradation kernel but also reconstructs HR image fast and accurately. The core components of the designed network adopt a lightweight SR model with region non-local feature similarity learning, i.e., Region Non-Local Feature Block (RNLFB), to establish region-wise global feature correlation. Concretely, more and fewer RNLFBs are used to learn more sophisticated feature representation and relatively simple degradation representation. Then, a degradation kernel projector is equipped to fulfill adaptive degradation-aware estimation from the degradation representation. With the help of the encoded degradation kernel, the reconstruction model can learn to restore a high-resolution image from the feature representation. By doing so, the degradation kernel estimation task and the HR image reconstruction task can be accomplished without designing two complex networks separately, which guides more stable and effective network training under low computational resource costs. Extensive experiments on synthetic and real-world datasets demonstrate that our method can fulfill precise degradation kernel prediction and HR image reconstruction compared with other state-of-the-art SR methods with lower model complexity.

Deep multi-scale feature mixture model for image super-resolution with multiple-focal-length degradation

Single image super-resolution is a classic problem in computer vision. In recent years, deep learning-based models have achieved unprecedented success with this problem. However, most existing deep super-resolution models unavoidably produce degraded results when applied to real-world images captured by cameras with different focal lengths. The degradation in these images is called multiple-focal-length degradation, which is spatially variant and more complicated than the bicubic downsampling degradation. To address such a challenging issue, we propose a multi-scale feature mixture model in this paper. The proposed model can intensively exploit local patterns from different scales for image super-resolution. To improve the performance, we further propose a novel loss function based on the Laplacian pyramid, which guides the model to recover the information separately of different frequency subbands. Comprehensive experiments show that our proposed model has a better ability to preserve the structure of objects and generate high-quality images, leading to the best performance compared with other state-of-the-art deep single image super-resolution methods.

Single-Image Super-Resolution Method for Rotating Synthetic Aperture System Using Masking Mechanism

The emerging technology of rotating synthetic aperture (RSA) presents a promising solution for the development of lightweight, large-aperture, and high-resolution optical remote sensing systems in geostationary orbit. However, the rectangular shape of the primary mirror and the distinctive imaging mechanism involving the continuous rotation of the mirror lead to a pronounced decline in image resolution along the shorter side of the rectangle compared to the longer side. The resolution also exhibits periodic time-varying characteristics. To address these limitations and enhance image quality, we begin by analyzing the imaging mechanism of the RSA system. Subsequently, we propose a single-image super-resolution method that utilizes a rotated varied-size window attention mechanism instead of full attention, based on the Vision Transformer architecture. We employ a two-stage training methodology for the network, where we pre-train it on images masked with stripe-shaped masks along the shorter side of the rectangular pupil. Following that, we fine-tune the network using unmasked images. Through the strip-wise mask sampling strategy, this two-stage training approach effectively circumvents the interference of lower confidence (clarity) information and outperforms training the network from scratch using the unmasked degraded images. Our digital simulation and semi-physical imaging experiments demonstrate that the proposed method achieves satisfactory performance. This work establishes a valuable reference for future space applications of the RSA system.

A Systematic Survey of Deep Learning-based Single-Image Super-Resolution

Single-image super-resolution (SISR) is an important task in image processing, which aims to enhance the resolution of imaging systems. Recently, SISR has made a huge leap and has achieved promising results with the help of deep learning (DL). In this survey, we give an overview of DL-based SISR methods and group them according to their design targets. Specifically, we first introduce the problem definition, research background, and the significance of SISR. Secondly, we introduce some related works, including benchmark datasets, upsampling methods, optimization objectives, and image quality assessment methods. Thirdly, we provide a detailed investigation of SISR and give some domain-specific applications of it. Fourthly, we present the reconstruction results of some classic SISR methods to intuitively know their performance. Finally, we discuss some issues that still exist in SISR and summarize some new trends and future directions. This is an exhaustive survey of SISR, which can help researchers better understand SISR and inspire more exciting research in this field. An investigation project for SISR is provided at https://github.com/CV-JunchengLi/SISR-Survey.

Non-local degradation modeling for spatially adaptive single image super-resolution

Existing methods for single image super-resolution (SISR) model the blur kernel as spatially invariant across the entire image, and are susceptible to the adverse effects of textureless patches. To achieve improved results, adaptive estimation of the degradation kernel is necessary. We explore the synergy of joint global and local degradation modeling for spatially adaptive blind SISR. Our model, named spatially adaptive network for blind super-resolution (SASR), employs a simple encoder to estimate global degradation representations and a decoder to extract local degradation. These two representations are fused with a cross-attention mechanism and applied using spatially adaptive filtering to enhance the local image detail. Specifically, SASR contains two novel features: (1) a non-local degradation modeling with contrastive learning to learn global and local degradation representations, and (2) a non-local spatially adaptive filtering module (SAFM) that incorporates the global degradation and spatial-detail factors to preserve and enhance local details. We demonstrate that SASR can efficiently estimate degradation representations and handle multiple types of degradation. The local representations avoid the detrimental effect of estimating the entire super-resolved image with only one kernel through locally adaptive adjustments. Extensive experiments are performed to quantitatively and qualitatively demonstrate that SASR not only performs favorably for degradation estimation but also leads to state-of-the-art blind SISR performance when compared to alternative approaches.

Dual-space high-frequency learning for transformer-based MRI super-resolution

Background and ObjectiveMagnetic resonance imaging (MRI) can provide rich and detailed high-contrast information of soft tissues, while the scanning of MRI is time-consuming. To accelerate MR imaging, a variety of Transformer-based single image super-resolution methods are proposed in recent years, achieving promising results thanks to their superior capability of capturing long-range dependencies. Nevertheless, most existing works prioritize the design of transformer attention blocks to capture global information. The local high-frequency details, which are pivotal to faithful MRI restoration, are unfortunately neglected. MethodsIn this work, we propose a high-frequency enhanced learning scheme to effectively improve the awareness of high frequency information in current Transformer-based MRI single image super-resolution methods. Specifically, we present two entirely plug-and-play modules designed to equip Transformer-based networks with the ability to recover high-frequency details from dual spaces: 1) in the feature space, we design a high-frequency block (Hi-Fe block) paralleled with Transformer-based attention layers to extract rich high-frequency features; while 2) in the image intensity space, we tailor a high-frequency amplification module (HFA) to further refine the high-frequency details. By fully exploiting the merits of the two modules, our framework can recover abundant and diverse high-frequency information, rendering faithful MRI super-resolved results with fine details. ResultsWe integrated our modules with six Transformer-based models and conducted experiments across three datasets. The results indicate that our plug-and-play modules can enhance the super-resolution performance of all foundational models to varying degrees, surpassing the capabilities of existing state-of-the-art single image super-resolution networks. ConclusionComprehensive comparison of super-resolution images and high-frequency maps from various methods, clearly demonstrating that our module possesses the capability to restore high-frequency information, showing huge potential in clinical practice for accelerated MRI reconstruction.

Noise-Free Optimization in Early Training Steps for Image Super-resolution

Recent deep-learning-based single image super-resolution (SISR) methods have shown impressive performance whereas typical methods train their networks by minimizing the pixel-wise distance with respect to a given high-resolution (HR) image. However, despite the basic training scheme being the predominant choice, its use in the context of ill-posed inverse problems has not been thoroughly investigated. In this work, we aim to provide a better comprehension of the underlying constituent by decomposing target HR images into two subcomponents: (1) the optimal centroid which is the expectation over multiple potential HR images, and (2) the inherent noise defined as the residual between the HR image and the centroid. Our findings show that the current training scheme cannot capture the ill-posed nature of SISR and becomes vulnerable to the inherent noise term, especially during early training steps. To tackle this issue, we propose a novel optimization method that can effectively remove the inherent noise term in the early steps of vanilla training by estimating the optimal centroid and directly optimizing toward the estimation. Experimental results show that the proposed method can effectively enhance the stability of vanilla training, leading to overall performance gain. Codes are available at github.com/2minkyulee/ECO.

Uncertainty-Aware GAN for Single Image Super Resolution

Generative adversarial network (GAN) has become a popular tool in the perceptual-oriented single image super-resolution (SISR) for its excellent capability to hallucinate details. However, the performance of most GAN-based SISR methods is impeded due to the limited discriminative ability of their discriminators. In specific, these discriminators only focus on the global image reconstruction quality and ignore the more fine-grained reconstruction quality for constraining the generator, as they predict the overall realness of an image instead of the pixel-level realness. Here, we first introduce the uncertainty into the GAN and propose an Uncertainty-aware GAN (UGAN) to regularize SISR solutions, where the challenging pixels with large reconstruction uncertainty and importance (e.g., texture and edge) are prioritized for optimization. The uncertainty-aware adversarial training strategy enables the discriminator to capture the pixel-level SR uncertainty, which constrains the generator to focus on image areas with high reconstruction difficulty, meanwhile, it improves the interpretability of the SR. To balance weights of multiple training losses, we introduce an uncertainty-aware loss weighting strategy to adaptively learn the optimal loss weights. Extensive experiments demonstrate the effectiveness of our approach in extracting the SR uncertainty and the superiority of the UGAN over the state-of-the-arts in terms of the reconstruction accuracy and perceptual quality.

An interpretable lightweight deep network with [formula omitted] model-driven for single image super-resolution

In order to address the expensive computation cost of deep networks, some Single Image Super-Resolution (SISR) methods tried to design the lightweight networks by means of recursion or expert prior. However, they discuss the theoretical interpretation for the network’s design less. To address this issue, we propose a novel method for constructing an interpretable lightweight deep network for SISR by fusing the idea of model-driven and data-driven. That is, we give a theoretical interpretation for the lightweight network’s design from the optimization model of image degeneration. Considering that ℓp(0<p<1)-norm is sparser than the ℓ1-norm and can describe the noises of image degeneration better, our proposed SISR method firstly deduces an iteration algorithm from the ℓp(0<p<1) degeneration model. Then according to this theoretical deduction, an effective deep network is designed. Since our proposed deep network is designed according to the iteration algorithm, our network can not only realize the lightweight structure because of the weight sharing decided by the iterative principle, but also show a theoretical interpretation for designing the deep network. Extensive experimental results illustrate that our proposed method is superior to some related popular SISR methods with the lightweight structure.

Single image super-resolution with denoising diffusion GANS.

Single image super-resolution (SISR) refers to the reconstruction from the corresponding low-resolution (LR) image input to a high-resolution (HR) image. However, since a single low-resolution image corresponds to multiple high-resolution images, this is an ill-posed problem. In recent years, generative model-based SISR methods have outperformed conventional SISR methods in performance. However, the SISR methods based on GAN, VAE, and Flow have the problems of unstable training, low sampling quality, and expensive computational cost. These models also struggle to achieve the trifecta of diverse, high-quality, and fast sampling. In particular, denoising diffusion probabilistic models have shown impressive variety and high quality of samples, but their expensive sampling cost prevents them from being well applied in the real world. In this paper, we investigate the fundamental reason for the slow sampling speed of the SISR method based on the diffusion model lies in the Gaussian assumption used in the previous diffusion model, which is only applicable for small step sizes. We propose a new Single Image Super-Resolution with Denoising Diffusion GANS (SRDDGAN) to achieve large-step denoising, sample diversity, and training stability. Our approach combines denoising diffusion models with GANs to generate images conditionally, using a multimodal conditional GAN to model each denoising step. SRDDGAN outperforms existing diffusion model-based methods regarding PSNR and perceptual quality metrics, while the added latent variable Z solution explores the diversity of likely HR spatial domain. Notably, the SRDDGAN model infers nearly 11 times faster than diffusion-based SR3, making it a more practical solution for real-world applications.

Wavelet Pyramid Recurrent Structure-Preserving Attention Network for Single Image Super-Resolution.

Many single image super-resolution (SISR) methods that use convolutional neural networks (CNNs) learn the relationship between low-and high-resolution images directly, without considering the context structure and detail fidelity. This can limit the potential of CNNs and result in unrealistic, distorted edges and textures in the reconstructed images. A more effective approach is to incorporate prior knowledge about the image into the model to aid in image reconstruction. In this study, we propose a novel recurrent structure-preserving mechanism that innovatively uses the multiscale wavelet transform (WT) as an image prior, namely, wavelet pyramid recurrent structure-preserving attention network (WRSANet), to process both low-and high-frequency subnetworks at each level separately and recursively. We propose a novel structure scale preservation (SSP) architecture that differs from traditional WTs. This architecture allows us to incorporate and learn structure preservation subnetworks at each level. By using our proposed structure scale fusion (SSF) combined with inverse WT, we can recursively restore and preserve rich low-frequency image structure through the combination of SSP at various levels. Furthermore, we also propose novel low-to-high-frequency information transmission (L2HIT) and detail enhancement (DE) mechanisms to address the issue of detail distortion in high-frequency images by transferring information from structure preservation subnetworks. This allows us to preserve the low-frequency structure while reconstructing high-frequency details, improving detail fidelity and avoiding structural distortion. Finally, a joint loss function is also used to balance the fusion of low-and high-frequency information at different degrees, with hyperparameters being adjusted during training. The experimental results demonstrate that the proposed WRSANet achieves better performance and visual presentation than the state-of-the-art (SOTA) on synthetic and real datasets, especially in terms of context structure and texture details.

SSIR: Spatial shuffle multi-head self-attention for Single Image Super-Resolution

Benefiting from the development of deep convolutional neural networks, CNN-based single-image super-resolution methods have achieved remarkable reconstruction results. However, the limited perceptual field of the convolutional kernel and the use of static weights in the inference process limit the performance of CNN-based methods. Recently, a few vision transformer-based image super-resolution methods have achieved excellent performance compared to CNN-based methods. These methods contain many parameters and require vast amounts of GPU memory for training. In this paper, we propose a spatial shuffle multi-head self-attention for single-image super-resolution that can significantly model long-range pixel dependencies without additional computational consumption. A local perception module is also proposed to combine convolutional neural networks’ local connectivity and translational invariance. Reconstruction results on five popular benchmarks show that the proposed method outperforms existing methods in both reconstruction accuracy and visual performance. The proposed method matches the performance of transformed-based methods but requires an inferior number of transformer blocks, which reduces the number of parameters by 40%, GPU memory by 30%, and inference time by 30% compared to transformer-based methods.

MSISR-STF: Spatiotemporal Fusion via Multilevel Single-Image Super-Resolution

Due to technological limitations and budget constraints, spatiotemporal image fusion uses the complementarity of high temporal–low spatial resolution (HTLS) and high spatial–low temporal resolution (HSLT) data to obtain high temporal and spatial resolution (HTHS) fusion data, which can effectively satisfy the demand for HTHS data. However, some existing spatiotemporal image fusion models ignore the large difference in spatial resolution, which yields worse results for spatial information under the same conditions. Based on the flexible spatiotemporal data fusion (FSDAF) framework, this paper proposes a multilevel single-image super-resolution (SISR) method to solve this issue under the large difference in spatial resolution. The following are the advantages of the proposed method. First, multilevel super-resolution (SR) can effectively avoid the limitation of a single SR method for a large spatial resolution difference. In addition, the issue of noise accumulation caused by multilevel SR can be alleviated by learning-based SR (the cross-scale internal graph neural network (IGNN)) and then interpolation-based SR (the thin plate spline (TPS)). Finally, we add the reference information to the super-resolution, which can effectively control the noise generation. This method has been subjected to comprehensive experimentation using two authentic datasets, affirming that our proposed method surpasses the current state-of-the-art spatiotemporal image fusion methodologies in terms of performance and effectiveness.

CHALLENGES IN PREPARING DATASETS FOR SUPER-RESOLUTION ON THE EXAMPLE OF SENTINEL-2 AND PLANET SCOPE IMAGES

Abstract. Benchmark datasets is an significant aspect in in many areas such as computer vision, deep learning, geospatial data as they serve as standardized test sets for evaluating the performance of models. Among many techniques of image processing, there is super-resolution (SR) which is aimed at reconstructing a low-resolution (LR) image into a high-resolution (HR) image. For training and validation SR models as a dataset the pairs of HR and LR images are needed, which should be the same apart from resolution. There is a lot of benchmark datasets for super-resolution methods, but they usually include conventional photographs of an common objects, while remote sensing data have different characteristic in general. This paper focuses on the process of preparing datasets for super-resolution in satellite images, where high-resolution and low-resolution image data come from different sources. The case of the single-image super-resolution method was considered. The experiment was performed on Sentinel-2 and PlanetScope data, but the assumptions can also be transferred to data obtained from other satellites. The procedure on how to make the pairs of HR and LR images consistent in terms of time, location and spectral values was proposed. The impact of the processes carried out was measured using image similarity measurement methods such as PSNR, SSIM and SCC.

Characterization and experimental verification of the rotating synthetic aperture optical imaging system

The rotating synthetic aperture (RSA) optical imaging system employs a rectangular primary mirror for detection. During the imaging process, the primary mirror rotates around the center to achieve the aperture equivalent to the long side of the rectangle at different rotation angles. As a result, the system’s point spread function changes over time, causing periodic time-varying characteristics in the acquired images’ resolution. Moreover, due to the rectangular primary mirror, the images obtained by the RSA system are spatially asymmetric, with a lower resolution in the short side’s direction than in the long side’s direction. Hence, image processing techniques are necessary to enhance the image quality. To provide reference for the study of image quality improvement methods, we first characterize the imaging quality degradation mechanism of the RSA system and the time–space evolution law of the imaging process. We then establish an imaging experiment platform to simulate the dynamic imaging process of the RSA system. We quantify the RSA system’s impact on image degradation using objective indexes. Subsequently, by comparing the imaging experiment results with theoretical analysis, we verify the spatially asymmetric and temporally periodic imaging characteristics of the RSA system. Lastly, we introduce image super-resolution experiments to assess the limitations of directly applying generic deep learning-based single image super-resolution methods to the images captured by the RSA system, thereby revealing the challenges involved in improving image quality for the RSA system.

Spatial and Channel Aggregation Network for Lightweight Image Super-Resolution.

Advanced deep learning-based Single Image Super-Resolution (SISR) techniques are designed to restore high-frequency image details and enhance imaging resolution through the use of rapid and lightweight network architectures. Existing SISR methodologies face the challenge of striking a balance between performance and computational costs, which hinders the practical application of SISR methods. In response to this challenge, the present study introduces a lightweight network known as the Spatial and Channel Aggregation Network (SCAN), designed to excel in image super-resolution (SR) tasks. SCAN is the first SISR method to employ large-kernel convolutions combined with feature reduction operations. This design enables the network to focus more on challenging intermediate-level information extraction, leading to improved performance and efficiency of the network. Additionally, an innovative 9 × 9 large kernel convolution was introduced to further expand the receptive field. The proposed SCAN method outperforms state-of-the-art lightweight SISR methods on benchmark datasets with a 0.13 dB improvement in peak signal-to-noise ratio (PSNR) and a 0.0013 increase in structural similarity (SSIM). Moreover, on remote sensing datasets, SCAN achieves a 0.4 dB improvement in PSNR and a 0.0033 increase in SSIM.

A Two-Stage Three-Dimensional Attention Network for Lightweight Image Super-Resolution

In recent years, single image super-resolution (SISR) methods using convolutional neural networks (CNN) have achieved satisfactory performance. Nevertheless, the large model scale and the slow inference speed of these methods greatly limit the application scenarios. In this paper, we propose a two-stage three-dimensional attention network (ATTNet) for lightweight image super-resolution. First, we put forward the spatial feature encoder–decoder (SFE-D) with a spatial attention mechanism. Next, the channel transposed attention module (CTAM) with a channel self-attention mechanism is designed. Both the modules are used for fine feature extraction in the low-resolution stage. Finally, the content-based pixel recombination module (CPRM) is proposed to reconstruct the detailed content with a joint attention mechanism in the high-resolution stage. According to our experimental results, significant performance in terms of the quantitative metrics and the subjective visual quality can be achieved on average compared with the state-of-the-art lightweight SISR algorithms.

Two-Stage Deep Single-Image Super-Resolution With Multiple Blur Kernels for Internet of Things

Single image super-resolution (SISR) aims to reconstruct a high-resolution image from a single low-resolution (LR) image. Although convolutional neural network (CNN)-based SISR methods greatly enhance image restoration, they face critical challenges. First, SISR models using CNNs process image patches uniformly regardless of importance, causing spatial inefficiency in computation and representation. However, due to resource constraints, edge devices in the Internet of Things (IoT) cannot bear heavy computations or large memory storage. Second, most of the existing SISR methods are designed only for the widely adopted bicubic degradation and cannot handle LR images with arbitrary blur kernels, resulting in poor recovery performance. To address these issues, in this paper we propose a two-stage semantic and spatial deep super-resolution (SSDSR) model suitable for the IoT environment. The proposed SSDSR model is capable of handling a variety of blur kernels (e.g., isotropic Gaussian, motion and disk blur) by effectively using their prior information. Moreover, the Semantic Feature Extraction (SFE) module enables the proposed model to focus on key areas of LR images rather than treating all pixels equally, which significantly reduces the computational load. The semantic information from the SFE module and the spatial information from the Spatial Attention module are fused adaptively, allowing the proposed model to extract key information in LR images, thereby increasing the representation capacity of the CNN and improving image recovery. Compared with state-of-the-art SISR methods on benchmark datasets, the proposed SSDSR model demonstrates superior performance. When run in real time on an IoT edge device, our model exhibits high computational efficiency and excellent image quality.

SOFNet: Optical-flow based large-scale slice augmentation of brain MRI

Brain magnetic resonance imaging (MRI) data from multiple centers often exhibit variations in imaging conditions, such as the types of nuclear magnetic resonance instruments used and the presence of random noise. Additionally, discrepancies in the gap between MRI slices further complicate the usability of the data for advanced artificial intelligence (AI) analysis. Deep learning-based methods have emerged as practical solutions to address the challenge. However, existing research has largely overlooked the augmentation of brain MRI data, particularly when confronted with significant slice gaps, such as around 6 mm observed in our clinical brain MRI slices. In response to this research gap, we aim to develop novel approaches for augmenting brain MRI data, focusing on more significant slice gaps. To achieve this, we propose SOFNet, which utilizes the optical flow-based and encoder–decoder backbone. The primary objective of our model is to interpolate MRI slices while preserving feature consistency. Leveraging the optical flow method, which has exhibited exceptional performance compared to other super-resolution algorithms, our proposed approach has been evaluated on three distinct brain MRI datasets, explicitly addressing the gap between 4.2 mm and 6.0 mm. The experimental results highlight the significant enhancement in super-resolution quality achieved by SOFNet in generating adapted brain MRI data, surpassing other single-image super-resolution (SISR) methods in feature completion. To ensure the credibility of the interpolated brain MRI slices, we conducted experiments on three axes of MRI based on metrics such as peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). These experiments demonstrate the effectiveness of our approach in transforming low-resolution MRI data into clear and reliable brain MRIs, thereby enabling improved analysis using AI techniques.

Hybrid knowledge distillation from intermediate layers for efficient Single Image Super-Resolution

Convolutional and Transformer models have achieved remarkable results for Single Image Super-Resolution (SISR). However, the tremendous memory and computation consumption of these models restricts their usage in resource-limited scenarios. Knowledge distillation, as an effective model compression technique, has received great research focus on the SISR task. In this paper, we propose a novel efficient SISR method via hybrid knowledge distillation from intermediate layers, termed HKDSR, which leverages the knowledge from frequency information into that RGB information. To accomplish this, we first pre-train the teacher with multiple intermediate upsampling layers to generate the intermediate SR outputs. We then construct two kinds of intermediate knowledge from the Frequency Similarity Matrix (FSM) and Adaptive Channel Fusion (ACF). FSM aims to mine the relationship of frequency similarity between the Ground-truth (GT) HR image, and the intermediate SR outputs of teacher and student by Discrete Wavelet Transformation. ACF merges the intermediate SR output of the teacher and GT HR image in a channel dimension to adaptively align the intermediate SR output of the student. Finally, we leverage the knowledge from FSM and ACF into reconstruction loss to effectively improve student performance. Extensive experiments demonstrate the effectiveness of HKDSR on different benchmark datasets and network architectures.