Single Image Super-resolution Methods Research Articles

This Microtubule Does Not Exist: Super-Resolution Microscopy Image Generation by a Diffusion Model.

Generative models, such as diffusion models, have made significant advancements in recent years, enabling the synthesis of high-quality realistic data across various domains. Here, the adaptation and training of a diffusion model on super-resolution microscopy images are explored. It is shown that the generated images resemble experimental images, and that the generation process does not exhibit a large degree of memorization from existing images in the training set. To demonstrate the usefulness of the generative model for data augmentation, the performance of a deep learning-based single-image super-resolution (SISR) method trained using generated high-resolution data is compared against training using experimental images alone, or images generated by mathematical modeling. Using a few experimental images, the reconstruction quality and the spatial resolution of the reconstructed images are improved, showcasing the potential of diffusion model image generation for overcoming the limitations accompanying the collection and annotation of microscopy images. Finally, the pipeline is made publicly available, runnable online, and user-friendly to enable researchers to generate their own synthetic microscopy data. This work demonstrates the potential contribution of generative diffusion models for microscopy tasks and paves the way for their future application in this field.

When guided diffusion model meets zero-shot image super-resolution

Existing deep learning-based single-image super-resolution (SR) methods typically rely on vast quantities of paired data. As an essential solution, zero-shot SR methods require only a single image to handle image-specific degradation. However, these methods still struggle to recover fine-grained details due to the lack of supervised information. In this work, we propose a novel guided Diffusion model for Zero-shot image SR (ZeroDiff) to explicitly direct image quality enhancement. Specifically, we elaborate two key guidance strategies: (1) high-frequency guidance and (2) content-consistent guidance. The former concentrates on boosting fine-grained textures by embedding high-frequency information into the cross-attention mechanism of the noise estimator. The latter avoids the sampling deviating from the original image in terms of structure and low-frequency content. Specifically, the noisy images at each diffusion step are injected into the corresponding sampling step, encouraging the sampled image to be consistent with that of the corresponding diffusion step. Moreover, we design a progressive zoom-in paradigm by gradually enlarging the image size and enriching the image details to boost the sampling efficiency of diffusion models, while enabling high-quality image reconstruction. Extensive experiments reveal that our method achieves comparable results with other state-of-the-art methods in quantitative and qualitative evaluations on both face and natural images from synthetic and real-world datasets.

Deep blind super-resolution for hyperspectral images

Current deep learning methods for single hyperspectral image super-resolution are non-blind ones, which adopt the simplistic bicubic degradation model. These models have poor generalization performance and cannot handle unknown degradation. Additionally, blind super-resolution methods for RGB images neglect rich spectral information in hyperspectral images, which causes spectral distortion in results. To address these issues, we consider the degeneration estimation and propose a single hyperspectral image blind super-resolution algorithm. Specifically, we first use a blur kernel estimation network and a deblurring network to obtain the image without blur. We change the receptive field by exchanging spatial channel information when estimating the kernel, thereby obtaining blurry information at different scales. In addition, to reduce the errors introduced by kernel mismatch and the relevant information among bands ignored by the group convolution, we fuse the spatial information of the blurred image to compensate for the errors and then utilize the spatial and spectral information of other bands to guide the feature extraction of the current band. Finally, we integrate the feature information from different branches in a global fusion network to improve spatial and spectral information fidelity further. Extensive experiments on synthetic and real-world hyperspectral datasets unequivocally demonstrate that our approach outperforms other state-of-the-art approaches in terms of both quantitative and qualitative evaluations. Our code is publicly available at https://github.com/YoungP2001/DBSR.

Unsupervised deep learning method for single image super-resolution of the thick pinhole imaging system using deep image prior

Thick pinhole imaging system is widely used for diagnosing intense pulsed radiation sources. However, owing to the trade-off among spatial resolution, field of view (FOV) and signal-to-noise ratio (SNR), the imaging system normally falls short in achieving high-precision spatial diagnosis. In this paper, we propose an unsupervised deep learning method for single image super-resolution (SISR) of the thick pinhole imaging system. The point spread function (PSF) of the imaging system is obtained by analytical calculation and Monte Carlo simulation methods, and the mathematical model of the imaging system is established using a linear equation. To solve the ill-posed inverse problem, we adopt randomly initialized deep convolutional neural networks (DCNNs) as an image prior without pre-training, which is named deep image prior (DIP). The results demonstrate that, by utilizing the SISR technique to increase the number of pixels in reconstructed images, the proposed DIP algorithm can mitigate the spatial resolution degradation caused by an insufficient spatial sampling frequency of the camera. Compared with various classical algorithms, the proposed DIP algorithm exhibits superior capabilities in recovering high-frequency signals and suppressing ringing artifacts. Furthermore, the convergence and robustness of the proposed DIP algorithm under different random seeds and SNR conditions are also verified.

MambaSR: Arbitrary-Scale Super-Resolution Integrating Mamba with Fast Fourier Convolution Blocks

Traditional single image super-resolution (SISR) methods, which focus on integer scale super-resolution, often require separate training for each scale factor, leading to increased computational resource consumption. In this paper, we propose MambaSR, a novel arbitrary-scale super-resolution approach integrating Mamba with Fast Fourier Convolution Blocks. MambaSR leverages the strengths of the Mamba state-space model to extract long-range dependencies. In addition, Fast Fourier Convolution Blocks are proposed to capture the global information in the frequency domain. The experimental results demonstrate that MambaSR achieves superior performance compared to different methods across various benchmark datasets. Specifically, on the Urban100 dataset, MambaSR outperforms MetaSR by 0.93 dB in PSNR and 0.0203 dB in SSIM, and on the Manga109 dataset, it achieves an average PSNR improvement of 1.00 dB and an SSIM improvement of 0.0093 dB. These results highlight the efficacy of MambaSR in enhancing image quality for arbitrary-scale super-resolution.

Multi-directional feature fusion super-resolution network based on nonlinear spiking neural P systems

Single-image super-resolution (SISR) aims to recover lost details and textures from limited information to improve the visual quality and detail clarity of an image. Most of the existing convolutional neural network (CNN)-based SISR methods do not make full use of the intermediate features of the network, and at the same time its processing process can result in the loss of information details. To address these problems, this paper recommends a novel multi-directional feature fusion super-resolution network. The direction-aware mechanism with nonlinear spiking neural P (NSNP) systems is integrated to design a multidirectional feature extraction module, while different NSNP-like convolution modules are constructed for extracting deep features on the basis of nonlinear spiking mechanism. Moreover, in the residual feature enhancement module, three different branches are designed using these NSNP-like convolution modules for feature enhancement and fusion to decrease the loss of information. The preposed network is evaluated on five benchmark datasets, and the experimental results show that the network performs a better reconstruction with good capability of generalization compared to most state-of-the-art methods.

REMA: A Rich Elastic Mixed Attention Module for Single Image Super-Resolution.

Detail preservation is a major challenge for single image super-resolution (SISR). Many deep learning-based SISR methods focus on lightweight network design, but these may fall short in real-world scenarios where performance is prioritized over network size. To address these problems, we propose a novel plug-and-play attention module, rich elastic mixed attention (REMA), for SISR. REMA comprises the rich spatial attention module (RSAM) and the rich channel attention module (RCAM), both built on Rich Structure. Based on the results of our research on the module's structure, size, performance, and compatibility, Rich Structure is proposed to enhance REMA's adaptability to varying input complexities and task requirements. RSAM learns the mutual dependencies of multiple LR-HR pairs and multi-scale features, while RCAM accentuates key features through interactive learning, effectively addressing detail loss. Extensive experiments demonstrate that REMA significantly improves performance and compatibility in SR networks compared to other attention modules. The REMA-based SR network (REMA-SRNet) outperforms comparative algorithms in both visual effects and objective evaluation quality. Additionally, we find that module compatibility correlates with cardinality and in-branch feature bandwidth, and that networks with high effective parameter counts exhibit enhanced robustness across various datasets and scale factors in SISR.

Towards real world stereo image super-resolution via hybrid degradation model and discriminator for implied stereo image information

Real-world stereo image super-resolution has a significant influence on enhancing the performance of computer vision systems. Although existing methods for single-image super-resolution can be applied to enhance stereo images, these methods often introduce notable modifications to the inherent disparity, resulting in a loss in the consistency of disparity between the original and the enhanced stereo images. To overcome this limitation, this paper proposes a novel approach that integrates an implicit stereo information discriminator and a hybrid degradation model. This combination ensures effective enhancement while preserving disparity consistency. The proposed method bridges the gap between the complex degradations in real-world stereo domain and the simpler degradations in real-world single-image super-resolution domain. Our results demonstrate impressive performance on synthetic and real datasets, enhancing visual perception while maintaining disparity consistency.

Meta-learning based blind image super-resolution approach to different degradations

Although recent studies on blind single image super-resolution (SISR) have achieved significant success, most of them typically require supervised training on synthetic low resolution (LR)-high resolution (HR) paired images. This leads to re-training necessity for different degradations and restricted applications in real-world scenarios with unfavorable inputs. In this paper, we propose an unsupervised blind SISR method with input underlying different degradations, named different degradations blind super-resolution (DDSR). It formulates a Gaussian modeling on blur degradation and employs a meta-learning framework for solving different image degradations. Specifically, a neural network-based kernel generator is optimized by learning from random kernel samples, referred to as random kernel learning. This operation provides effective initialization for blur degradation optimization. At the same time, a meta-learning framework is proposed to resolve multiple degradation modelings on the basis of alternative optimization between blur degradation and image restoration, respectively. Differing from the pre-trained deep-learning methods, the proposed DDSR is implemented in a plug-and-play manner, and is capable of restoring HR image from unfavorable LR input with degradations such as partial coverage, noise addition, and darkening. Extensive simulations illustrate the superior performance of the proposed DDSR approach compared to the state-of-the-arts on public datasets with comparable memory load and time consumption, yet exhibiting better application flexibility and convenience, and significantly better generalization ability towards multiple degradations. Our code is available at https://github.com/XYLGroup/DDSR.

Super-resolution reconstruction of single image for latent features

Single-image super-resolution (SISR) typically focuses on restoring various degraded low-resolution (LR) images to a single high-resolution (HR) image. However, during SISR tasks, it is often challenging for models to simultaneously maintain high quality and rapid sampling while preserving diversity in details and texture features. This challenge can lead to issues such as model collapse, lack of rich details and texture features in the reconstructed HR images, and excessive time consumption for model sampling. To address these problems, this paper proposes a Latent Feature-oriented Diffusion Probability Model (LDDPM). First, we designed a conditional encoder capable of effectively encoding LR images, reducing the solution space for model image reconstruction and thereby improving the quality of the reconstructed images. We then employed a normalized flow and multimodal adversarial training, learning from complex multimodal distributions, to model the denoising distribution. Doing so boosts the generative modeling capabilities within a minimal number of sampling steps. Experimental comparisons of our proposed model with existing SISR methods on mainstream datasets demonstrate that our model reconstructs more realistic HR images and achieves better performance on multiple evaluation metrics, providing a fresh perspective for tackling SISR tasks.

Efficient blind super-resolution imaging via adaptive degradation-aware estimation

Single image super-resolution (SISR) has achieved prominent success based on deep learning. However, most SISR methods based on the specific degradation pattern, e.g., bicubic interpolation with a bulky network structure, are unsuitable for real-world images reconstructed on edge devices. To mitigate the above limitations, unlike the majority of blind super-resolution methods, which consist of a separate complex degradation kernel projection network and a SR reconstruction network, we propose an end-to-end lightweight multi-degradation oriented network that not only can estimate degradation kernel but also reconstructs HR image fast and accurately. The core components of the designed network adopt a lightweight SR model with region non-local feature similarity learning, i.e., Region Non-Local Feature Block (RNLFB), to establish region-wise global feature correlation. Concretely, more and fewer RNLFBs are used to learn more sophisticated feature representation and relatively simple degradation representation. Then, a degradation kernel projector is equipped to fulfill adaptive degradation-aware estimation from the degradation representation. With the help of the encoded degradation kernel, the reconstruction model can learn to restore a high-resolution image from the feature representation. By doing so, the degradation kernel estimation task and the HR image reconstruction task can be accomplished without designing two complex networks separately, which guides more stable and effective network training under low computational resource costs. Extensive experiments on synthetic and real-world datasets demonstrate that our method can fulfill precise degradation kernel prediction and HR image reconstruction compared with other state-of-the-art SR methods with lower model complexity.

A Systematic Survey of Deep Learning-Based Single-Image Super-Resolution

Single-image super-resolution (SISR) is an important task in image processing, which aims to enhance the resolution of imaging systems. Recently, SISR has made a huge leap and has achieved promising results with the help of deep learning (DL). In this survey, we give an overview of DL-based SISR methods and group them according to their design targets. Specifically, we first introduce the problem definition, research background, and the significance of SISR. Secondly, we introduce some related works, including benchmark datasets, upsampling methods, optimization objectives, and image quality assessment methods. Thirdly, we provide a detailed investigation of SISR and give some domain-specific applications of it. Fourthly, we present the reconstruction results of some classic SISR methods to intuitively know their performance. Finally, we discuss some issues that still exist in SISR and summarize some new trends and future directions. This is an exhaustive survey of SISR, which can help researchers better understand SISR and inspire more exciting research in this field. An investigation project for SISR is provided at https://github.com/CV-JunchengLi/SISR-Survey.

Deep multi-scale feature mixture model for image super-resolution with multiple-focal-length degradation

Single image super-resolution is a classic problem in computer vision. In recent years, deep learning-based models have achieved unprecedented success with this problem. However, most existing deep super-resolution models unavoidably produce degraded results when applied to real-world images captured by cameras with different focal lengths. The degradation in these images is called multiple-focal-length degradation, which is spatially variant and more complicated than the bicubic downsampling degradation. To address such a challenging issue, we propose a multi-scale feature mixture model in this paper. The proposed model can intensively exploit local patterns from different scales for image super-resolution. To improve the performance, we further propose a novel loss function based on the Laplacian pyramid, which guides the model to recover the information separately of different frequency subbands. Comprehensive experiments show that our proposed model has a better ability to preserve the structure of objects and generate high-quality images, leading to the best performance compared with other state-of-the-art deep single image super-resolution methods.

Single-Image Super-Resolution Method for Rotating Synthetic Aperture System Using Masking Mechanism

The emerging technology of rotating synthetic aperture (RSA) presents a promising solution for the development of lightweight, large-aperture, and high-resolution optical remote sensing systems in geostationary orbit. However, the rectangular shape of the primary mirror and the distinctive imaging mechanism involving the continuous rotation of the mirror lead to a pronounced decline in image resolution along the shorter side of the rectangle compared to the longer side. The resolution also exhibits periodic time-varying characteristics. To address these limitations and enhance image quality, we begin by analyzing the imaging mechanism of the RSA system. Subsequently, we propose a single-image super-resolution method that utilizes a rotated varied-size window attention mechanism instead of full attention, based on the Vision Transformer architecture. We employ a two-stage training methodology for the network, where we pre-train it on images masked with stripe-shaped masks along the shorter side of the rectangular pupil. Following that, we fine-tune the network using unmasked images. Through the strip-wise mask sampling strategy, this two-stage training approach effectively circumvents the interference of lower confidence (clarity) information and outperforms training the network from scratch using the unmasked degraded images. Our digital simulation and semi-physical imaging experiments demonstrate that the proposed method achieves satisfactory performance. This work establishes a valuable reference for future space applications of the RSA system.

Non-local degradation modeling for spatially adaptive single image super-resolution

Existing methods for single image super-resolution (SISR) model the blur kernel as spatially invariant across the entire image, and are susceptible to the adverse effects of textureless patches. To achieve improved results, adaptive estimation of the degradation kernel is necessary. We explore the synergy of joint global and local degradation modeling for spatially adaptive blind SISR. Our model, named spatially adaptive network for blind super-resolution (SASR), employs a simple encoder to estimate global degradation representations and a decoder to extract local degradation. These two representations are fused with a cross-attention mechanism and applied using spatially adaptive filtering to enhance the local image detail. Specifically, SASR contains two novel features: (1) a non-local degradation modeling with contrastive learning to learn global and local degradation representations, and (2) a non-local spatially adaptive filtering module (SAFM) that incorporates the global degradation and spatial-detail factors to preserve and enhance local details. We demonstrate that SASR can efficiently estimate degradation representations and handle multiple types of degradation. The local representations avoid the detrimental effect of estimating the entire super-resolved image with only one kernel through locally adaptive adjustments. Extensive experiments are performed to quantitatively and qualitatively demonstrate that SASR not only performs favorably for degradation estimation but also leads to state-of-the-art blind SISR performance when compared to alternative approaches.

Dual-space high-frequency learning for transformer-based MRI super-resolution

Background and ObjectiveMagnetic resonance imaging (MRI) can provide rich and detailed high-contrast information of soft tissues, while the scanning of MRI is time-consuming. To accelerate MR imaging, a variety of Transformer-based single image super-resolution methods are proposed in recent years, achieving promising results thanks to their superior capability of capturing long-range dependencies. Nevertheless, most existing works prioritize the design of transformer attention blocks to capture global information. The local high-frequency details, which are pivotal to faithful MRI restoration, are unfortunately neglected. MethodsIn this work, we propose a high-frequency enhanced learning scheme to effectively improve the awareness of high frequency information in current Transformer-based MRI single image super-resolution methods. Specifically, we present two entirely plug-and-play modules designed to equip Transformer-based networks with the ability to recover high-frequency details from dual spaces: 1) in the feature space, we design a high-frequency block (Hi-Fe block) paralleled with Transformer-based attention layers to extract rich high-frequency features; while 2) in the image intensity space, we tailor a high-frequency amplification module (HFA) to further refine the high-frequency details. By fully exploiting the merits of the two modules, our framework can recover abundant and diverse high-frequency information, rendering faithful MRI super-resolved results with fine details. ResultsWe integrated our modules with six Transformer-based models and conducted experiments across three datasets. The results indicate that our plug-and-play modules can enhance the super-resolution performance of all foundational models to varying degrees, surpassing the capabilities of existing state-of-the-art single image super-resolution networks. ConclusionComprehensive comparison of super-resolution images and high-frequency maps from various methods, clearly demonstrating that our module possesses the capability to restore high-frequency information, showing huge potential in clinical practice for accelerated MRI reconstruction.

Noise-Free Optimization in Early Training Steps for Image Super-resolution

Recent deep-learning-based single image super-resolution (SISR) methods have shown impressive performance whereas typical methods train their networks by minimizing the pixel-wise distance with respect to a given high-resolution (HR) image. However, despite the basic training scheme being the predominant choice, its use in the context of ill-posed inverse problems has not been thoroughly investigated. In this work, we aim to provide a better comprehension of the underlying constituent by decomposing target HR images into two subcomponents: (1) the optimal centroid which is the expectation over multiple potential HR images, and (2) the inherent noise defined as the residual between the HR image and the centroid. Our findings show that the current training scheme cannot capture the ill-posed nature of SISR and becomes vulnerable to the inherent noise term, especially during early training steps. To tackle this issue, we propose a novel optimization method that can effectively remove the inherent noise term in the early steps of vanilla training by estimating the optimal centroid and directly optimizing toward the estimation. Experimental results show that the proposed method can effectively enhance the stability of vanilla training, leading to overall performance gain. Codes are available at github.com/2minkyulee/ECO.

Uncertainty-Aware GAN for Single Image Super Resolution

Generative adversarial network (GAN) has become a popular tool in the perceptual-oriented single image super-resolution (SISR) for its excellent capability to hallucinate details. However, the performance of most GAN-based SISR methods is impeded due to the limited discriminative ability of their discriminators. In specific, these discriminators only focus on the global image reconstruction quality and ignore the more fine-grained reconstruction quality for constraining the generator, as they predict the overall realness of an image instead of the pixel-level realness. Here, we first introduce the uncertainty into the GAN and propose an Uncertainty-aware GAN (UGAN) to regularize SISR solutions, where the challenging pixels with large reconstruction uncertainty and importance (e.g., texture and edge) are prioritized for optimization. The uncertainty-aware adversarial training strategy enables the discriminator to capture the pixel-level SR uncertainty, which constrains the generator to focus on image areas with high reconstruction difficulty, meanwhile, it improves the interpretability of the SR. To balance weights of multiple training losses, we introduce an uncertainty-aware loss weighting strategy to adaptively learn the optimal loss weights. Extensive experiments demonstrate the effectiveness of our approach in extracting the SR uncertainty and the superiority of the UGAN over the state-of-the-arts in terms of the reconstruction accuracy and perceptual quality.

An interpretable lightweight deep network with [formula omitted] model-driven for single image super-resolution

In order to address the expensive computation cost of deep networks, some Single Image Super-Resolution (SISR) methods tried to design the lightweight networks by means of recursion or expert prior. However, they discuss the theoretical interpretation for the network’s design less. To address this issue, we propose a novel method for constructing an interpretable lightweight deep network for SISR by fusing the idea of model-driven and data-driven. That is, we give a theoretical interpretation for the lightweight network’s design from the optimization model of image degeneration. Considering that ℓp(0<p<1)-norm is sparser than the ℓ1-norm and can describe the noises of image degeneration better, our proposed SISR method firstly deduces an iteration algorithm from the ℓp(0<p<1) degeneration model. Then according to this theoretical deduction, an effective deep network is designed. Since our proposed deep network is designed according to the iteration algorithm, our network can not only realize the lightweight structure because of the weight sharing decided by the iterative principle, but also show a theoretical interpretation for designing the deep network. Extensive experimental results illustrate that our proposed method is superior to some related popular SISR methods with the lightweight structure.

Single image super-resolution with denoising diffusion GANS

Single image super-resolution (SISR) refers to the reconstruction from the corresponding low-resolution (LR) image input to a high-resolution (HR) image. However, since a single low-resolution image corresponds to multiple high-resolution images, this is an ill-posed problem. In recent years, generative model-based SISR methods have outperformed conventional SISR methods in performance. However, the SISR methods based on GAN, VAE, and Flow have the problems of unstable training, low sampling quality, and expensive computational cost. These models also struggle to achieve the trifecta of diverse, high-quality, and fast sampling. In particular, denoising diffusion probabilistic models have shown impressive variety and high quality of samples, but their expensive sampling cost prevents them from being well applied in the real world. In this paper, we investigate the fundamental reason for the slow sampling speed of the SISR method based on the diffusion model lies in the Gaussian assumption used in the previous diffusion model, which is only applicable for small step sizes. We propose a new Single Image Super-Resolution with Denoising Diffusion GANS (SRDDGAN) to achieve large-step denoising, sample diversity, and training stability. Our approach combines denoising diffusion models with GANs to generate images conditionally, using a multimodal conditional GAN to model each denoising step. SRDDGAN outperforms existing diffusion model-based methods regarding PSNR and perceptual quality metrics, while the added latent variable Z solution explores the diversity of likely HR spatial domain. Notably, the SRDDGAN model infers nearly 11 times faster than diffusion-based SR3, making it a more practical solution for real-world applications.