Blind Super-resolution Research Articles

Blind Super-Resolution Network with Dual-Channel Attention for Images Captured by Sub-Millimeter-Diameter Fiberscope

A blind super-resolution network with dual-channel attention is proposed for images captured by the 0.37 mm diameter sub-millimeter fiberscope. The fiberscope can used in scenarios where other image acquisition devices cannot be applied based on its flexible, soft, and minimally invasive characteristics. However, the images have black reticulated noise and only 3000 pixels. To improve image quality, the Butterworth band-stop filter is used to reduce the frequency of the reticulated noise. By optimizing the blind super-resolution model, high-quality images can be reconstructed that do not require a lot of synthetic paired fiberscope image data. Perceptual loss is utilized as a loss function, and channel and spatial attention mechanisms are introduced to the model to enhance the high-frequency detail information of the reconstructed image. In the comparative experiment with other methods, our method showed improvements of 2.25 in peak signal-to-noise ratio (PSNR) and 0.09 in structural similarity (SSIM) based on objective evaluation metrics. The learned perceptual image patch similarity (LPIPS) based on learning was reduced by 0.6. Furthermore, four different methods were used to enhance the resolution of the fiberscope images by a factor of four. The results of this paper improve the information entropy and Laplace clarity by 0.44 and 2.54, respectively, compared to the average of other methods. Validation results show that the approach in this paper is more applicable to sub-millimeter-diameter fiberscopes.

Dynamic learnable degradation for blind super-resolution

Blind super-resolution endeavors to restore clear and high-resolution images from obscure degraded images. To comprehensively encompass the diverse range of real image degradations, prevailing research methods either explicitly or implicitly simulate image degradation. Nevertheless, a majority of prior approaches employ conventional interpolation algorithms for resize operations, thereby impeding the breadth of image degradation diversity. In the case of implicit models, we propose a dynamic learnable degradation (DLD) model, achieved through self-supervised training. This model leverages a dynamic downsampling module to generate a multitude of low-resolution images. By introducing DLD as a novel downsampling method, we devise an original explicit degradation model, which generates additional sets of training image pairs to enhance the practical restoration outcomes of ESRGAN for natural images, called Dynamic-ESRGAN. We also account for the overall degradation and put forth a hybrid-order degradation model to encompass the manifold degradations prevalent in natural images. Exhaustive experimental comparisons conducted on both synthetic and real datasets have unequivocally demonstrated the significant advantages of our DLD and Dynamic-ESRGAN models, both in terms of objective indicators and subjective perception.

Blind super-resolution model based on unsupervised degenerate indication learning

Blind super-resolution of images based on contrast learning has achieved better performance, which can distinguish between different degraded information. However, there is no significant improvement in the processing of image super-resolution. To further improve the performance of image blind super-resolution, a new degenerate higher-order attention network (DHAN) is designed for image super-resolution, in which a higher-order channel attention mechanism is introduced to the framework. Meanwhile, it also introduces normalized variance and global covariance pooling, which can provide higher-order feature information and more discriminative feature information. Comparing the experimental results, it can be found that the proposed higher-order attention network (DHAN) is able to distinguish various complex degradation patterns and thus obtain accurate degradation information. Numerous experiments have shown that higher-order attention network (DHAN) achieves good visual results in the image blind super-resolution task.

End-to-End Alternating Optimization for Real-World Blind Super Resolution

End-to-End Alternating Optimization for Real-World Blind Super Resolution

Lightweight Implicit Blur Kernel Estimation Network for Blind Image Super-Resolution

Blind image super-resolution (Blind-SR) is the process of leveraging a low-resolution (LR) image, with unknown degradation, to generate its high-resolution (HR) version. Most of the existing blind SR techniques use a degradation estimator network to explicitly estimate the blur kernel to guide the SR network with the supervision of ground truth (GT) kernels. To solve this issue, it is necessary to design an implicit estimator network that can extract discriminative blur kernel representation without relying on the supervision of ground-truth blur kernels. We design a lightweight approach for blind super-resolution (Blind-SR) that estimates the blur kernel and restores the HR image based on a deep convolutional neural network (CNN) and a deep super-resolution residual convolutional generative adversarial network. Since the blur kernel for blind image SR is unknown, following the image formation model of blind super-resolution problem, we firstly introduce a neural network-based model to estimate the blur kernel. This is achieved by (i) a Super Resolver that, from a low-resolution input, generates the corresponding SR image; and (ii) an Estimator Network generating the blur kernel from the input datum. The output of both models is used in a novel loss formulation. The proposed network is end-to-end trainable. The methodology proposed is substantiated by both quantitative and qualitative experiments. Results on benchmarks demonstrate that our computationally efficient approach (12x fewer parameters than the state-of-the-art models) performs favorably with respect to existing approaches and can be used on devices with limited computational capabilities.

Blind Super-Resolution of 3D MRI via Unsupervised Domain Transformation.

High-resolution medical images can be effectively used for clinical diagnosis. However, the acquisition of high-resolution images is difficult and often limited by medical instruments. Super-resolution (SR) methods provide a solution, where high-resolution (HR) images can be reconstructed from low-resolution (LR) ones. Most of existing deep neural networks for 3D SR medical images trained in a non-blind process, where LR images are directly degraded from HR data via a pre-determined downscale method. Such approaches rely heavily on the assumed degradation model, resulting in inevitable deviations in real clinical practice. Blind super-resolution, as a more attractive research line for this field, aims to generate HR images from LR inputs containing unknown degradation. Towards generalizing SR models for diverse types of degradation, we propose a robust blind SR of 3D medical images in an unsupervised manner with domain correction and upscaling treatment. First, a CycleGAN-based architecture is implemented to generate the LR data from the source domain to the target one for domain correction. Then, an upscaling network is learned via pre-determined HR-LR couples for reconstruction. The proposed framework is able to automatically learn noisy and blurry correction kernels for unpaired 3D SR magnetic resonance images (MRI). Our method achieves better and more robust performances in reconstruction of HR images from LR MRI with multiple unknown degradation processes, and show its superiority to other state-of-the-art supervised models and cycle-consistency based methods, especially in severe distortion cases.

Medical image blind super‐resolution based on improved degradation process

AbstractClinical diagnosis has high requirements for the resolution of medical images, but most existing medical images super‐ resolution (SR) methods are performed under a known or specific degradation kernel. However, the difference between the actual degradations and their assumed degradation kernels results in a severe performance drop for the advanced SR methods in real applications. This paper proposes a medical image blind super‐resolution model (Med‐BSR) based on an improved degradation process to handle this issue. The model makes each of the degradation factors in medical image blind SR, such as blur, noise, and downsampling, more complex and practical. Specifically, the authors use the random select/combine strategy to randomly arrange and combine the type and order of each degradation factor, which significantly expands the degradation space. The authors also improved the loss function of the primary enhanced super‐resolution generative adversarial networks (ESRGAN) network. The extensive experimental results demonstrate that the authors’ designed model can accurately restore the natural degradation process, which can reconstruct high‐quality SR medical images. It also has a good generalization ability to realistic images simultaneously.

Blind Super-Resolution for SAR Images with Speckle Noise Based on Deep Learning Probabilistic Degradation Model and SAR Priors

As an active microwave coherent imaging technology, synthetic aperture radar (SAR) images suffer from severe speckle noise and low-resolution problems due to the limitations of the imaging system, which cause difficulties in image interpretation and target detection. However, the existing SAR super-resolution (SR) methods usually reconstruct the images by a determined degradation model and hardly consider multiplicative speckle noise, meanwhile, most SR models are trained with synthetic datasets in which the low-resolution (LR) images are down-sampled from their high-resolution (HR) counterparts. These constraints cause a serious domain gap between the synthetic and real SAR images. To solve the above problems, this paper proposes an unsupervised blind SR method for SAR images by introducing SAR priors in a cycle-GAN framework. First, a learnable probabilistic degradation model combined with SAR noise priors was presented to satisfy various SAR images produced from different platforms. Then, a degradation model and a SR model in a unified cycle-GAN framework were trained simultaneously to learn the intrinsic relationship between HR–LR domains. The model was trained with real LR and HR SAR images instead of synthetic paired images to conquer the domain gap. Finally, experimental results on both synthetic and real SAR images demonstrated the high performance of the proposed method in terms of image quality and visual perception. Additionally, we found the proposed SR method demonstrates the tremendous potential for target detection tasks by reducing missed detection and false alarms significantly.

Deep Learning-Based Single-Image Super-Resolution: A Comprehensive Review

High-fidelity information, such as 4K quality videos and photographs, is increasing as high-speed internet access becomes more widespread and less expensive. Even though camera sensors’ performance is constantly improving, artificially enhanced photos and videos created by intelligent image processing algorithms have significantly improved image fidelity in recent years. Single image super-resolution is a class of algorithms that produces a high-resolution image from a given low-resolution image. Since the advent of deep learning a decade ago, this field has made significant strides. This paper presents a comprehensive review of the deep learning assisted single image super-resolution domain including generative adversarial network (GAN) models that discusses the prominent architectures, models used, and their merits and demerits. The reason behind covering the GAN models is that it is been known to perform better than the conventional deep learning methods given the resources and the time. For real-world applications with noise and other issues that can cause low-fidelity super resolution (SR) images, we examine another solution based on GAN model. This GAN model-based technique popularly known as blind super resolution is more resilient. We examined the various super-resolution techniques by varying image scaling factors (i.e., 2x, 3x, 4x) to measure PSNR and SSIM metrics for the different datasets. PSNR across the different datasets covered in the experimental section shows an average of 14-17 % decrease in the score as we move up the image resolution scale from 2x to 4x. This is observed across all the datasets and for every model mentioned in the experimental section of the paper. The results also show that blind super-resolution outperforms the conventional deep learning methods and the more complex GAN models. GAN models are complex and preferred when the upscale factor is high, while residual and dense models are recommended for smaller upscaling factors. This paper also discusses the applications of image super-resolution, and finally, the paper is concluded with challenges and future directions.

Contrastive Learning for Blind Super-Resolution via A Distortion-Specific Network

Previous deep learning-based super-resolution (SR) methods rely on the assumption that the degradation process is predefined (e.g., bicubic downsampling). Thus, their performance would suffer from deterioration if the real degradation is not consistent with the assumption. To deal with real-world scenarios, existing blind SR methods are committed to estimating both the degradation and the super-resolved image with an extra loss or iterative scheme. However, degradation estimation that requires more computation would result in limited SR performance due to the accumulated estimation errors. In this paper, we propose a contrastive regularization built upon contrastive learning to exploit both the information of blurry images and clear images as negative and positive samples, respectively. Contrastive regularization ensures that the restored image is pulled closer to the clear image and pushed far away from the blurry image in the representation space. Furthermore, instead of estimating the degradation, we extract global statistical prior information to capture the character of the distortion. Considering the coupling between the degradation and the low-resolution image, we embed the global prior into the distortion-specific SR network to make our method adaptive to the changes of distortions. We term our distortion-specific network with contrastive regularization as CRDNet. The extensive experiments on synthetic and real-world scenes demonstrate that our lightweight CRDNet surpasses state-of-the-art blind super-resolution approaches.

Learning Detail-Structure Alternative Optimization for Blind Super-Resolution

Existing convolutional neural networks (CNN) based image super-resolution (SR) methods have achieved impressive performance on bicubic kernel, which is not valid to handle unknown degradations in real-world applications. Recent blind SR methods suggest to reconstruct SR images relying on blur kernel estimation. However, their results still remain visible artifacts and detail distortion due to the estimation errors. To alleviate these problems, in this paper, we propose an effective and kernel-free network, namely DSSR, which enables recurrent detail-structure alternative optimization without blur kernel prior incorporation for blind SR. Specifically, in our DSSR, a detail-structure modulation module (DSMM) is built to exploit the interaction and collaboration of image details and structures. The DSMM consists of two components: a detail restoration unit (DRU) and a structure modulation unit (SMU). The former aims at regressing the intermediate HR detail reconstruction from LR structural contexts, and the latter performs structural contexts modulation conditioned on the learned detail maps at both HR and LR spaces. Besides, we use the output of DSMM as the hidden state and design our DSSR architecture from a recurrent convolutional neural network (RCNN) view. In this way, the network can alternatively optimize the image details and structural contexts, achieving co-optimization across time. Moreover, equipped with the recurrent connection, our DSSR allows low- and high-level feature representations complementary by observing previous HR details and contexts at every unrolling time. Extensive experiments on synthetic datasets and real-world images demonstrate that our method achieves the state-of-the-art against existing methods. The source code can be found at https://github.com/Arcananana/DSSR.

Blind Super-Resolution of Single Remotely Sensed Hyperspectral Image

Blind Super-Resolution of Single Remotely Sensed Hyperspectral Image

Blind Superresolution of Satellite Videos by Ghost Module-Based Convolutional Networks

Deep learning (DL)-based video satellite superresolution (SR) methods have recently yielded superior performance over traditional model-based methods by using an end-to-end manner. Existing DL-based methods usually assume that the blur kernels are known and, thus, do not model the blur kernels during restoration. However, this assumption is rarely held for real satellite videos and leads to oversmoothed results. In this article, we propose a Ghost module-based convolution network model for blind SR of satellite videos. The proposed Ghost module-based video SR (GVSR) method, which assumes that the blur kernel is unknown, consists of two main modules, i.e., the preliminary image generation module and the SR results’ reconstruction module. First, the motion information from adjacent video frames and the wrapped images are explored by an optical flow estimation network, the blur kernel is flexibly obtained by a blur kernel estimation network, and the preliminary high-resolution image is generated by feeding both blur kernel and wrapped images. Second, a reconstruction network consisting of three paths with attention-based Ghost (AG) bottlenecks is designed to remove artifacts in the preliminary image and obtain the final high-quality SR results. Experiments conducted on Jilin-1 and OVS-1 satellite videos demonstrate that the qualitative and quantitative performance of our proposed method is superior to current state-of-the-art methods.

Blind super-resolution of point sources via fast iterative hard thresholding

Blind super-resolution of point sources via fast iterative hard thresholding

Meta-Learning based Degradation Representation for Blind Super-Resolution.

Blind image super-resolution (blind SR) aims to generate high-resolution (HR) images from low-resolution (LR) input images with unknown degradations. To enhance the performance of SR, the majority of blind SR methods introduce an explicit degradation estimator, which helps the SR model adjust to unknown degradation scenarios. Unfortunately, it is impractical to provide concrete labels for the multiple combinations of degradations (e.g., blurring, noise, or JPEG compression) to guide the training of the degradation estimator. Moreover, the special designs for certain degradations hinder the models from being generalized for dealing with other degradations. Thus, it is imperative to devise an implicit degradation estimator that can extract discriminative degradation representations for all types of degradations without requiring the supervision of degradation ground-truth. To this end, we propose a Meta-Learning based Region Degradation Aware SR Network (MRDA), including Meta-Learning Network (MLN), Degradation Extraction Network (DEN), and Region Degradation Aware SR Network (RDAN). To handle the lack of ground-truth degradation, we use the MLN to rapidly adapt to the specific complex degradation after several iterations and extract implicit degradation information. Subsequently, a teacher network MRDAT is designed to further utilize the degradation information extracted by MLN for SR. However, MLN requires iterating on paired LR and HR images, which is unavailable in the inference phase. Therefore, we adopt knowledge distillation (KD) to make the student network learn to directly extract the same implicit degradation representation (IDR) as the teacher from LR images. Furthermore, we introduce an RDAN module that is capable of discerning regional degradations, allowing IDR to adaptively influence various texture patterns. Extensive experiments under classic and real-world degradation settings show that MRDA achieves SOTA performance and can generalize to various degradation processes.

Self-FuseNet: Data Free Unsupervised Remote Sensing Image Super-Resolution

Real-world degradations deviate from ideal degradations, as most deep learning-based scenarios involve the ideal synthesis of low-resolution (LR) counterpart images by popularly used bicubic interpolation. Moreover, supervised learning approaches rely on many high-resolution (HR) and LR image pairings to reconstruct missing information based on their association, developed by complex long hours of deep neural network training. Additionally, the trained model's generalizability on various image datasets with various distributions is not guaranteed. To overcome this challenge, we proposed our novel Self-FuseNet, particularly for extremely poor-resolution satellite images. Also, the network exhibits strong generalization performance on additional datasets (both “ideal” and “nonideal” scenarios). The network is especially for those image datasets suffering from the following two significant limitations: 1) nonavailability of ground truth HR images; 2) limitation of a large count of the unpaired dataset for deep neural network training. The benefit of the proposed model is threefold: 1) it does not require any significant extensive training data, either paired or unpaired but only a single LR image without prior knowledge of its distribution; 2) it is a simple and effective model for super-resolving very poor-resolution images, saving computational resources and time; 3) using UNet, the processing of data are accelerated by the network's wide skip connections, allowing image reconstruction with fewer parameters. Rather than using an inverse approach, as common in most deep learning scenarios, we introduced a forward approach to super-resolve exceptionally LR remote sensing images. This demonstrates its supremacy over recently proposed state-of-the-art methods for unsupervised single real-world image blind super-resolution.

The Best of Both Worlds: A Framework for Combining Degradation Prediction with High Performance Super-Resolution Networks.

To date, the best-performing blind super-resolution (SR) techniques follow one of two paradigms: (A) train standard SR networks on synthetic low-resolution-high-resolution (LR-HR) pairs or (B) predict the degradations of an LR image and then use these to inform a customised SR network. Despite significant progress, subscribers to the former miss out on useful degradation information and followers of the latter rely on weaker SR networks, which are significantly outperformed by the latest architectural advancements. In this work, we present a framework for combining any blind SR prediction mechanism with any deep SR network. We show that a single lightweight metadata insertion block together with a degradation prediction mechanism can allow non-blind SR architectures to rival or outperform state-of-the-art dedicated blind SR networks. We implement various contrastive and iterative degradation prediction schemes and show they are readily compatible with high-performance SR networks such as RCAN and HAN within our framework. Furthermore, we demonstrate our framework's robustness by successfully performing blind SR on images degraded with blurring, noise and compression. This represents the first explicit combined blind prediction and SR of images degraded with such a complex pipeline, acting as a baseline for further advancements.

Frequency aggregation network for blind super-resolution based on degradation representation

Single image super-resolution (SISR) research has made significant progress with advances in convolutional neural networks (CNNs). However, most existing super-resolution (SR) methods ignore the differences in frequency distribution between uniformly degraded images (e.g., bicubic downsampling) and complicated degraded images, leading to unsatisfactory restoration performance in real-world images. Recent blind SR studies use degradation estimation to solve the problem, but they rarely capture the true degradation discrepancy from the perspective of the frequency domain. This gap motivates us to solve the blind SR problem by fusing the information of different frequency components to compensate for information loss comprehensively. Thus, we propose a frequency aggregation SR (FASR) network for blind SR that fully utilizes frequency information to guide high-quality image reconstruction based on degradation representation with flexible adaption to various degradations. Specifically, we design an effective degradation-aware module (EDAM), which incorporates a local cross-channel interaction (LCCI) method designed by us. LCCI performs feature adaption more effectively by degradation information. Furthermore, we design the detail-enhanced convolution block (DECB) within the EDAM to implicitly extract different spatial derivatives for retaining edge information and benefiting reconstruction quality. To comprehensively compensate for frequency information, we propose a frequency compensation module (FCM), which uses different frequency components to obtain different frequency information. This module further mitigates the problem of frequency information loss caused by complicated realistic degradation. Extensive experiments on synthetic and real images show that our network outperforms other advanced blind SR methods in both quantitative and qualitative terms.

Multi-Frame Blind Super-Resolution Based on Joint Motion Estimation and Blur Kernel Estimation

Multi-frame super-resolution makes up for the deficiency of sensor hardware and significantly improves image resolution by using the information of inter-frame and intra-frame images. Inaccurate blur kernel estimation will enlarge the distortion of the estimated high-resolution image. Therefore, multi-frame blind super resolution with unknown blur kernel is more challenging. For the purpose of reducing the impact of inaccurate motion estimation and blur kernel estimation on the super-resolved image, we propose a novel method combining motion estimation, blur kernel estimation and super resolution. The confidence weight of low-resolution images and the parameter value of the motion model obtained in image reconstruction are added to the modified motion estimation and blur kernel estimation. At the same time, Jacobian matrix, which can better describe the motion change, is introduced to further correct the error of motion estimation. Based on the results acquired from the experiments on synthetic data and real data, the superiority of the proposed method over others is obvious. The reconstructed high-resolution image retains the details of the image effectively, and the artifacts are greatly reduced.

Deep Unfolded Projected Alternating Minimization Algorithm for Blind Image Super-resolution

Blind super-resolution (BSR) has a wide range of applications in fruitful fields, such as pattern recognition, image processing, and signal processing. This problem focuses on recovering the original high-resolution (HR) details and blur kernel from a low-resolution (LR) blurry image. In these years, learning-based BSRs have evolved to become increasingly powerful through more extensive labelled data, better-designed end-to-end model and stronger training algorithm. However, the classic model-based methods typically provide good interpretability for applying prior information and generalizability of dealing with variational factors (blur kernels) with respect to their explicit alternative optimization framework, while the learning-based methods generally highly rely on the training data and are typically perform less effective on the unseen kernels. In this paper, a learning-aided model-based algorithm is proposed to solve the aforementioned issues by unfolding a well-designed model-based BSR approach, the projected alternating minimization (PAM) algorithm, that establishes an explicit alternative optimization iterations with data-driven network-based update module. As a result, the proposed unfolding PAM (UPAM) inherits the good interpretability and generalizability of model-based methods towards solving arbitrary blur kernels, while retaining the high performance of learning-based methods. Extensive experiments verify that the proposed UPAM outperforms both of the model-based and learning-based BSR methods, and gains significantly better performance when dealing with the blur kernels.