Single Image Super-resolution Research Articles

Single image super-resolution based on deep networks and wavelet transform

Single image super-resolution based on deep networks and wavelet transform

Perceptual super-resolution in multiple sclerosis MRI

IntroductionMagnetic resonance imaging (MRI) is crucial for diagnosing and monitoring of multiple sclerosis (MS) as it is used to assess lesions in the brain and spinal cord. However, in real-world clinical settings, MRI scans are often acquired with thick slices, limiting their utility for automated quantitative analyses. This work presents a single-image super-resolution (SR) reconstruction framework that leverages SR convolutional neural networks (CNN) to enhance the through-plane resolution of structural MRI in people with MS (PwMS).MethodsOur strategy involves the supervised fine-tuning of CNN architectures, guided by a content loss function that promotes perceptual quality, as well as reconstruction accuracy, to recover high-level image features.ResultsExtensive evaluation with MRI data of PwMS shows that our SR strategy leads to more accurate MRI reconstructions than competing methods. Furthermore, it improves lesion segmentation on low-resolution MRI, approaching the performance achievable with high-resolution images.DiscussionResults demonstrate the potential of our SR framework to facilitate the use of low-resolution retrospective MRI from real-world clinical settings to investigate quantitative image-based biomarkers of MS.

Transforming Image Super-Resolution: A ConvFormer-based Efficient Approach.

Recent progress in single-image super-resolution (SISR) has achieved remarkable performance, yet the computational costs of these methods remain a challenge for deployment on resource-constrained devices. In particular, transformer-based methods, which leverage self-attention mechanisms, have led to significant breakthroughs but also introduce substantial computational costs. To tackle this issue, we introduce the Convolutional Transformer layer (ConvFormer) and propose a ConvFormer-based Super-Resolution network (CFSR), offering an effective and efficient solution for lightweight image super-resolution. The proposed method inherits the advantages of both convolution-based and transformer-based approaches. Specifically, CFSR utilizes large kernel convolutions as a feature mixer to replace the self-attention module, efficiently modeling long-range dependencies and extensive receptive fields with minimal computational overhead. Furthermore, we propose an edge-preserving feed-forward network (EFN) designed to achieve local feature aggregation while effectively preserving high-frequency information. Extensive experiments demonstrate that CFSR strikes an optimal balance between computational cost and performance compared to existing lightweight SR methods. When benchmarked against state-of-the-art methods such as ShuffleMixer, the proposed CFSR achieves a gain of 0.39 dB on the Urban100 dataset for the x2 super-resolution task while requiring 26% and 31% fewer parameters and FLOPs, respectively. The code and pre-trained models are available at https://github.com/Aitical/CFSR.

This Microtubule Does Not Exist: Super-Resolution Microscopy Image Generation by a Diffusion Model.

Generative models, such as diffusion models, have made significant advancements in recent years, enabling the synthesis of high-quality realistic data across various domains. Here, the adaptation and training of a diffusion model on super-resolution microscopy images are explored. It is shown that the generated images resemble experimental images, and that the generation process does not exhibit a large degree of memorization from existing images in the training set. To demonstrate the usefulness of the generative model for data augmentation, the performance of a deep learning-based single-image super-resolution (SISR) method trained using generated high-resolution data is compared against training using experimental images alone, or images generated by mathematical modeling. Using a few experimental images, the reconstruction quality and the spatial resolution of the reconstructed images are improved, showcasing the potential of diffusion model image generation for overcoming the limitations accompanying the collection and annotation of microscopy images. Finally, the pipeline is made publicly available, runnable online, and user-friendly to enable researchers to generate their own synthetic microscopy data. This work demonstrates the potential contribution of generative diffusion models for microscopy tasks and paves the way for their future application in this field.

An efficient feature reuse distillation network for lightweight image super-resolution

In recent research, single-image super-resolution (SISR) using deep Convolutional Neural Networks (CNN) has seen significant advancements. While previous methods excelled at learning complex mappings between low-resolution (LR) and high-resolution (HR) images, they often required substantial computational and memory resources. We propose the Efficient Feature Reuse Distillation Network (EFRDN) to alleviate these challenges. EFRDN primarily comprises Asymmetric Convolutional Distillation Modules (ACDM), incorporating the Multiple Self-Calibrating Convolution (MSCC) units for spatial and channel feature extraction. It includes an Asymmetric Convolution Residual Block (ACRB) to enhance the skeleton information of the square convolution kernel and a Feature Fusion Lattice Block (FFLB) to convert low-order input signals into higher-order representations. Introducing a Transformer module for global features, we enhance feature reuse and gradient flow, improving model performance and efficiency. Extensive experimental results demonstrate that EFRDN outperforms existing methods in performance while conserving computing and memory resources.

Dual-Path Large Kernel Learning and Its Applications in Single-Image Super-Resolution.

To enhance the performance of super-resolution models, neural networks frequently employ module stacking. However, this approach inevitably results in an excessive proliferation of parameter counts and information redundancy, ultimately constraining the deployment of these models on mobile devices. To surmount this limitation, this study introduces the application of Dual-path Large Kernel Learning (DLKL) to the task of image super-resolution. Within the DLKL framework, we harness a multiscale large kernel decomposition technique to efficiently establish long-range dependencies among pixels. This network not only maintains excellent performance but also significantly mitigates the parameter burden, achieving an optimal balance between network performance and efficiency. When compared with other prevalent algorithms, DLKL exhibits remarkable proficiency in generating images with sharper textures and structures that are more akin to natural ones. It is particularly noteworthy that on the challenging texture dataset Urban100, the network proposed in this study achieved a significant improvement in Peak Signal-to-Noise Ratio (PSNR) for the ×4 upscaling task, with an increase of 0.32 dB and 0.19 dB compared with the state-of-the-art HAFRN and MICU networks, respectively. This remarkable result not only validates the effectiveness of the present model in complex image super-resolution tasks but also highlights its superior performance and unique advantages in the field.

When guided diffusion model meets zero-shot image super-resolution

Existing deep learning-based single-image super-resolution (SR) methods typically rely on vast quantities of paired data. As an essential solution, zero-shot SR methods require only a single image to handle image-specific degradation. However, these methods still struggle to recover fine-grained details due to the lack of supervised information. In this work, we propose a novel guided Diffusion model for Zero-shot image SR (ZeroDiff) to explicitly direct image quality enhancement. Specifically, we elaborate two key guidance strategies: (1) high-frequency guidance and (2) content-consistent guidance. The former concentrates on boosting fine-grained textures by embedding high-frequency information into the cross-attention mechanism of the noise estimator. The latter avoids the sampling deviating from the original image in terms of structure and low-frequency content. Specifically, the noisy images at each diffusion step are injected into the corresponding sampling step, encouraging the sampled image to be consistent with that of the corresponding diffusion step. Moreover, we design a progressive zoom-in paradigm by gradually enlarging the image size and enriching the image details to boost the sampling efficiency of diffusion models, while enabling high-quality image reconstruction. Extensive experiments reveal that our method achieves comparable results with other state-of-the-art methods in quantitative and qualitative evaluations on both face and natural images from synthetic and real-world datasets.

Revealing the Dark Side of Non-Local Attention in Single Image Super-Resolution.

Single Image Super-Resolution (SISR) aims to reconstruct a high-resolution image from its corresponding low-resolution input. A common technique to enhance the reconstruction quality is Non-Local Attention (NLA), which leverages self-similar texture patterns in images. However, we have made a novel finding that challenges the prevailing wisdom. Our research reveals that NLA can be detrimental to SISR and even produce severely distorted textures. For example, when dealing with severely degrade textures, NLA may generate unrealistic results due to the inconsistency of non-local texture patterns. This problem is overlooked by existing works, which only measure the average reconstruction quality of the whole image, without considering the potential risks of using NLA. To address this issue, we propose a new perspective for evaluating the reconstruction quality of NLA, by focusing on the sub-pixel level that matches the pixel-wise fusion manner of NLA. From this perspective, we provide the approximate reconstruction performance upper bound of NLA, which guides us to design a concise yet effective Texture-Fidelity Strategy (TFS) to mitigate the degradation caused by NLA. Moreover, the proposed TFS can be conveniently integrated into existing NLA-based SISR models as a general building block. Based on the TFS, we develop a Deep Texture-Fidelity Network (DTFN), which achieves state-of-the-art performance for SISR. Our code and a pre-trained DTFN are available on GitHub for verification.

A novel theoretical analysis on optimal pipeline of multi-frame image super-resolution using sparse coding

Super-resolution is the process of obtaining a high-resolution (HR) image from one or more low-resolution (LR) images. Single image super-resolution (SISR) deals with one LR image while multi-frame super-resolution (MFSR) employs several LR ones to reach the HR output. MFSR pipeline consists of alignment, fusion, and reconstruction. We conduct a theoretical analysis using sparse coding (SC) and iterative shrinkage-thresholding algorithm to fill the gap of mathematical justification in the execution order of the optimal MFSR pipeline. Our analysis recommends executing alignment and fusion before the reconstruction stage (whether through deconvolution or SISR techniques). The suggested order ensures enhanced performance in terms of peak signal-to-noise ratio and structural similarity index. The optimal pipeline also reduces computational complexity compared to intuitive approaches that apply SISR to each input LR image. Also, we demonstrate the usefulness of SC in analysis of computer vision tasks such as MFSR, leveraging the sparsity assumption in natural images. Simulation results support the findings of our theoretical analysis, both quantitatively and qualitatively.

A Multi-Branch Feature Extraction Residual Network for Lightweight Image Super-Resolution

Single-image super-resolution (SISR) seeks to elucidate the mapping relationships between low-resolution and high-resolution images. However, high-performance network models often entail a significant number of parameters and computations, presenting limitations in practical applications. Therefore, prioritizing a light weight and efficiency becomes crucial when applying image super-resolution (SR) to real-world scenarios. We propose a straightforward and efficient method, the Multi-Branch Feature Extraction Residual Network (MFERN), to tackle lightweight image SR through the fusion of multi-information self-calibration and multi-attention information. Specifically, we have devised a Multi-Branch Residual Feature Fusion Module (MRFFM) that leverages a multi-branch residual structure to succinctly and effectively fuse multiple pieces of information. Within the MRFFM, we have designed the Multi-Scale Attention Feature Fusion Block (MAFFB) to adeptly extract features via convolution and self-calibration attention operations. Furthermore, we introduce a Dual Feature Calibration Block (DFCB) to dynamically fuse feature information using dynamic weight values derived from the upper and lower branches. Additionally, to overcome the limitation of convolution in solely extracting local information, we incorporate a Transformer module to effectively integrate global information. The experimental results demonstrate that our MFERN exhibits outstanding performance in terms of model parameters and overall performance.

Cross-Dimensional Attention Fusion Network for Simulated Single Image Super-Resolution

Cross-Dimensional Attention Fusion Network for Simulated Single Image Super-Resolution

GlobalSR: Global context network for single image super-resolution via deformable convolution attention and fast Fourier convolution

Vision Transformer have achieved impressive performance in image super-resolution. However, they suffer from low inference speed mainly because of the quadratic complexity of multi-head self-attention (MHSA), which is the key to learning long-range dependencies. On the contrary, most CNN-based methods neglect the important effect of global contextual information, resulting in inaccurate and blurring details. If one can make the best of both Transformers and CNNs, it will achieve a better trade-off between image quality and inference speed. Based on this observation, firstly assume that the main factor affecting the performance in the Transformer-based SR models is the general architecture design, not the specific MHSA component. To verify this, some ablation studies are made by replacing MHSA with large kernel convolutions, alongside other essential module replacements. Surprisingly, the derived models achieve competitive performance. Therefore, a general architecture design GlobalSR is extracted by not specifying the core modules including blocks and domain embeddings of Transformer-based SR models. It also contains three practical guidelines for designing a lightweight SR network utilizing image-level global contextual information to reconstruct SR images. Following the guidelines, the blocks and domain embeddings of GlobalSR are instantiated via Deformable Convolution Attention Block (DCAB) and Fast Fourier Convolution Domain Embedding (FCDE), respectively. The instantiation of general architecture, termed GlobalSR-DF, proposes a DCA to extract the global contextual feature by utilizing Deformable Convolution and a Hadamard product as the attention map at the block level. Meanwhile, the FCDE utilizes the Fast Fourier to transform the input spatial feature into frequency space and then extract image-level global information from it by convolutions. Extensive experiments demonstrate that GlobalSR is the key part in achieving a superior trade-off between SR quality and efficiency. Specifically, our proposed GlobalSR-DF outperforms state-of-the-art CNN-based and ViT-based SISR models regarding accuracy-speed trade-offs with sharp and natural details.

Rectified Binary Network for Single-Image Super-Resolution.

Binary neural network (BNN) is an effective approach to reduce the memory usage and the computational complexity of full-precision convolutional neural networks (CNNs), which has been widely used in the field of deep learning. However, there are different properties between BNNs and real-valued models, making it difficult to draw on the experience of CNN composition to develop BNN. In this article, we study the application of binary network to the single-image super-resolution (SISR) task in which the network is trained for restoring original high-resolution (HR) images. Generally, the distribution of features in the network for SISR is more complex than those in recognition models for preserving the abundant image information, e.g., texture, color, and details. To enhance the representation ability of BNN, we explore a novel activation-rectified inference (ARI) module that achieves a more complete representation of features by combining observations from different quantitative perspectives. The activations are divided into several parts with different quantification intervals and are inferred independently. This allows the binary activations to retain more image detail and yield finer inference. In addition, we further propose an adaptive approximation estimator (AAE) for gradually learning the accurate gradient estimation interval in each layer to alleviate the optimization difficulty. Experiments conducted on several benchmarks show that our approach is able to learn a binary SISR model with superior performance over the state-of-the-art methods. The code will be released at https://github.com/jwxintt/Rectified-BSR.

Deep learning for the super resolution of Mediterranean sea surface temperature fields

Abstract. Sea surface temperature (SST) is one of the essential variables of the Earth's climate system. Being at the air–sea interface, SST modulates heat fluxes in and out of the ocean, provides insight into several upper and interior ocean dynamical processes, and is a fundamental indicator of climate variability potentially impacting the health of marine ecosystems. Its accurate estimation and regular monitoring from space is therefore crucial. However, even if satellite infrared/microwave measurements provide much better coverage than what is achievable from in situ platforms, they cannot sense the sea surface under cloudy and rainy conditions. Large gaps are present even in merged multi-sensor satellite products, and different statistical strategies, mostly based on optimal interpolation (OI) algorithms, have thus been proposed to obtain gap-free (L4) images. These techniques, however, filter out the signals below the space–time decorrelation scales considered, significantly smoothing most of the small mesoscale and submesoscale features. Here, deep learning models, originally designed for single-image super resolution (SR), are applied to enhance the effective resolution of SST products and the accuracy of SST gradients. SR schemes include a set of computer vision techniques leveraging convolutional neural networks to retrieve high-resolution data from low-resolution images. A dilated convolutional multi-scale learning network, which includes an adaptive residual strategy and implements a channel attention mechanism, is used to reconstruct features in SST data at 1/100° spatial resolution starting from 1/16° data over the Mediterranean Sea. The application of this technique shows an improvement in the high-resolution reconstruction, capturing small-scale features and providing a root-mean-squared-difference improvement of 0.02 °C with respect to the L3 ground-truth data.

JOA‐GAN: An improved single‐image super‐resolution network for remote sensing based on GAN

AbstractImage super‐resolution (SR) has been widely applied in remote sensing to generate high‐resolution (HR) images without increasing hardware costs. However, SR is a severe ill‐posed problem. As deep learning advances, existing methods have solved this problem to a certain extent. However, the complex spatial distribution of remote sensing images still poses a challenge in effectively extracting abundant high‐frequency details from the images. Here, a single‐image super‐resolution (SISR) network based on the generative adversarial network (GAN) for remote sensing is presented, called JOA‐GAN. Firstly, a joint‐attention module (JOA) is proposed to focus the network on high‐frequency regions in remote sensing images to enhance the quality of image reconstruction. In the generator network, a multi‐scale densely connected feature extraction block (ERRDB) is proposed, which acquires features at different scales using MSconv blocks containing multi‐scale convolutions and automatically adjusts the features by JOA. In the discriminator network, the relative discriminator is used to compute the relative probability instead of the absolute probability, which helps the network learn clearer and more realistic texture details. JOA‐GAN is compared with other advanced methods, and the results demonstrate that JOA‐GAN has improved objective evaluation metrics and achieved superior visual effects.

Deep blind super-resolution for hyperspectral images

Current deep learning methods for single hyperspectral image super-resolution are non-blind ones, which adopt the simplistic bicubic degradation model. These models have poor generalization performance and cannot handle unknown degradation. Additionally, blind super-resolution methods for RGB images neglect rich spectral information in hyperspectral images, which causes spectral distortion in results. To address these issues, we consider the degeneration estimation and propose a single hyperspectral image blind super-resolution algorithm. Specifically, we first use a blur kernel estimation network and a deblurring network to obtain the image without blur. We change the receptive field by exchanging spatial channel information when estimating the kernel, thereby obtaining blurry information at different scales. In addition, to reduce the errors introduced by kernel mismatch and the relevant information among bands ignored by the group convolution, we fuse the spatial information of the blurred image to compensate for the errors and then utilize the spatial and spectral information of other bands to guide the feature extraction of the current band. Finally, we integrate the feature information from different branches in a global fusion network to improve spatial and spectral information fidelity further. Extensive experiments on synthetic and real-world hyperspectral datasets unequivocally demonstrate that our approach outperforms other state-of-the-art approaches in terms of both quantitative and qualitative evaluations. Our code is publicly available at https://github.com/YoungP2001/DBSR.

Lightweight Single Image Super-Resolution via Efficient Mixture of Transformers and Convolutional Networks.

In this paper, we propose a Local Global Union Network (LGUN), which effectively combines the strengths of Transformers and Convolutional Networks to develop a lightweight and high-performance network suitable for Single Image Super-Resolution (SISR). Specifically, we make use of the advantages of Transformers to provide input-adaptation weighting and global context interaction. We also make use of the advantages of Convolutional Networks to include spatial inductive biases and local connectivity. In the shallow layer, the local spatial information is encoded by Multi-order Local Hierarchical Attention (MLHA). In the deeper layer, we utilize Dynamic Global Sparse Attention (DGSA), which is based on the Multi-stage Token Selection (MTS) strategy to model global context dependencies. Moreover, we also conduct extensive experiments on both natural and satellite datasets, acquired through optical and satellite sensors, respectively, demonstrating that LGUN outperforms existing methods.

Multi-dimensional attention-aided transposed ConvBiLSTM network for hyperspectral image super-resolution

Hyperspectral (HS) image always suffers from the deficiency of low spatial resolution, compared with conventional optical image types, which has limited its further applications in remote sensing areas. Therefore, HS image super-resolution (SR) techniques are broadly employed in order to observe finer spatial structures while preserving the spectra of ground covers. In this paper, a novel multi-dimensional attention-aided transposed convolutional long-short term memory (LSTM) network is proposed for single HS image super-resolution task. The proposed network employs the convolutional bi-directional LSTM for the purpose of local and non-local spatial–spectral feature explorations, and transposed convolution for the purpose of image amplification and reconstruction. Moreover, a multi-dimensional attention module is proposed, aiming to capture the salient features on spectral, channel, and spatial dimensions, simultaneously, to further improve the learning abilities of network. Experiments on four commonly-used HS images demonstrate the effectiveness of this approach, compared with several state-of-the-art deep learning-based SR methods.

Asymmetric convolutional modulation network for efficient image super-resolution

Thanks to the effectiveness of large kernel convolutions in acquiring large receptive fields, lightweight single image super-resolution (SISR) approaches based on large kernel designs have achieved significant performance. However, these state-of-the-art methods still require high computational costs, and their model efficiency remains to be improved. To address these challenges, we propose an asymmetric convolutional modulation network (ACMN) for efficient image super-resolution. In detail, we design an asymmetric convolutional modulation unit (ACMU) that combines large kernel design and modulation mechanism for adaptively selecting representative features from a large receptive field. Furthermore, we adopt depth-wise asymmetric convolutions to further reduce computational burden. To supplement the local context information and promote inter-channel interaction, we further develop a channel shuffle feedforward network (CSFN) to extract local feature information and facilitate inter-channel information interaction. Experimental results demonstrate that our ACMN outperforms other state-of-the-art efficient SR methods with fewer parameters and Multi-Adds.

Efficient image super-resolution based on transformer with bidirectional interaction

In single-image super-resolution (SISR) tasks, many methods benefit from the local and global contexts of the image. Despite that, no methods use the bidirectional interaction between these two contexts. So, we were inspired by the fully adaptive Transformer for high-level vision. We propose a fully adaptive Transformer super-resolution (FATSRN) for SISR. The model uses local and global information and their bidirectional interaction in a context-aware manner. The model is based on fully adaptive self-attention (FASA) as the main block, which uses self-modulated convolutions to extract local representation adaptively. Also, the FASA uses self-attention in down-sampled space to extract global representation. In addition, this FASA uses a bidirectional adaptation process between local and global representation to model the interaction. Moreover, a fine-grained downsampling strategy is used to improve the down-sampled self-attention mechanism. Based on the FASA, we built a fully adaptive self-attention block (FASAB) as the main block of our model. Then, the fully adaptive self-attention group (FASAG) is used as the backbone for our FATSRN. Extensive experiments are done to show the efficiency of the model against the state-of-the-art methods. For example, our model improved the PSNR from 27.69 to 27.73 compared to the SwinIR-light for the B100 dataset at the scale of × 4. In addition, our model achieved 0.04 dB better PSNR compared to the state-of-the-art STSN model for the Set5 dataset at the scale of × 2 with 64% and 48% fewer parameters and Mult-adds.