Super-resolution Model Research Articles

A dual branch attention network based on practical degradation model for face super resolution.

In recent years, the field of face super-resolution (FSR) has advanced rapidly. However, complex degradation factors in real-world scenarios can severely deteriorate image quality, significantly affecting the reconstruction performance of FSR methods. Currently, there is a lack of research on degradation modeling for real-world facial images, which impacts the generalization ability of existing FSR methods. In this paper, a practical degradation model based on hybrid degradation processes is proposed to select multiple degradation processes including Gaussian noise, Rayleigh noise, Motion blur, Salt-and-Pepper noise, and Mean blur through a stochastic strategy to more realistically simulate the effect of image distortion in real scenarios. We also design a dual-branch attention network called DBANet for face super-resolution and conduct experiments on the SCUT_FBP, Helen and PFHQ datasets, achieving satisfactory results. Our proposed model is effective in handling image distortion under different degradation modalities, which improves the robustness of super-resolution reconstruction. This study introduces an innovative approach to the field of face image super-resolution, which has the potential for a wide range of practical applications. The code of DBANet will be available at https://github.com/bxzha/DBANet .

Stochastic Super-resolution of Cosmological Simulations with Denoising Diffusion Models

In recent years, deep learning models have been successfully employed for augmenting low-resolution cosmological simulations with small-scale information, a task known as “super-resolution”. So far, these cosmological super-resolution models have relied on generative adversarial networks (GANs), which can achieve highly realistic results, but suffer from various shortcomings (e.g. low sample diversity). We introduce denoising diffusion models as a powerful generative model for super-resolving cosmic large-scale structure predictions (as a first proof-of-concept in two dimensions). To obtain accurate results down to small scales, we develop a new “filter-boosted” training approach that redistributes the importance of different scales in the pixel-wise training objective. We demonstrate that our model not only produces convincing super-resolution images and power spectra consistent at the percent level, but is also able to reproduce the diversity of small-scale features consistent with a given low-resolution simulation. This enables uncertainty quantification for the generated small-scale features, which is critical for the usefulness of such super-resolution models as a viable surrogate model for cosmic structure formation.

Deep learning-driven super-resolution in Raman hyperspectral imaging: Efficient high-resolution reconstruction from low-resolution data

Deep learning (DL) has become an indispensable tool in hyperspectral data analysis, automatically extracting valuable features from complex, high-dimensional datasets. Super-resolution reconstruction, an essential aspect of hyperspectral data, involves enhancing spatial resolution, particularly relevant to low-resolution hyperspectral data. Yet, the pursuit of super-resolution in hyperspectral analysis is fraught with challenges, including acquiring ground truth high-resolution data for training, generalization, and scalability. The pressing issue of extended spectral acquisition times, notably for high-resolution scans, is a significant roadblock in hyperspectral imaging. Super-resolution methods offer a promising solution by providing higher spatial resolution data to expedite data collection and yield more efficient outcomes. This paper delves into a practical application of these concepts using Raman imaging, where spectral acquisition times can be prohibitively long. In this context, DL-based super-resolution models demonstrate their efficacy by predicting and reconstructing high-resolution Raman data from low-resolution input, eliminating the need for resource-intensive high-resolution scans. While previous work often relied on substantial high-resolution datasets, this study showcases the ability to achieve similar outcomes even with limited data, presenting a more practical and cost-effective approach. The results offer a glimpse into the transformative potential of this technology to streamline hyperspectral imaging applications by saving valuable time and resources through the successful generation of high-resolution data from low-resolution inputs.

Residual trio feature network for efficient super-resolution

Deep learning-based approaches have demonstrated impressive performance in single-image super-resolution (SISR). Efficient super-resolution compromises the reconstructed image’s quality to have fewer parameters and Flops. Ensured efficiency in image reconstruction and improved reconstruction quality of the model are significant challenges. This paper proposes a trio branch module (TBM) based on structural reparameterization. TBM achieves equivalence transformation through structural reparameterization operations, which use a complex network structure in the training phase and convert it to a more lightweight structure in the inference, achieving efficient inference while maintaining accuracy. Based on the TBM, we further design a lightweight version of the enhanced spatial attention mini (ESA-mini) and the residual trio feature block (RTFB). Moreover, the multiple RTFBs are combined to construct the residual trio network (RTFN). Finally, we introduce a localized contrast loss for better applicability to the super-resolution task, which enhances the reconstruction quality of the super-resolution model. Experiments show that the RTFN framework proposed in this paper outperforms other state-of-the-art efficient super-resolution methods in terms of inference speed and reconstruction quality.

Enhancing Image Classification with Super-Resolution Techniques: A Comparative Study of SRResNet and SRGAN

Abstract. This paper explores the use of super-resolution (SR) approaches to enhance the performance of image classification, with particular attention to the effects of various SR models on classification accuracy. Using the ImageNet Dogs dataset for assessment, the paper examines two SR techniques: Super Resolution Generative Adversarial Networks (SRGAN) and Super Resolution Residual Networks (SRResNet). The research demonstrates that deep learning-based SR methods can enhance classification accuracy. Notably, SRResNet is identified as the superior method for improving classification performance, despite generating less visually appealing images compared to SRGAN. These finding highlights that while Generative Adversarial Networks (GANs) and perceptual loss functions can enhance image quality, their impact on classification accuracy may not always be substantial. The study offers important new perspectives on the relative merits of different SR approaches, highlighting the necessity of choosing the right SR methods in accordance with the particular demands of picture classification tasks. The results suggest that SRResNet, with its focus on accuracy over visual appeal, is more effective for boosting classification model performance, offering guidance for optimizing SR methods in practical image classification applications.

Neural Operator for Planetary Remote Sensing Super-Resolution with Spectral Learning

High-resolution planetary remote sensing imagery provides detailed information for geomorphological and topographic analyses. However, acquiring such imagery is constrained by limited deep-space communication bandwidth and challenging imaging environments. Conventional super-resolution methods typically employ separate models for different scales, treating them as independent tasks. This approach limits deployment and real-time applications in planetary remote sensing. Moreover, capturing global context is crucial in planetary remote sensing images due to their contextual similarities. To address these limitations, we propose Discrete Cosine Transform (DCT)–Global Super Resolution Neural Operator (DG-SRNO), a global context-aware arbitrary-scale super-resolution model. DG-SRNO achieves super-resolution at any scale using a single framework by learning the mapping between low-resolution (LR) and high-resolution (HR) function spaces. We mathematically prove the global receptive field of DG-SRNO. To evaluate DG-SRNO’s performance in planetary remote sensing tasks, we introduce the Ceres 800 dataset, a planetary remote sensing super-resolution dataset. Extensive quantitative and qualitative experiments demonstrate DG-SRNO’s impressive reconstruction capabilities.

HVASR: Enhancing 360-degree video delivery with viewport-aware super resolution

In recent years, 360-degree videos have gained significant traction due to their capacity to provide immersive experiences. However, the adoption of 360-degree videos substantially escalates bandwidth demands, necessitating approximately four to ten times more bandwidth than traditional video formats do. This presents a considerable challenge in maintaining high-quality videos in environments characterized by limited bandwidth or unstable networks. A trend has emerged where client-side computational power and deep neural networks are employed to enhance video quality while mitigating bandwidth requirements within contemporary video delivery systems. These approaches segment a video into discrete chunks and apply super resolution (SR) models to each segment, streaming low-resolution (LR) chunks alongside their corresponding SR models to the client. Although these methods enhance both video quality and transmission efficiency for conventional videos, they impose greater computational resource demands when applied to 360-degree content, thereby constraining widespread implementation. This paper introduces an innovative method called HVASR for 360-degree videos that leverages viewport information for more precise segmentation and minimizes model training costs as well as bandwidth requirements. Additionally, HVASR incorporates a viewport-aware training strategy that is aimed at further enhancing performance while reducing computational expenses. The experimental results demonstrate that HVASR achieves an average utility increase ranging from 12.46% to 40.89% across various scenes.

GAN-Based Dual Image Super Resolution for Satellite Imagery Decreasing Radiometric Uncertainty

Abstract. Super resolution for satellite imagery possesses specific challenges due to its unique feature geometry compared to classic computer vision. Specifically, satellite images contain a high amount of relatively small, distributed high-frequency features that are hard to preserve when sampling up to a higher resolution. General adversarial networks (GANs) are suitable for super resolution tasks but need special attention concerning the geometric and radiometric accuracy of the results. We propose a network for versatile satellite imagery super resolution (VSISR) that focuses on high-frequency detail preservation and radiometric consistency. As a novelty, it is able to utilize a reference high resolution image during inference to lower hallucinations without altering the source image’s radiometric characteristics.A GAN that is already handling well high frequencies in standard computer vision cases is adapted for satellite imagery and supplemented with a mixed pixel approach for data augmentation. Training on a diverse RGB dataset from four satellite missions results in a versatile super resolution model that is optimized to preserve radiometric features and minimize hallucinations. Compared to other neural networks for satellite image super resolution in their respective datasets, VSISR performs in the middle regarding mean PSNR (25.30 dB) and SSIM (0.8098). Nevertheless, it is the first time that, using mixed pixel training on a comparably small dataset, this versatility concerning the satellite data source is achieved while maintaining their unique radiometric traits.

A First-Order Image Super-Resolution Model Promoting Sharp Boundaries and Suppressing the Staircase Effect

A First-Order Image Super-Resolution Model Promoting Sharp Boundaries and Suppressing the Staircase Effect

A conditional deep learning model for super-resolution reconstruction of small-scale turbulent structures in particle-Laden flows

Super-resolution reconstruction of turbulent flows using deep learning has gained significant attention, yet challenges remain in accurately capturing physical small-scale structures. This study introduces the Conditional Enhanced Super-Resolution Generative Adversarial Network (CESRGAN) for reconstructing high-resolution turbulent velocity fields from low-resolution inputs. CESRGAN consists of a conditional discriminator and a conditional generator, the latter being called CoGEN. CoGEN incorporates subgrid-scale (SGS) turbulence kinetic energy as conditional information, improving the recovery of small-scale turbulent structures with the desired level of energy. By being aware of SGS turbulence kinetic energy, CoGEN is relatively insensitive to the degree of detail in the input. As shown in the paper, its advantages become more pronounced when the model is applied to heavily filtered input. We evaluate the model using direct numerical simulation (DNS) data of forced homogeneous isotropic turbulence. The analysis of Q-criterion isosurfaces, energy spectra, and probability density functions shows that the proposed CoGEN reconstructs fine-scale vortical structures more precisely and captures turbulent intermittency better compared to the traditional generator. Particle-pair dispersion simulations validate the physical fidelity of CoGEN-reconstructed fields, closely matching DNS results across various Stokes numbers and filtering levels. This paper demonstrates how incorporating available physical information enhances super-resolution models for turbulent flows.

Spectral network based on lattice convolution and adversarial training for noise-robust speech super-resolution.

Speech super-resolution aims to predict a high-resolution speech signal from its low-resolution counterpart. The previous models usually perform this task at a fixed sampling rate, reconstructing only high-frequency spectrogram components and merging them with low-frequency ones in noise-free cases. These methods achieve high accuracy, but they are less effective in real-world settings, where ambient noise and flexible sampling rates are presented. To develop a robust model that fits practical applications, in this work, we introduce Super Denoise Net (SDNet), a neural network for noise-robust super-resolution with flexible input sampling rates. To this end, SDNet's design includes gated and lattice convolution blocks for enhanced repair and temporal-spectral information capture. The frequency transform blocks are employed to model long frequency dependencies, and a multi-scale discriminator is proposed to facilitate the multi-adversarial loss training. The experiments show that SDNet outperforms current state-of-the-art noise-robust speech super-resolution models on multiple test sets, indicating its robustness and effectiveness in real-world scenarios.

Simultaneous single image super‐resolution and blind Gaussian denoising via slim ghost full‐frequency residual blocks

AbstractGiven that super‐resolution (SR) aims to recover lost information, and low‐resolution (LR) images in real‐world conditions might be corrupted with multiple degradations, considering basic bicubic down‐sampling as the sole degradation significantly limits the performance of most existing SR models. This paper presents a model for simultaneous super‐resolution and blind additive white Gaussian noise (AWGN) denoising with two components (netdeg and netSR) that is based on a generative adversarial network (GAN) to achieve detailed results. netdeg, featuring residual and innovative cost‐effective ghost residual blocks with a frequency separation module for obtaining long‐range information, blindly restores a clean version of the LR image. netSR leverages slim ghost full‐frequency residual blocks to process low‐frequency (LF) and high‐frequency (HF) information via static large convolutions and pixel‐wise highlighted input‐adaptive dynamic convolutions, respectively. To address the susceptibility of dynamic layers to noise and preserve feature diversity while reducing model’s costs, static and dynamic layer features are combined and highlighted. Diverse and non‐redundant features are then processed using ghost‐style blocks. The proposed model achieves comparable SR results in bicubic down‐sampling scenarios, outperform existing SR methods in the complex task of concurrent SR and AWGN denoising, and demonstrate robustness in handling images corrupted with varying levels of AWGN.

Enhancement of Image Quality in Low-Field Knee MR Imaging Using Deep Learning.

The purpose of this study is to investigate the potential of deep learning (DL) techniques to enhance the image quality of low-field knee MR images, with the ultimate goal of approximating the standards ofhigh-field knee MR imaging. We analyzed knee MR images collected from 45 patients with knee disorders and six normal subjects using a 3T MR scannerand those collected from 25 patients with knee disorders using a 0.4T MR scanner. Two DL models were developed: a fat-suppression contrast-generation model and a super-resolution model. These DL models were trained using 3T knee MR imaging data and applied to 0.4T knee MR imaging data. Visual assessments of anatomical structures and image noise and abnormality detection with diagnostic confidence levels on the original 0.4T MR images and those afterDL enhancement were conducted by two board-certified radiologists. Statistical analyses were performed using McNemar's test and the Wilcoxon signed-rank test. DL-enhanced MR images significantly improved the depiction of anatomical structures and reduced image noise compared to the original MR images. The number of abnormal findings detected and the diagnostic confidence levels were higher in the DL-enhanced MR images, indicating the potential for more accurate diagnoses. DL techniques effectively enhance the image quality of low-field knee MR images by leveraging 3T MR imaging data. This enhancement significantly improves image quality and diagnostic confidence levels, making low-field MR images much more reliable for detecting abnormalities. This advancement offers a useful alternative for clinical settings, especially in resource-limited environments, without compromising diagnostic accuracy.

OGRN: optical-guided residual dense network for SAR image super-resolution reconstruction network

ABSTRACT Although Convolutional Neural Networks have significantly improved the development of SAR image super-resolution (SR) technology in recent years, it is a very challenging problem to reconstruct SAR image with large-scale factors, such as ×4 and ×8 due to limited available information from low-resolution image. The co-registered high-resolution optical image has been successfully applied to enhance the quality of SAR image due to its discriminative characteristics. Compared with single-frame SAR image SR reconstruction technology, optical image-guided SAR image SR reconstruction technology has better performance. This paper proposes an optical-guided residual dense network (OGRN) for SAR image super-resolution reconstruction network. The network is an end-to-end network. Its most important characteristic is that the SR model of SAR images can be obtained through optical image assistance during training, and high-quality SAR images can be generated without optical image assistance during testing. Firstly, a new dense residual backbone network that extracts high-frequency information is constructed. It can extract and propagate high-frequency features of SAR images efficiently to improve the feature representation ability of the network for high-frequency features. Then, the extracted high-frequency features of the SAR image are used to generate an optical image through the designed SAR-to-Optical image translation network. Finally, a reconstruction module is constructed based on the proposed fusion module for optical and SAR images. The fusion module fuses the high-frequency feature information of the extracted optical images with the SAR image features to provide features for SAR image SR reconstruction. Extensive experiments conducted on Sen1-2 and QXS datasets demonstrated that under the guidance of optical image, our OGRN can achieve excellent performance in both quantitative assessment metrics and visual quality.

Integrating hydrological knowledge into deep learning for DEM super-resolution

Deep learning-based super-resolution methods have been successfully applied to digital elevation model (DEM) downscaling studies by designing structures and loss functions of the model. However, little attention has been paid to the design of super-resolution models that can maintain the hydrological characteristics of the DEM, which is important for hydrological studies. This study introduces a super-resolution model that integrates hydrologic knowledge (HKSRCGAN), with the aim to effectively maintain topographic features as well as the hydrologic connectivity of the DEM. The hydrological knowledge derived from surface flow direction and hydrological features are integrated into a deep learning algorithm to guide model training. The 30 m spatial resolution FABDEM is used to demonstrate the utility of the proposed method. Results show that the HKSRCGAN outperforms the bicubic interpolation, SRCNN, SRGAN, SRResNet and TfaSR methods in reducing topographic errors and maintaining hydrologic characteristics. In the test area, the entropy difference analysis shows that the DEM generated by HKSRCGAN is similar to the information contained in the reference DEM. Furthermore, super-resolution models integrating hydrological knowledge are valuable for modeling terrain primarily shaped by gravity and surface water flows. In the future, deep learning-based models integrating hydrologic knowledge are expected to be applied in DEM upscaling to maintain consistent hydrological characteristics.

Evaluation of super resolution technology for digestive endoscopic images

ObjectThis study aims to evaluate the value of super resolution (SR) technology in augmenting the quality of digestive endoscopic images. MethodsIn the retrospective study, we employed two advanced SR models, i.e., SwimIR and ESRGAN. Two discrete datasets were utilized, with training conducted using the dataset of the First Affiliated Hospital of Soochow University (12,212 high-resolution images) and evaluation conducted using the HyperKvasir dataset (2,566 low-resolution images). Furthermore, an assessment of the impact of enhanced low-resolution images was conducted using a 5-point Likert scale from the perspectives of endoscopists. Finally, two endoscopic image classification tasks were employed to evaluate the effect of SR technology on computer vision (CV). ResultsSwinIR demonstrated superior performance, which achieved a PSNR of 32.60, an SSIM of 0.90, and a VIF of 0.47 in test set. 90 % of endoscopists supported that SR preprocessing moderately ameliorated the readability of endoscopic images. For CV, enhanced images bolstered the performance of convolutional neural networks, whether in the classification task of Barrett's esophagus (improved F1-score: 0.04) or Mayo Endoscopy Score (improved F1-score: 0.04). ConclusionsSR technology demonstrates the capacity to produce high-resolution endoscopic images. The approach enhanced clinical readability and CV models’ performance of low-resolution endoscopic images.

Spectral Super-Resolution for High-Accuracy Rice Variety Classification using Hybrid CNN-Transformer Model

Rice variety classification is crucial for ensuring the purity and quality of rice production. In this study, hyperspectral images (HSI) of rice varieties were reconstructed using a spectral super-resolution technique, serving as a basis for an enhanced classification process. RGB images of various rice varieties were utilized to reconstruct the HSI, and a hybrid CNN-Transformer model featuring an efficient feature fusion module aimed at reducing redundancy was developed. Building on the spectral super-resolution model, a Swin-Transformer model was employed for rice variety classification, chosen for its capacity to effectively handle high-dimensional data with fewer parameters and lower FLOPs. The classification accuracy based on actual HSI reached an impressive 99.961%, while the accuracy based on reconstructed HSI was a close 99.413%. These results demonstrate that the spectral super-resolution method not only effectively reconstructs HSI images of rice varieties but also supports highly reliable classification. It is indicated by the study that spectral super-resolution can significantly outperform advanced CNN and Transformer-based methods in terms of accuracy and computational efficiency, offering a promising approach for precise rice variety classification and potentially other agricultural applications.

Dual-Path Large Kernel Learning and Its Applications in Single-Image Super-Resolution.

To enhance the performance of super-resolution models, neural networks frequently employ module stacking. However, this approach inevitably results in an excessive proliferation of parameter counts and information redundancy, ultimately constraining the deployment of these models on mobile devices. To surmount this limitation, this study introduces the application of Dual-path Large Kernel Learning (DLKL) to the task of image super-resolution. Within the DLKL framework, we harness a multiscale large kernel decomposition technique to efficiently establish long-range dependencies among pixels. This network not only maintains excellent performance but also significantly mitigates the parameter burden, achieving an optimal balance between network performance and efficiency. When compared with other prevalent algorithms, DLKL exhibits remarkable proficiency in generating images with sharper textures and structures that are more akin to natural ones. It is particularly noteworthy that on the challenging texture dataset Urban100, the network proposed in this study achieved a significant improvement in Peak Signal-to-Noise Ratio (PSNR) for the ×4 upscaling task, with an increase of 0.32 dB and 0.19 dB compared with the state-of-the-art HAFRN and MICU networks, respectively. This remarkable result not only validates the effectiveness of the present model in complex image super-resolution tasks but also highlights its superior performance and unique advantages in the field.

Retraction Note: High dimensional image super-resolution model based on infrared spectral imaging for aerobics training simulation monitoring

Retraction Note: High dimensional image super-resolution model based on infrared spectral imaging for aerobics training simulation monitoring

Lightweight image super-resolution with sliding Proxy Attention Network

Recently, image super-resolution (SR) models using window-based Transformers have demonstrated superior performance compared to SR models based on convolutional neural networks. Nevertheless, Transformer-based SR models often entail high computational demands. This is due to the adoption of shifted window self-attention following the window self-attention layer to model long-range relationships, resulting in additional computational overhead. Moreover, extracting local image features only with the self-attention mechanism is insufficient to reconstruct rich high-frequency image content. To overcome these challenges, we propose the Sliding Proxy Attention Network (SPAN), capable of recovering high-quality High-Resolution (HR) images from Low-Resolution (LR) inputs with substantially fewer model parameters and computational operations. The primary innovation of SPAN lies in the Sliding Proxy Transformer Block (SPTB), integrating the local detail sensitivity of convolution with the long-range dependency modeling of self-attention mechanism. Key components within SPTB include the Enhanced Local Feature Extraction Block (ELFEB) and the Sliding Proxy Attention Block (SPAB). ELFEB is designed to enhance the local receptive field with lightweight parameters for high-frequency details compensation. SPAB optimizes computational efficiency by implementing intra-window and cross-window attention in a single operation through leveraging window overlap. Experimental results demonstrate that SPAN can produce high-quality SR images while effectively managing computational complexity. The code is publicly available at: https://github.com/zononhzy/SPAN.