Image Super-resolution Task Research Articles

Towards Lightweight Super-Resolution With Dual Regression Learning.

Deep neural networks have exhibited remarkable performance in image super-resolution (SR) tasks by learning a mapping from low-resolution (LR) images to high-resolution (HR) images. However, the SR problem is typically an ill-posed problem and existing methods would come with several limitations. First, the possible mapping space of SR can be extremely large since there may exist many different HR images that can be super-resolved from the same LR image. As a result, it is hard to directly learn a promising SR mapping from such a large space. Second, it is often inevitable to develop very large models with extremely high computational cost to yield promising SR performance. In practice, one can use model compression techniques to obtain compact models by reducing model redundancy. Nevertheless, it is hard for existing model compression methods to accurately identify the redundant components due to the extremely large SR mapping space. To alleviate the first challenge, we propose a dual regression learning scheme to reduce the space of possible SR mappings. Specifically, in addition to the mapping from LR to HR images, we learn an additional dual regression mapping to estimate the downsampling kernel and reconstruct LR images. In this way, the dual mapping acts as a constraint to reduce the space of possible mappings. To address the second challenge, we propose a dual regression compression (DRC) method to reduce model redundancy in both layer-level and channel-level based on channel pruning. Specifically, we first develop a channel number search method that minimizes the dual regression loss to determine the redundancy of each layer. Given the searched channel numbers, we further exploit the dual regression manner to evaluate the importance of channels and prune the redundant ones. Extensive experiments show the effectiveness of our method in obtaining accurate and efficient SR models.

Blind Super-Resolution via Meta-Learning and Markov Chain Monte Carlo Simulation.

Learning based approaches have witnessed great successes in blind single image super-resolution (SISR) tasks, however, handcrafted kernel priors and learning based kernel priors are typically required. In this paper, we propose a meta-learning and Markov Chain Monte Carlo (MCMC) based SISR approach to learn kernel priors from organized randomness. In concrete, a lightweight network is adopted as kernel generator, and is optimized via learning from the MCMC simulation on random Gaussian distributions. This procedure provides an approximation for the rational blur kernel, and introduces a network-level Langevin dynamics into SISR optimization processes, which contributes to preventing bad local optimal solutions for kernel estimation. Meanwhile, a meta-learning based alternating optimization procedure is proposed to optimize the kernel generator and image restorer, respectively. In contrast to the conventional alternating minimization strategy, a meta-learning based framework is applied to learn an adaptive optimization strategy, which is less-greedy and results in better convergence performance. These two procedures are iteratively processed in a plug-and-play fashion, for the first time, realizing a learning-based but plug-and-play blind SISR solution in unsupervised inference. Extensive simulations demonstrate the superior performance and generalization ability of the proposed approach when compared with the Start-of-the-Art solutions on synthesis and real-world datasets.

DESAT: A Distance-Enhanced Strip Attention Transformer for Remote Sensing Image Super-Resolution

Transformer-based methods have demonstrated impressive performance in image super-resolution tasks. However, when applied to large-scale Earth observation images, the existing transformers encounter two significant challenges: (1) insufficient consideration of spatial correlation between adjacent ground objects; and (2) performance bottlenecks due to the underutilization of the upsample module. To address these issues, we propose a novel distance-enhanced strip attention transformer (DESAT). The DESAT integrates distance priors, easily obtainable from remote sensing images, into the strip window self-attention mechanism to capture spatial correlations more effectively. To further enhance the transfer of deep features into high-resolution outputs, we designed an attention-enhanced upsample block, which combines the pixel shuffle layer with an attention-based upsample branch implemented through the overlapping window self-attention mechanism. Additionally, to better simulate real-world scenarios, we constructed a new cross-sensor super-resolution dataset using Gaofen-6 satellite imagery. Extensive experiments on both simulated and real-world remote sensing datasets demonstrate that the DESAT outperforms state-of-the-art models by up to 1.17 dB along with superior qualitative results. Furthermore, the DESAT achieves more competitive performance in real-world tasks, effectively balancing spatial detail reconstruction and spectral transform, making it highly suitable for practical remote sensing super-resolution applications.

Deep learning enables the use of ultra-high-density array in DNBSEQ.

DNBSEQ employs a patterned array to facilitate massively parallel sequencing of DNA nanoballs (DNBs), leading to a considerable boost in throughput. By employing the ultra-high-density (UHD) array with an increased density of DNB binding sites, the throughput of DNBSEQ can be further expanded. However, the typical imaging system of the DNBSEQ sequencer is unable to resolve adjacent DNBs spaced smaller than the resolution limit, resulting in poor base-calling performance of the UHD array and hindering its practical application. In this study, we propose a deep-learning-based DNB image super-resolution network named DNBSRN to address this problem. DNBSRN has a specifically designed structure for DNB images and employs a histogram-matching-based preprocessing approach. For the eight DNB image datasets generated from the DNBSEQ sequencer using UHD arrays with 360nm pitch, the base-calling performances are significantly improved after super-resolution reconstruction by DNBSRN and reached a comparable level to those of the regular density array. In terms of reconstruction speed, DNBSRN takes only 7.61ms for an input image with 500 × 500 pixels, which minimizes its influence on throughput. Furthermore, compared with state-of-the-art super-resolution networks, DNBSRN demonstrates superior performance in terms of both the quality and speed of DNB image reconstruction. DNBSRN successfully addresses the DNB image super-resolution task. Integrating DNBSRN into the image analysis workflow of DNBSEQ will allow for the application of UHD array, hence enabling a considerable improvement in throughput as well as tremendous savings in unit reagent cost.

Ablation Study of Diffusion Model with Transformer Backbone for Low-count PET Denoising.

Diffusion models (DM) built from a hierarchy of denoising autoencoders have achieved remarkable progress in image generation, and are increasingly influential in the field of image restoration (IR) tasks. In the meantime, its backbone of autoencoders also evolved from UNet to vision transformer, e.g. Restormer. Therefore, it is important to disentangle the contribution of backbone networks and the additional generative learning scheme. Notably, DM shows varied performance across IR tasks, and the performance of recent advanced transformer-based DM on PET denoising is under-explored. In this study, we further raise an intuitive question, "{if we have a sufficiently powerful backbone, whether DM can be a general add-on generative learning scheme to further boost PET denoising}". Specifically, we investigate one of the best-in-class IR models, i.e., DiffIR, which is a latent DM based on the Restormer backbone. We provide a qualitative and quantitative comparison with UNet, SR3 (UNet+pixel DM), and Restormer, on the 25% low dose 18F-FDG whole-body PET denoising task, aiming to identify the best practices. We trained and tested on 93 and 12 subjects, and each subject has 644 slices. It appears that Restormer outperforms UNet in terms of PSNR and MSE. However, additional latent DM over Restormer does not contribute to better MSE, SSIM, or PSNR in our task, which is even inferior to the conventional UNet. In addition, SR3 with pixel space DM is not stable to synthesize satisfactory results. The results are consistent with the natural image super-resolution tasks, which also suffer from limited spatial information. A possible reason would be the denoising iteration at latent feature space cannot well support detailed structure and texture restoration. This issue is more crucial in the IR tasks taking inputs with limited details, e.g., SR and PET denoising.

Spectral domain strategies for hyperspectral super-resolution: Transfer learning and channel enhance network

As the network structures continue to innovate and evolve, significant achievements have been achieved in hyperspectral image super-resolution tasks. However, how to further explore the spectral domain potential from prior knowledge and channel-enhanced structures to achieve better performance has inspired the following two works: Firstly, to systematically compare prior knowledge of spectral with spatial domain for HSI-SR tasks, four transfer learning strategies are proposed. The superior performance of the Relevant Channel/Random Space (RCRS) strategy reveals the importance of spectral feature reconstruction in HSI-SR tasks. Meanwhile, an interesting phenomenon has been observed that even without training on real datasets, the model can already exhibit a core or even decent super-resolution capability based solely on prior knowledge of above four strategies. Secondly, a dual-branch channel network with complementary channel feature extraction (CCFE) and adjacent channel feature extraction (ACFE) module is designed for spectral feature enhancement, which demonstrate superior performance compared to state-of-the-art methods on six datasets. To conclude, the effectiveness of RCRS strategy with pseudo prior channel knowledge on seven dual-input and eight single-input networks, as well as superiority of the proposed channel-enhanced network indicate the importance of spectral properties for HSI-SR tasks.

Dual-Path Large Kernel Learning and Its Applications in Single-Image Super-Resolution.

To enhance the performance of super-resolution models, neural networks frequently employ module stacking. However, this approach inevitably results in an excessive proliferation of parameter counts and information redundancy, ultimately constraining the deployment of these models on mobile devices. To surmount this limitation, this study introduces the application of Dual-path Large Kernel Learning (DLKL) to the task of image super-resolution. Within the DLKL framework, we harness a multiscale large kernel decomposition technique to efficiently establish long-range dependencies among pixels. This network not only maintains excellent performance but also significantly mitigates the parameter burden, achieving an optimal balance between network performance and efficiency. When compared with other prevalent algorithms, DLKL exhibits remarkable proficiency in generating images with sharper textures and structures that are more akin to natural ones. It is particularly noteworthy that on the challenging texture dataset Urban100, the network proposed in this study achieved a significant improvement in Peak Signal-to-Noise Ratio (PSNR) for the ×4 upscaling task, with an increase of 0.32 dB and 0.19 dB compared with the state-of-the-art HAFRN and MICU networks, respectively. This remarkable result not only validates the effectiveness of the present model in complex image super-resolution tasks but also highlights its superior performance and unique advantages in the field.

Inf-OSRGAN: Optimized Blind Super-Resolution GAN for Infrared Images

With the widespread application of infrared technology in military, security, medical, and other fields, the demand for high-definition infrared images has been increasing. However, the complexity of the noise introduced during the imaging process and high acquisition costs limit the scope of research on super-resolution algorithms for infrared images, particularly when compared to the visible light domain. Furthermore, the lack of high-quality infrared image datasets poses challenges in algorithm design and evaluation. To address these challenges, this paper proposes an optimized super-resolution algorithm for infrared images. Firstly, we construct an infrared image super-resolution dataset, which serves as a robust foundation for algorithm design and rigorous evaluation. Secondly, in the degradation process, we introduce a gate mechanism and random shuffle to enrich the degradation space and more comprehensively simulate the real-world degradation of infrared images. We train an RRDBNet super-resolution generator integrating the aforementioned degradation model. Additionally, we incorporate spatially correlative loss to leverage spatial–structural information, thereby enhancing detail preservation and reconstruction in the super-resolution algorithm. Through experiments and evaluations, our method achieved considerable performance improvements in the infrared image super-resolution task. Compared to traditional methods, our method was able to better restore the details and clarity of infrared images.

Multi-dimensional attention-aided transposed ConvBiLSTM network for hyperspectral image super-resolution

Hyperspectral (HS) image always suffers from the deficiency of low spatial resolution, compared with conventional optical image types, which has limited its further applications in remote sensing areas. Therefore, HS image super-resolution (SR) techniques are broadly employed in order to observe finer spatial structures while preserving the spectra of ground covers. In this paper, a novel multi-dimensional attention-aided transposed convolutional long-short term memory (LSTM) network is proposed for single HS image super-resolution task. The proposed network employs the convolutional bi-directional LSTM for the purpose of local and non-local spatial–spectral feature explorations, and transposed convolution for the purpose of image amplification and reconstruction. Moreover, a multi-dimensional attention module is proposed, aiming to capture the salient features on spectral, channel, and spatial dimensions, simultaneously, to further improve the learning abilities of network. Experiments on four commonly-used HS images demonstrate the effectiveness of this approach, compared with several state-of-the-art deep learning-based SR methods.

Hybrid attention transformer with re-parameterized large kernel convolution for image super-resolution

Single image super-resolution is a well-established low-level vision task that aims to reconstruct high-resolution images from low-resolution images. Methods based on Transformer have shown remarkable success and achieved outstanding performance in SISR tasks. While Transformer effectively models global information, it is less effective at capturing high frequencies such as stripes that primarily provide local information. Additionally, it has the potential to further enhance the capture of global information. To tackle this, we propose a novel Large Kernel Hybrid Attention Transformer using re-parameterization. It combines different kernel sizes and different steps re-parameterized convolution layers with Transformer to effectively capture global and local information to learn comprehensive features with low-frequency and high-frequency information. Moreover, in order to solve the problem of using batch normalization layer to introduce artifacts in SISR, we propose a new training strategy which is fusing convolution layer and batch normalization layer after certain training epochs. This strategy can enjoy the acceleration convergence effect of batch normalization layer in training and effectively eliminate the problem of artifacts in the inference stage. For re-parameterization of multiple parallel branch convolution layers, adopting this strategy can further reduce the amount of calculation of training. By coupling these core improvements, our LKHAT achieves state-of-the-art performance for single image super-resolution task.

Deep Learning and Neural Architecture Search for Optimizing Binary Neural Network Image Super Resolution.

The evolution of super-resolution (SR) technology has seen significant advancements through the adoption of deep learning methods. However, the deployment of such models by resource-constrained devices necessitates models that not only perform efficiently, but also conserve computational resources. Binary neural networks (BNNs) offer a promising solution by minimizing the data precision to binary levels, thus reducing the computational complexity and memory requirements. However, for BNNs, an effective architecture is essential due to their inherent limitations in representing information. Designing such architectures traditionally requires extensive computational resources and time. With the advancement in neural architecture search (NAS), differentiable NAS has emerged as an attractive solution for efficiently crafting network structures. In this paper, we introduce a novel and efficient binary network search method tailored for image super-resolution tasks. We adapt the search space specifically for super resolution to ensure it is optimally suited for the requirements of such tasks. Furthermore, we incorporate Libra Parameter Binarization (Libra-PB) to maximize information retention during forward propagation. Our experimental results demonstrate that the network structures generated by our method require only a third of the parameters, compared to conventional methods, and yet deliver comparable performance.

Super-resolution reconstruction of ultrasound image using a modified diffusion model

Objective. This study aims to perform super-resolution (SR) reconstruction of ultrasound images using a modified diffusion model, designated as the diffusion model for ultrasound image super-resolution (DMUISR). SR involves converting low-resolution images to high-resolution ones, and the proposed model is designed to enhance the suitability of diffusion models for this task in the context of ultrasound imaging. Approach. DMUISR incorporates a multi-layer self-attention (MLSA) mechanism and a wavelet-transform based low-resolution image (WTLR) encoder to enhance its suitability for ultrasound image SR tasks. The model takes interpolated and magnified images as input and outputs high-quality, detailed SR images. The study utilized 1,334 ultrasound images from the public fetal head-circumference dataset (HC18) for evaluation. Main results. Experiments were conducted at 2× , 4× , and 8× magnification factors. DMUISR outperformed mainstream ultrasound SR methods (Bicubic, VDSR, DECUSR, DRCN, REDNet, SRGAN) across all scales, providing high-quality images with clear structures and rich detailed textures in both hard and soft tissue regions. DMUISR successfully accomplished multiscale SR reconstruction while suppressing over-smoothing and mode collapse problems. Quantitative results showed that DMUISR achieved the best performance in terms of learned perceptual image patch similarity, with a significant decrease of over 50% at all three magnification factors (2× , 4× , and 8× ), as well as improvements in peak signal-to-noise ratio and structural similarity index measure. Ablation experiments validated the effectiveness of the MLSA mechanism and WTLR encoder in improving DMUISR’s SR performance. Furthermore, by reducing the number of diffusion steps, the computational time of DMUISR was shortened to nearly one-tenth of its original while maintaining image quality without significant degradation. Significance. This study demonstrates that the modified diffusion model, DMUISR, provides superior performance for SR reconstruction of ultrasound images and has potential in improving imaging quality in the medical ultrasound field.

PlantSR: Super-Resolution Improves Object Detection in Plant Images.

Recent advancements in computer vision, especially deep learning models, have shown considerable promise in tasks related to plant image object detection. However, the efficiency of these deep learning models heavily relies on input image quality, with low-resolution images significantly hindering model performance. Therefore, reconstructing high-quality images through specific techniques will help extract features from plant images, thus improving model performance. In this study, we explored the value of super-resolution technology for improving object detection model performance on plant images. Firstly, we built a comprehensive dataset comprising 1030 high-resolution plant images, named the PlantSR dataset. Subsequently, we developed a super-resolution model using the PlantSR dataset and benchmarked it against several state-of-the-art models designed for general image super-resolution tasks. Our proposed model demonstrated superior performance on the PlantSR dataset, indicating its efficacy in enhancing the super-resolution of plant images. Furthermore, we explored the effect of super-resolution on two specific object detection tasks: apple counting and soybean seed counting. By incorporating super-resolution as a pre-processing step, we observed a significant reduction in mean absolute error. Specifically, with the YOLOv7 model employed for apple counting, the mean absolute error decreased from 13.085 to 5.71. Similarly, with the P2PNet-Soy model utilized for soybean seed counting, the mean absolute error decreased from 19.159 to 15.085. These findings underscore the substantial potential of super-resolution technology in improving the performance of object detection models for accurately detecting and counting specific plants from images. The source codes and associated datasets related to this study are available at Github.

DHBSR: A deep hybrid representation-based network for blind image super resolution

Image super resolution involves enhancing the spatial resolution of low-quality images and improving their visual quality. As in many real-life situations, the image degradation process is unknown, performing the task of image super resolution in a blind manner is of paramount importance. Deep neural networks provide high performances for the task of blind image super resolution, in view of their end-to-end learning capability between the low-resolution images and their ground truth versions. Generally speaking, deep blind image super resolution networks initially estimate the parameters of the image degradation process, such as blurring kernel, and then use them for super-resolving the low-resolution images. In this paper, we develop a novel deep learning-based scheme for the task of blind image super resolution, in which the idea of leveraging the hybrid representations is utilized. Specifically, we employ the deterministic and stochastic representations of the blurring kernel parameters to train a deep blind super resolution network in an effective manner. The results of extensive experiments prove the effectiveness of various ideas used in the development of the proposed deep blind image super resolution network.

Lightweight interactive feature inference network for single-image super-resolution

The emergence of convolutional neural network (CNN) and transformer has recently facilitated significant advances in image super-resolution (SR) tasks. However, these networks commonly construct complex structures, having huge model parameters and high computational costs, to boost reconstruction performance. In addition, they do not consider the structural prior well, which is not conducive to high-quality image reconstruction. In this work, we devise a lightweight interactive feature inference network (IFIN), complementing the strengths of CNN and Transformer, for effective image SR reconstruction. Specifically, the interactive feature aggregation module (IFAM), implemented by structure-aware attention block (SAAB), Swin Transformer block (SWTB), and enhanced spatial adaptive block (ESAB), serves as the network backbone, progressively extracts more dedicated features to facilitate the reconstruction of high-frequency details in the image. SAAB adaptively recalibrates local salient structural information, and SWTB effectively captures rich global information. Further, ESAB synergetically complements local and global priors to ensure the consistent fusion of diverse features, achieving high-quality reconstruction of images. Comprehensive experiments reveal that our proposed networks attain state-of-the-art reconstruction accuracy on benchmark datasets while maintaining low computational demands. Our code and results are available at: https://github.com/wwaannggllii/IFIN.

Non-local sparse attention based swin transformer V2 for image super-resolution

In single image super resolution tasks, distortion measurement (such as PSNR, SSIM) and perceptual quality (such as PI, NIQE) are contradictory, and methods that perform well in perceptual quality often perform poorly in distortion measurement, and vice versa. In this article, we propose a method of balancing the two, which is divided into three stages. Firstly, this article proposes an image super-resolution model NLSAV2 that focuses on PSNR and SSIM metrics. The entire NLSAV2 consists of three modules: shallow feature extraction, deep feature extraction, and high-quality image reconstruction. In the shallow feature extraction module, non-local sparse attention is used to identify the most abundant feature information in mapping input from low-dimensional space to high-dimensional space, and the deep feature extraction module mainly consists of residual Swin Transformer V2 Block. Then, NLSAV2 is used as a generator and a relative discriminator is introduced to further train the model, which is called NLSAV2-GAN. The experimental results indicate that NLSAV2 and NLSAV2-GAN exhibit advantages in distortion measurement and perceptual quality respectively. Finally, network interpolation and image interpolation strategies are used to continuously adjust the reconstruction style and smoothness to achieve a balance between the two.

Multi-attention fusion transformer for single-image super-resolution

Recently, Transformer-based methods have gained prominence in image super-resolution (SR) tasks, addressing the challenge of long-range dependence through the incorporation of cross-layer connectivity and local attention mechanisms. However, the analysis of these networks using local attribution maps has revealed significant limitations in leveraging the spatial extent of input information. To unlock the inherent potential of Transformer in image SR, we propose the Multi-Attention Fusion Transformer (MAFT), a novel model designed to integrate multiple attention mechanisms with the objective of expanding the number and range of pixels activated during image reconstruction. This integration enhances the effective utilization of input information space. At the core of our model lies the Multi-attention Adaptive Integration Groups, which facilitate the transition from dense local attention to sparse global attention through the introduction of Local Attention Aggregation and Global Attention Aggregation blocks with alternating connections, effectively broadening the network's receptive field. The effectiveness of our proposed algorithm has been validated through comprehensive quantitative and qualitative evaluation experiments conducted on benchmark datasets. Compared to state-of-the-art methods (e.g. HAT), the proposed MAFT achieves 0.09 dB gains on Urban100 dataset for × 4 SR task while containing 32.55% and 38.01% fewer parameters and FLOPs, respectively.

Rethinking Regularization with Random Label Smoothing

Regularization helps to improve machine learning techniques by penalizing the models during training. Such approaches act in either the input, internal, or output layers. Regarding the latter, label smoothing is widely used to introduce noise in the label vector, making learning more challenging. This work proposes a new label regularization method, Random Label Smoothing, that attributes random values to the labels while preserving their semantics during training. The idea is to change the entire label into fixed arbitrary values. Results show improvements in image classification and super-resolution tasks, outperforming state-of-the-art techniques for such purposes.

A boosted degradation representation learning for blind image super-resolution

The significant gap between the assumed and actual degradation model will undoubtedly lead to a serious performance decrease for deep learning based image super-resolution (SR) methods. To solve this shortcoming, this paper presents an enhanced blind image super-resolution (EBSR) network based on unsupervised degradation representation learning. The proposed EBSR mainly is made up of two branching subnetworks. The first subnetwork is a residual-based degradation encoder, which is responsible to learn a high-dimensional abstract degradation representation vector from the input low-resolution (LR) image patches using contrast learning. Another branch is the degradation-aware fusion SR (DAFSR) network that completes the image SR task with the aid of the degradation representation vector learned by the encoder. To obtain an improved SR performance under various degradation settings, the degradation-aware fusion block (DAFB), the core block of DAFSR, has been elaborated, which embeds the degradation model information learned by the encoder into every layer of the network. Additionally, a shallow feature extraction block (SFEB) is constructed for DAFSR to extract global information from the LR image. The proposed EBSR can flexibly adapt to various degradation models. Extensive experimental results on synthetic datasets and real images show that the proposed EBSR is able to achieve a leading reconstruction performance over other blind SR algorithms.

Lightweight blueprint residual network for single image super-resolution

The application of deep convolutional neural networks (CNNs) makes the lightweight single image super-resolution (SISR) task develop rapidly in recent years. However, existing lightweight SISR networks for the terminals with limited computing resources are still not efficient enough. In order to further alleviate the model complexity while keeping remarkable reconstruction performance, in this paper, we propose a lightweight blueprint residual network (LBRN), which is a powerful CNN-based SR model while simultaneously maintaining extremely lightweight. Specifically, we design a novel lightweight blueprint residual block (LBRB) to learn the high-frequency information efficiently for lightweight SISR. The LBRB with fewer parameters consists of more efficient convolution operations to extract high-frequency information which helps recover more visual details. In addition, we propose an effective feature fusion residual block (EFFRB) which is composed of three LBRBs with convolutional kernels of different sizes. The EFFRB, which is designed with effective multi-scale receptive fields, can availably extract and fuse multi-scale high-frequency information to obtain discriminative feature, and it further improves model performance. Extensive experiments show that our LBRN achieves superior reconstruction performance quantitatively and visually compared with other lightweight state-of-the-art SR methods.