Single-image Super-resolution Algorithm Research Articles

Local Means Binary Networks for Image Super-Resolution.

The success of modern single image super-resolution (SISR) algorithms is inspired by the development of deep convolutional neural networks (CNNs). However, these CNN-based methods require considerable computation and complexity, making it impossible for these methods to perform real-time calculations in edge devices. Thus, lightweight model design has become a development trend in the super-resolution field, including pruning, quantization, and other methods. The 1-bit quantization is an extreme lightweight method which can reduce the calculation amount of the model in an extreme manner and is friendly to hardware such as edge devices. Most existing binary quantization approaches lead to a large information loss during forward propagation, especially in detailed color information (e.g., edge, texture, and contrast). The loss of color information makes modern binary methods unsuitable for SISR tasks. We think the loss occurs because these methods typically utilize a uniform threshold to quantize the weights and activations. Thus, in this article, we thoroughly analyze the difference between normal classification tasks and SISR tasks, and present a binarization scheme based on local means. The proposed method can maintain more detailed information in feature maps using dynamic thresholds during quantization. Specifically, each value in the full precision activations has a corresponding threshold during the quantization process, and those thresholds are determined by the full precision values of the surroundings. In addition, a gradient approximator is introduced to adaptively optimize the gradient for updating binary weights. We then verify the effectiveness of our method for training binary networks on several SISR benchmarks including VDSR and SRResNet. Experimental results show that the proposed method can outperform the state-of-the-art algorithms to obtain binary networks for image super-resolution with better peak signal-to-noise ratio (PSNR) values and visual quality.

GSDD: Generative Space Dataset Distillation for Image Super-resolution

Single image super-resolution (SISR), especially in the real world, usually builds a large amount of LR-HR image pairs to learn representations that contain rich textural and structural information. However, relying on massive data for model training not only reduces training efficiency, but also causes heavy data storage burdens. In this paper, we attempt a pioneering study on dataset distillation (DD) for SISR problems to explore how data could be slimmed and compressed for the task. Unlike previous coreset selection methods which select a few typical examples directly from the original data, we remove the limitation that the selected data cannot be further edited, and propose to synthesize and optimize samples to preserve more task-useful representations. Concretely, by utilizing pre-trained GANs as a suitable approximation of realistic data distribution, we propose GSDD, which distills data in a latent generative space based on GAN-inversion techniques. By optimizing them to match with the practical data distribution in an informative feature space, the distilled data could then be synthesized. Experimental results demonstrate that when trained with our distilled data, GSDD can achieve comparable performance to the state-of-the-art (SOTA) SISR algorithms, while a nearly ×8 increase in training efficiency and a saving of almost 93.2% data storage space can be realized. Further experiments on challenging real-world data also demonstrate the promising generalization ability of GSDD.

A pluggable single-image super-resolution algorithm based on second-order gradient loss

Convolutional neural networks for single-image super-resolution have been widely used with great success. However, most of these methods use L1 loss to guide network optimization, resulting in blurry restored images with sharp edges smoothed. This is because L1 loss limits the optimization goal of the network to the statistical average of all solutions within the solution space of that task. To go beyond the L1 loss, this paper designs an image super-resolution algorithm based on second-order gradient loss. We impose additional constraints by considering the high-order gradient level of the image so that the network can focus on the recovery of fine details such as texture during the learning process. This helps to alleviate the problem of restored image texture over-smoothing to some extent. During network training, we extract the second-order gradient map of the generated image and the target image of the network by minimizing the distance between them, this guides the network to pay attention to the high-frequency detail information in the image and generate a high-resolution image with clearer edge and texture. Besides, the proposed loss function has good embeddability and can be easily integrated with existing image super-resolution networks. Experimental results show that the second-order gradient loss can significantly improve both Learned Perceptual Image Patch Similarity (LPIPS) and Frechet Inception Distance score (FID) performance over other image super-resolution deep learning models.

SR-GAN for SR-gamma: super resolution of photon calorimeter images at collider experiments

We study single-image super-resolution algorithms for photons at collider experiments based on generative adversarial networks. We treat the energy depositions of simulated electromagnetic showers of photons and neutral-pion decays in a toy electromagnetic calorimeter as 2D images and we train super-resolution networks to generate images with an artificially increased resolution by a factor of four in each dimension. The generated images are able to reproduce features of the electromagnetic showers that are not obvious from the images at nominal resolution. Using the artificially-enhanced images for the reconstruction of shower-shape variables and of the position of the shower center results in significant improvements. We additionally investigate the utilization of the generated images as a pre-processing step for deep-learning photon-identification algorithms and observe improvements in the case of training samples of small size.

A Two-Stage Three-Dimensional Attention Network for Lightweight Image Super-Resolution

In recent years, single image super-resolution (SISR) methods using convolutional neural networks (CNN) have achieved satisfactory performance. Nevertheless, the large model scale and the slow inference speed of these methods greatly limit the application scenarios. In this paper, we propose a two-stage three-dimensional attention network (ATTNet) for lightweight image super-resolution. First, we put forward the spatial feature encoder–decoder (SFE-D) with a spatial attention mechanism. Next, the channel transposed attention module (CTAM) with a channel self-attention mechanism is designed. Both the modules are used for fine feature extraction in the low-resolution stage. Finally, the content-based pixel recombination module (CPRM) is proposed to reconstruct the detailed content with a joint attention mechanism in the high-resolution stage. According to our experimental results, significant performance in terms of the quantitative metrics and the subjective visual quality can be achieved on average compared with the state-of-the-art lightweight SISR algorithms.

Super-Resolution Neural Network Models Comparison Metod for Brain MRI Images Based on PSNR and SSIM Metrics

The diagnosis of many diseases is largely possible thanks to MRI(Magnetic Resonance Imaging). This technology allows to study internal organs of the patient: the brain, spine, bones, joints, vessels and etc. The resolution of the MRI image is limited due to various factors: movement of the patient during the scan, the continuous movement of internal organs. The higher the quality of the MRI image, the longer it takes to scan. For more accurate diagnostics it is possible to increase resolution of the yielded images. This is achieved by using SISR(Single Image Super Resolution) algorithms, which allow you to obtain images with increased resolution from a single input image. In this paper the idea of the image super-resolution algorithms is presented, various forms of the problem and solutions to it are provided. The advantages of the SISR algorithms are described. The relevance of this task in the field of medical MRI images is explained. Metrics for comparing image quality PSNR and SSIM are given and described. A dataset for testing is presented. The stage of data preparation is described: the principle of selecting images from a set of datasets, converting data into the required format, compressing images to obtain input data for selected neural network models. The PSNR, SSIM metrics of two neural network models mDCSRN and FAWDN are measured on equally prepared input data. The comparison results are presented in the form of images and averaged data for the entire sample is stored in the table.

Multimodal-Boost: Multimodal Medical Image Super-Resolution Using Multi-Attention Network With Wavelet Transform.

Multimodal medical images are widely used by clinicians and physicians to analyze and retrieve complementary information from high-resolution images in a non-invasive manner. Loss of corresponding image resolution adversely affects the overall performance of medical image interpretation. Deep learning-based single image super resolution (SISR) algorithms have revolutionized the overall diagnosis framework by continually improving the architectural components and training strategies associated with convolutional neural networks (CNN) on low-resolution images. However, existing work lacks in two ways: i) the SR output produced exhibits poor texture details, and often produce blurred edges, ii) most of the models have been developed for a single modality, hence, require modification to adapt to a new one. This work addresses (i) by proposing generative adversarial network (GAN) with deep multi-attention modules to learn high-frequency information from low-frequency data. Existing approaches based on the GAN have yielded good SR results; however, the texture details of their SR output have been experimentally confirmed to be deficient for medical images particularly. The integration of wavelet transform (WT) and GANs in our proposed SR model addresses the aforementioned limitation concerning textons. While the WT divides the LR image into multiple frequency bands, the transferred GAN uses multi-attention and upsample blocks to predict high-frequency components. Additionally, we present a learning method for training domain-specific classifiers as perceptual loss functions. Using a combination of multi-attention GAN loss and a perceptual loss function results in an efficient and reliable performance. Applying the same model for medical images from diverse modalities is challenging, our work addresses (ii) by training and performing on several modalities via transfer learning. Using two medical datasets, we validate our proposed SR network against existing state-of-the-art approaches and achieve promising results in terms of structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR).

An efficient lightweight network for single image super-resolution

The outstanding performance of convolutional neural networks (CNNs) shown in single image super-resolution (SISR) strongly depends on network’s depth, which hampers its application in low-power computing devices. In this paper, a lightweight and efficient network (LESR) is proposed for SISR by constructing the shallow feature extraction block (SFBlock), the cascaded sparse mask blocks (SMBlocks) and the feature fusion block (FFBlock). The SFBlock efficiently extracts global informative features from the original low resolution image using sparse self-attention, SMBlock skips the redundant computation in extracted features, and more meaningful information is distilled for the sequential reconstruction block by the FFBlock. In addition, a recently proposed activation function called ACON-C is used to replace the ReLU function to ease the training difficulty. Extensive experiments show that our proposed network performs better than most advanced lightweight SISR algorithms with comparable parameters and less FLOPs on benchmark database for ×2/3/4 SISR.

An Arbitrary Scale Super-Resolution Approach for 3D MR Images via Implicit Neural Representation

High Resolution (HR) medical images provide rich anatomical structure details to facilitate early and accurate diagnosis. In magnetic resonance imaging (MRI), restricted by hardware capacity, scan time, and patient cooperation ability, isotropic 3-dimensional (3D) HR image acquisition typically requests long scan time and, results in small spatial coverage and low signal-to-noise ratio (SNR). Recent studies showed that, with deep convolutional neural networks, isotropic HR MR images could be recovered from low-resolution (LR) input via single image super-resolution (SISR) algorithms. However, most existing SISR methods tend to approach scale-specific projection between LR and HR images, thus these methods can only deal with fixed up-sampling rates. In this paper, we propose ArSSR, an Arbitrary Scale Super-Resolution approach for recovering 3D HR MR images. In the ArSSR model, the LR image and the HR image are represented using the same implicit neural voxel function with different sampling rates. Due to the continuity of the learned implicit function, a single ArSSR model is able to achieve arbitrary and infinite up-sampling rate reconstructions of HR images from any input LR image. Then the SR task is converted to approach the implicit voxel function via deep neural networks from a set of paired HR and LR training examples. The ArSSR model consists of an encoder network and a decoder network. Specifically, the convolutional encoder network is to extract feature maps from the LR input images and the fully-connected decoder network is to approximate the implicit voxel function. Experimental results on three datasets show that the ArSSR model can achieve state-of-the-art SR performance for 3D HR MR image reconstruction while using a single trained model to achieve arbitrary up-sampling scales.

Meta-Learning Multi-Scale Radiology Medical Image Super-Resolution

High-resolution medical images have important medical value, but are difficult to obtain directly. Limited by hardware equipment and patient’s physical condition, the resolution of directly acquired medical images is often not high. Therefore, many researchers have thought of using super-resolution algorithms for secondary processing to obtain high-resolution medical images. However, current super-resolution algorithms only work on a single scale, and multiple networks need to be trained when super-resolution images of different scales are needed. This definitely raises the cost of acquiring high-resolution medical images. Thus, we propose a multi-scale super-resolution algorithm using meta-learning. The algorithm combines a meta-learning approach with an enhanced depth of residual super-resolution network to design a meta-upscale module. The meta-upscale module utilizes the weight prediction property of meta-learning and is able to perform the super-resolution task of medical images at any scale. Meanwhile, we design a non-integer mapping relation for super-resolution, which allows the network to be trained under non-integer magnification requirements. Compared to the state-of-the-art single-image super-resolution algorithm on computed tomography images of the pelvic region. The meta-learning multiscale super-resolution algorithm obtained a surpassing of about 2% at a smaller model volume. Testing on different parts proves the high generalizability of our algorithm. Multi-scale super-resolution algorithms using meta-learning can compensate for hardware device defects and reduce secondary harm to patients while obtaining high-resolution medical images. It can be of great use in imaging related fields.

Super-Resolution Based on Curvelet Transform and Sparse Representation

Super-resolution techniques are used to reconstruct an image with a high resolution from one or more low-resolution image(s). In this paper, we proposed a single image super-resolution algorithm. It uses the nonlocal mean filter as a prior step to produce a denoised image. The proposed algorithm is based on curvelet transform. It converts the denoised image into low and high frequencies (sub-bands). Then we applied a multi-dimensional interpolation called Lancozos interpolation over both sub-bands. In parallel, we applied sparse representation with over complete dictionary for the denoised image. The proposed algorithm then combines the dictionary learning in the sparse representation and the interpolated sub-bands using inverse curvelet transform to have an image with a higher resolution. The experimental results of the proposed super-resolution algorithm show superior performance and obviously better-recovering images with enhanced edges. The comparison study shows that the proposed super-resolution algorithm outperforms the state-of-the-art. The mean absolute error is 0.021 ± 0.008 and the structural similarity index measure is 0.89 ± 0.08.

Image super-resolution with PCA reduced generalized Gaussian mixture models in materials science

Single Image Super-Resolution algorithms based on patches have been noticed and widely used over the past decade. Recently, generalized Gaussian mixture models (GGMMs) have been shown to be a suitable tool for many image processing problems because of the flexible shape parameter. In this work, we introduce a supervised GGMM-based approach for super-resolution of two- and three-dimensional images, in particular materials images. We first propose to use a joint GGMM learned from concatenated vectors of high- and low-resolution training patches. For each low-resolution patch, we compute the minimum mean square error (MMSE) estimator and generate the high-resolution image by averaging these estimates. We select the MMSE approach using GGMM as the method is invariant to affine contrast change and also invariant to a linear super-resolution operator. Unfortunately, the large dimension of the concatenated high- and low-resolution patches leads to instabilities and an intractable computational effort when estimating the parameters of the GGMM. Thus, we propose to combine a GGMM with a principal component analysis and derive an EM algorithm for estimating the parameters of the arising model. We demonstrate the performance of our model by numerical examples on synthetic and real images of material microstructure.

TMSO-Net: Texture adaptive multi-scale observation for light field image depth estimation

Light field can record the four-dimensional information of light rays, i.e. the position and direction information in which depth information is implied. To improve the depth estimation accuracy, we propose a depth estimation algorithm based on convolutional neural network (CNN). First, a single image super resolution algorithm is adopted to spatially super resolve the sub-aperture images (SAIs). Second, to adapt the texture complexity, the SAIs are partitioned into two regions, i.e., simple texture region and complex texture region, based on the texture analysis of the central SAI. Third, the epipolar plane images (EPIs) in horizontal, vertical, 45 degree diagonal, and 135 degree diagonal directions for both complex and simple texture regions are extracted, and the corresponding EPIs for the simple and complex texture regions are fed into the specified network branches. Finally, a fusion module is designed to generate the depth map. Experimental results show that the quality of the estimated depth maps by the proposed method is better than the state-of-the-art methods in terms of both objective quality and subjective quality. Moreover, the proposed method is more robust to noise.

Image Super-Resolution Reconstruction Method for Lung Cancer CT-Scanned Images Based on Neural Network.

The super-resolution (SR) reconstruction of a single image is an important image synthesis task especially for medical applications. This paper is studying the application of image segmentation for lung cancer images. This research work is utilizing the power of deep learning for resolution reconstruction for lung cancer-based images. At present, the neural networks utilized for image segmentation and classification are suffering from the loss of information where information passes through one layer to another deep layer. The commonly used loss functions include content-based reconstruction loss and generative confrontation network. The sparse coding single-image super-resolution reconstruction algorithm can easily lead to the phenomenon of incorrect geometric structure in the reconstructed image. In order to solve the problem of excessive smoothness and blurring of the reconstructed image edges caused by the introduction of this self-similarity constraint, a two-layer reconstruction framework based on a smooth layer and a texture layer is proposed for a medical application of lung cancer. This method uses a global nonzero gradient number constrained reconstruction model to reconstruct the smooth layer. The proposed sparse coding method is used to reconstruct high-resolution texture images. Finally, a global and local optimization models are used to further improve the quality of the reconstructed image. An adaptive multiscale remote sensing image super-division reconstruction network is designed. The selective core network and adaptive gating unit are integrated to extract and fuse features to obtain a preliminary reconstruction. Through the proposed dual-drive module, the feature prior drive loss and task drive loss are transmitted to the super-resolution network. The proposed work not only improves the subjective visual effect but the robustness has also been enhanced with more accurate construction of edges. The statistical evaluators are used to test the viability of the proposed scheme.

Single Image Super-Resolution Algorithm based on Fixed-Point Multi-Model Gaussian Process Regression

To solve the problem of large amount of computation in matrix inversion of Gaussian process regression model, a super-resolution algorithm based on local Gaussian process regression of fixed point multi-model is proposed. Firstly, the training samples are classified by Gaussian mixture model, and image patches are randomly selected as fixed points in each type of training samples, and its K nearest neighbor patch are searched. Secondly, local Gaussian process regression model by using its low-mid-frequency components and the corresponding high frequency components. Again, low resolution test image patch is classified, only the K nearest neighbor test image patch is searched during reconstruction. And then, find the nearest fixed point in each kind of image patch, and use the local Gaussian process regression model based on the fixed point to predict its corresponding high-frequency component. Finally, corresponding high frequency information is predicted by using this trained model, which improves the reconstruction efficiency. Experimental results have demonstrated that the proposed algorithm is superior in both quantitative and qualitative aspects against other algorithms.

Learning Local Distribution for Extremely Efficient Single-Image Super-Resolution

Achieving balance between efficiency and performance is a key problem for convolution neural network (CNN)-based single-image super-resolution (SISR) algorithms. Existing methods tend to directly output high-resolution (HR) pixels or residuals to reconstruct the HR image and focus a lot of attention on designing powerful CNN backbones. However, this reconstruction way requires the CNN backbone to have good ability to fit the mapping function from LR pixels to HR pixels, which certainly held these methods back from achieving extreme efficiency and from working in embedded environments. In this work, we propose a novel distribution learning architecture to estimate the local distribution and reconstruct HR pixels by sampling the local distribution with the corresponding 2D coordinates. We also improve the backbone structure to better support the proposed distribution learning architecture. The experimental results demonstrate that the proposed method achieves state-of-the-art performance for extremely efficient SISR and exhibits a good balance between efficiency and performance.

Multi-Scale Factor Image Super-Resolution Algorithm with Information Distillation Network

Deep convolutional neural networks with strong expressive ability have achieved impressive performances in single-image super-resolution algorithms. However, excessive convolutions usually consume high computational cost, which limits the application of super-resolution technology in low computing power devices. Besides, super-resolution of arbitrary scale factor has been ignored for a long time. Most previous researchers have trained a specific network model separately for each factor, and taken the super-resolution of several integer scale factors into consideration. In this paper, we put forward a multi-scale factor network (MFN), which dynamically predicts the weights of the upscale filter by taking the scale factor as input, and generates HR images with corresponding scale factors from the weights. This method is suitable for arbitrary scale factors (integer or non-integer). In addition, we use an information distillation structure to gradually extract multi-scale spatial features. Extensive experiments suggest that the proposed method performs favorably against the state-of-the-art SR algorithms in term of visual quality, PSNR/SSIM evaluation indicators, and model parameters.

Single Image Super-Resolution Quality Assessment: A Real-World Dataset, Subjective Studies, and an Objective Metric.

Numerous single image super-resolution (SISR) algorithms have been proposed during the past years to reconstruct a high-resolution (HR) image from its low-resolution (LR) observation. However, how to fairly compare the performance of different SISR algorithms/results remains a challenging problem. So far, the lack of comprehensive human subjective study on large-scale real-world SISR datasets and accurate objective SISR quality assessment metrics makes it unreliable to truly understand the performance of different SISR algorithms. We in this paper make efforts to tackle these two issues. Firstly, we construct a real-world SISR quality dataset (i.e., RealSRQ) and conduct human subjective studies to compare the performance of the representative SISR algorithms. Secondly, we propose a new objective metric, i.e., KLTSRQA, based on the Karhunen-Loéve Transform (KLT) to evaluate the quality of SISR images in a no-reference (NR) manner. Experiments on our constructed RealSRQ and the latest synthetic SISR quality dataset (i.e., QADS) have demonstrated the superiority of our proposed KLTSRQA metric, achieving higher consistency with human subjective scores than relevant existing NR image quality assessment (NR-IQA) metrics. The dataset and the code will be made available at https://github.com/Zhentao-Liu/RealSRQ-KLTSRQA.

Iterative Network for Image Super-Resolution

Single image super-resolution (SISR), as a traditional ill-conditioned inverse problem, has been greatly revitalized by the recent development of convolutional neural networks (CNN). These CNN-based methods generally map a low-resolution image to its corresponding high-resolution version with sophisticated network structures and loss functions, showing impressive performances. This paper provides a new insight on conventional SISR algorithm, and proposes a substantially different approach relying on the iterative optimization. A novel iterative super-resolution network (ISRN) is proposed on top of the iterative optimization. We first analyze the observation model of image SR problem, inspiring a feasible solution by mimicking and fusing each iteration in a more general and efficient manner. Considering the drawbacks of batch normalization, we propose a feature normalization (F-Norm, FN) method to regulate the features in network. Furthermore, a novel block with FN is developed to improve the network representation, termed as FNB. Residual-in-residual structure is proposed to form a very deep network, which groups FNBs with a long skip connection for better information delivery and stabling the training phase. Extensive experimental results on testing benchmarks with bicubic (BI) degradation show our ISRN can not only recover more structural information, but also achieve competitive or better PSNR/SSIM results with much fewer parameters compared to other works. Besides BI, we simulate the real-world degradation with blur-downscale (BD) and downscale-noise (DN). ISRN and its extension ISRN+ both achieve better performance than others with BD and DN degradation models.

Selfie Periocular Verification Using an Efficient Super-Resolution Approach

Selfie-based biometrics has great potential for a wide range of applications since, e.g. periocular verification is contactless and is safe to use in pandemics such as COVID-19, when a major portion of a face is covered by a facial mask. Despite its advantages, selfie-based biometrics presents challenges since there is limited control over data acquisition at different distances. Therefore, Super-Resolution (SR) has to be used to increase the quality of the eye images and to keep or improve the recognition performance. We propose an Efficient Single Image Super-Resolution algorithm, which takes into account a trade-off between the efficiency and the size of its filters. To that end, the method implements a loss function based on the Sharpness metric used to evaluate iris images quality. Our method drastically reduces the number of parameters compared to the state-of-the-art: from 2,170,142 to 28,654. Our best results on remote verification systems with no redimensioning reached an EER of 8.89% for FaceNet, 12.14% for VGGFace, and 12.81% for ArcFace. Then, embedding vectors were extracted from SR images, the FaceNet-based system yielded an EER of 8.92% for a resizing of x2, 8.85% for x3, and 9.32% for x4.