Single Image Super-resolution Model Research Articles

Rectified Binary Network for Single-Image Super-Resolution.

Binary neural network (BNN) is an effective approach to reduce the memory usage and the computational complexity of full-precision convolutional neural networks (CNNs), which has been widely used in the field of deep learning. However, there are different properties between BNNs and real-valued models, making it difficult to draw on the experience of CNN composition to develop BNN. In this article, we study the application of binary network to the single-image super-resolution (SISR) task in which the network is trained for restoring original high-resolution (HR) images. Generally, the distribution of features in the network for SISR is more complex than those in recognition models for preserving the abundant image information, e.g., texture, color, and details. To enhance the representation ability of BNN, we explore a novel activation-rectified inference (ARI) module that achieves a more complete representation of features by combining observations from different quantitative perspectives. The activations are divided into several parts with different quantification intervals and are inferred independently. This allows the binary activations to retain more image detail and yield finer inference. In addition, we further propose an adaptive approximation estimator (AAE) for gradually learning the accurate gradient estimation interval in each layer to alleviate the optimization difficulty. Experiments conducted on several benchmarks show that our approach is able to learn a binary SISR model with superior performance over the state-of-the-art methods. The code will be released at https://github.com/jwxintt/Rectified-BSR.

Learn from orientation prior for radiograph super-resolution: Orientation operator transformer

Background and objective: High-resolution radiographic images play a pivotal role in the early diagnosis and treatment of skeletal muscle-related diseases. It is promising to enhance image quality by introducing single-image super-resolution (SISR) model into the radiology image field. However, the conventional image pipeline, which can learn a mixed mapping between SR and denoising from the color space and inter-pixel patterns, poses a particular challenge for radiographic images with limited pattern features. To address this issue, this paper introduces a novel approach: Orientation Operator Transformer - O2former. Methods: We incorporate an orientation operator in the encoder to enhance sensitivity to denoising mapping and to integrate orientation prior. Furthermore, we propose a multi-scale feature fusion strategy to amalgamate features captured by different receptive fields with the directional prior, thereby providing a more effective latent representation for the decoder. Based on these innovative components, we propose a transformer-based SISR model, i.e., O2former, specifically designed for radiographic images. Results: The experimental results demonstrate that our method achieves the best or second-best performance in the objective metrics compared with the competitors at ×4 upsampling factor. For qualitative, more objective details are observed to be recovered. Conclusions: In this study, we propose a novel framework called O2former for radiological image super-resolution tasks, which improves the reconstruction model's performance by introducing an orientation operator and multi-scale feature fusion strategy. Our approach is promising to further promote the radiographic image enhancement field.

Fusing bi-directional global–local features for single image super-resolution

Image super resolution, which obtains high resolution output from a corresponding low resolution image, has been challenging due to inefficiencies in establishing complex high dimensional mapping for massive raw data. Single image super resolution can dramatically improve performance compared with current algorithms due to the proliferation of deep learning systems. However, convolutional kernels in deep neural networks are locally connected to the input feature maps, whereas features only interact with their local neighbors. Mutual interference between local features without considering global features causes blurring and staircase effects. This paper proposes an end-to-end single image super resolution model by simultaneously separating high and low frequency features and learning adaptive local and global features to effectively reconstruct the high resolution image by minimizing the loss of edges and texture information. The image frequency decomposition module with an attention block emphasizes self-representative low frequency features to separate high and low frequency features. The bidirectional global and local feature exchange module extracts global and local features from the separated network and fuses each feature to improve performance. Quantitative and qualitative analyses for the proposed frequency adaptive network validated that the proposed method is stable and robust against blurring and staircase effects by separating texture and the structure into adaptive and shared networks.

Improved wavelet prediction superresolution reconstruction based on U‐Net

AbstractDeep learning can be used to achieve single‐image superresolution (SR) reconstruction. To address problems encountered during this process, such as the number of network parameters, high training requirements on equipment performance, and inability to downsample certain SR images accurately, an image SR reconstruction algorithm based on deep residual network optimization is proposed. The model introduces wavelet transforms based on the original U‐Net, where the U‐Net is trained to obtain SR wavelet feature images at multiple scales simultaneously. This approach reduces the mapping space for the network to learn low‐ to high‐resolution image mapping, which in turn reduces the training difficulty of the model. In terms of network details, the inverse wavelet transform is used in image upsampling to enhance the sparsity of the reconstruction layer in the original network. The network structure of the U‐Net upsampling is adjusted slightly to enable the network to distinguish wavelet images from feature images, thereby improving the richness of the features extracted by the model. The experimental results show that the peak signal‐to‐noise ratio (PSNR) of the fourfold SR model is 32.35 and 28.68 on the Set5 and Set14 validation sets, respectively. Compared with networks that use wavelet prediction mechanisms, such as the deep wavelet prediction SR (DWSR) and deep wavelet prediction‐based residual SR (DWRSR) models, the PSNR for all the tested public datasets is improved by 0.5. The method yields superior results in terms of both visual effect and PSNR, demonstrating the feasibility of the wavelet prediction mechanism in SR reconstruction and thus offering application value and research significance.

Learnable Nonlocal Contrastive Network for Single Image Super-Resolution

Single image super-resolution (SISR) aims to recover a high-resolution image from a single low-resolution image. In recent years, SISR methods based on deep convolutional neural networks have achieved remarkable success, and some methods further improve the performance of the SISR model by introducing nonlocal attention into the model. However, most SISR methods that introduce nonlocal attention focus on more complex attention mechanisms and only use fixed functions for measurement when exploring image similarity. In addition, the model penalizes the algorithm in terms of loss when the output predicted by the model does not match the target data, even if this output is a potentially valid solution. To this end, we propose learnable nonlocal contrastive attention (LNLCA), which flexibly aggregates image features while maintaining linear computational complexity. Then, we introduce the adaptive target generator (ATG) model to address the problem of the single model training mode. Based on LNLCA, we construct a learnable nonlocal contrastive network (LNLCN). The experimental results demonstrate the effectiveness of the algorithm, which produces reconstructed images with more natural texture details.

Adapting Single-Image Super-Resolution Models to Video Super-Resolution: A Plug-and-Play Approach.

The quality of videos varies due to the different capabilities of sensors. Video super-resolution (VSR) is a technology that improves the quality of captured video. However, the development of a VSR model is very costly. In this paper, we present a novel approach for adapting single-image super-resolution (SISR) models to the VSR task. To achieve this, we first summarize a common architecture of SISR models and perform a formal analysis of adaptation. Then, we propose an adaptation method that incorporates a plug-and-play temporal feature extraction module into existing SISR models. The proposed temporal feature extraction module consists of three submodules: offset estimation, spatial aggregation, and temporal aggregation. In the spatial aggregation submodule, the features obtained from the SISR model are aligned to the center frame based on the offset estimation results. The aligned features are fused in the temporal aggregation submodule. Finally, the fused temporal feature is fed to the SISR model for reconstruction. To evaluate the effectiveness of our method, we adapt five representative SISR models and evaluate these models on two popular benchmarks. The experiment results show the proposed method is effective on different SISR models. In particular, on the Vid4 benchmark, the VSR-adapted models achieve at least 1.26 dB and 0.067 improvement over the original SISR models in terms of PSNR and SSIM metrics, respectively. Additionally, these VSR-adapted models achieve better performance than the state-of-the-art VSR models.

Deep convolutional neural network for single remote sensing image super resolution

The existing deep learning-based remote sensing image super-resolution (SR) reconstruction methods have some problems, such as insufficient feature extraction and detail loss. We propose an efficient and fast deep convolutional neural network (CNN) single-image SR model, which achieves relatively advanced reconstruction performance. The number of layers and filter layers of CNNs are completely optimized, and the computing cost is significantly reduced. The global and local features of the original image are extracted by multiscale features; then, contextual information is enhanced in the inference network, and the reconstruction is done in advance of the dimensionality reduction operation to reduce the computational effort. The reconstruction performance is further improved using skip connection network in the whole network to solve the inefficiency of information in the transmission process. The experimental results show that the algorithm is 0.32 to 4.38 dB higher than other algorithms in peak signal-to-noise ratio value, the structure similarity index measure value reaches 0.9618, and the reconstructed image extracts more features and is richer in details.

Semantic Super-Resolution of Text Images via Self-Distillation

This research develops an effective single-image super-resolution (SR) method that increases the resolution of scanned text or document images and improves their readability. To this end, we introduce a new semantic loss and propose a semantic SR method that guides an SR network to learn implicit text-specific semantic priors through self-distillation. Experiments on the enhanced deep SR (EDSR) model, one of the most popular SR networks, confirmed that semantic loss can contribute to further improving the quality of text SR images. Although the improvement varied depending on image resolution and dataset, the peak signal-to-noise ratio (PSNR) value was increased by up to 0.3 dB by introducing the semantic loss. The proposed method outperformed an existing semantic SR method.

Neural texture transfer assisted video coding with adaptive up-sampling

Deep learning techniques have been extensively investigated for the purpose of further increasing the efficiency of traditional video compression. Some deep learning techniques for down/up-sampling-based video coding were found to be especially effective when the bandwidth or storage is limited. Existing works mainly differ in the super-resolution models used. Some works simply use a single image super-resolution model, ignoring the rich information in the correlation between video frames, while others explore the correlation between frames by simply concatenating the features across adjacent frames. This, however, may fail when the textures are not well aligned. In this paper, we propose to utilize neural texture transfer which exploits the semantic correlation between frames and is able to explore the correlated information even when the textures are not aligned. Meanwhile, an adaptive group of pictures (GOP) method is proposed to automatically decide whether a frame should be down-sampled or not. Experimental results show that the proposed method outperforms the standard HEVC and state-of-the-art methods under different compression configurations. When compared to standard HEVC, the BD-rate (PSNR) and BD-rate (SSIM) of the proposed method are up to -19.1% and -26.5%, respectively.

Adjustable super-resolution network via deep supervised learning and progressive self-distillation

With the use of convolutional neural networks, Single-Image Super-Resolution (SISR) has advanced dramatically in recent years. However, we notice a phenomenon that the structure of all these models must be consistent during training and testing. This severely limits the flexibility of the model, making the same model difficult to be deployed on different sizes of platforms (e.g., computers, smartphones, and embedded devices). Therefore, it is crucial to develop a model that can adapt to different needs without retraining. To achieve this, we propose a lightweight Adjustable Super-Resolution Network (ASRN). Specifically, ASRN consists of a series of Multi-scale Aggregation Blocks (MABs), which is a lightweight and efficient module specially designed for feature extraction. Meanwhile, the Deep Supervised Learning (DSL) strategy is introduced into the model to guarantee the performance of each sub-network and a novel Progressive Self-Distillation (PSD) strategy is proposed to further improve the intermediate results of the model. With the help of DSL and PSD strategies, ASRN can achieve elastic image reconstruction. Meanwhile, ASRN is the first elastic SISR model, which shows good results after directly changing the model size without retraining.

Multi-fidelity and learning-regularization for single image super resolution

Ill-posedness results in regularization-based methods being widely used in single image super-resolution (SISR). However, producing super-resolved images with finer details and fewer artifacts is still a great challenge. With the help of the plug-and-play framework, we introduce a novel fidelity term and learned prior knowledge to produce a powerful SISR model. In the proposed fidelity term, called multi-fidelity, the similarity between the observed data and simulated data is measured in terms of both intensity and gradient; these measurements can reflect the tendency of feature response results to degrade into multiple image layers. Based on the half quadratic splitting (HQS) method, the proposed SISR model is split into two sub-problems, which include the multi-fidelity and regularization terms, respectively. In this paper, we design a deep network, named as R&BED, to learn prior image knowledge. Compared with the manually designed regularization term, learned knowledge can preserve easily ignored features. Experimental results indicate that the subjective and objective metrics corresponding to the proposed method are better than those obtained using the comparison methods.

Flexible Style Image Super-Resolution Using Conditional Objective

Recent studies have significantly enhanced the performance of single-image super-resolution (SR) using convolutional neural networks (CNNs). While there can be many high-resolution (HR) solutions for a given input, most existing CNN-based methods do not explore alternative solutions during the inference. A typical approach to obtaining alternative SR results is to train multiple SR models with different loss weightings and exploit the combination of these models. Instead of using multiple models, we present a more efficient method to train a single adjustable SR model on various combinations of losses by taking advantage of multi-task learning. Specifically, we optimize an SR model with a conditional objective during training, where the objective is a weighted sum of multiple perceptual losses at different feature levels. The weights vary according to given conditions, and the set of weights is defined as a style controller. Also, we present an architecture appropriate for this training scheme, which is the Residual-in-Residual Dense Block equipped with spatial feature transformation layers. At the inference phase, our trained model can generate locally different outputs conditioned on the style control map. Extensive experiments show that the proposed SR model produces various desirable reconstructions without artifacts and yields comparable quantitative performance to state-of-the-art SR methods. Code and trained models are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/seungho-snu/FxSR</uri> .

Spatial resolution enhancement method for Landsat imagery using a Generative Adversarial Network

ABSTRACT Landsat and Sentinel-2 are two freely accessible satellite data that are relevant for global land cover monitoring. However, the uses of the latter data set are growing because of its higher spatial resolutions and the availability of benchmark data sets for deep learning applications. In this study, we integrate a style transfer (perceptual loss estimation from Sentinel 2 benchmark data) into a Generative Adversarial Network (GAN) to construct a single image super-resolution model. The proposed model upscales Landsat 8 images (using red, green, blue, and near-infrared bands at 30 m and Panchromatic band 15 m for high-resolution features exploiting) to 10 m (with Sentinel-2 as reference). Compared to pan-sharpening and other upscaling methods, the proposed method can produce more realistic, spatial convincing images at 10 m resolution and more similar to Sentinel-2 images than the other commonly used super-resolution imaging algorithms. As a result, the proposed method extends the usage of high-resolution benchmark data sets for lower resolution imagery to enrich supplement data sources for land cover classification.

Image super-resolution by learning weighted convolutional sparse coding

Single image super-resolution (SISR) has witnessed substantial progress recently by deep learning-based methods, due to the data-driven end-to-end training. However, most existing DL-based models are built intuitively, with little thought on priors. And the lack of interpretability limits their further improvements. To avoid this, this paper presents an end-to-end trainable unfolding network which leverages both DL- and prior-based methods. Specifically, we introduce the reweighted algorithm into CSC model and solve it by learning weighted iterative soft thresholding algorithm in a convolutional manner. Based on this, we present a SISR model by learning weighted convolutional sparse coding, in which the channel attention is resorted to learn the weight. Extensive experiments demonstrate the superiority of our method to recent state-of-the-art SISR methods, in terms of both quantitative and qualitative results.

Meta-USR: A Unified Super-Resolution Network for Multiple Degradation Parameters.

Recent research on single image super-resolution (SISR) has achieved great success due to the development of deep convolutional neural networks. However, most existing SISR methods merely focus on super-resolution of a single fixed integer scale factor. This simplified assumption does not meet the complex conditions for real-world images which often suffer from various blur kernels or various levels of noise. More importantly, previous methods lack the ability to cope with arbitrary degradation parameters (scale factors, blur kernels, and noise levels) with a single model. A few methods can handle multiple degradation factors, e.g., noninteger scale factors, blurring, and noise, simultaneously within a single SISR model. In this work, we propose a simple yet powerful method termed meta-USR which is the first unified super-resolution network for arbitrary degradation parameters with meta-learning. In Meta-USR, a meta-restoration module (MRM) is proposed to enhance the traditional upscale module with the capability to adaptively predict the weights of the convolution filters for various combinations of degradation parameters. Thus, the MRM can not only upscale the feature maps with arbitrary scale factors but also restore the SR image with different blur kernels and noise levels. Moreover, the lightweight MRM can be placed at the end of the network, which makes it very efficient for iteratively/repeatedly searching the various degradation factors. We evaluate the proposed method through extensive experiments on several widely used benchmark data sets on SISR. The qualitative and quantitative experimental results show the superiority of our Meta-USR.

Deep Learning-Based Single Image Super-Resolution: An Investigation for Dense Scene Reconstruction with UAS Photogrammetry

The deep convolutional neural network (DCNN) has recently been applied to the highly challenging and ill-posed problem of single image super-resolution (SISR), which aims to predict high-resolution (HR) images from their corresponding low-resolution (LR) images. In many remote sensing (RS) applications, spatial resolution of the aerial or satellite imagery has a great impact on the accuracy and reliability of information extracted from the images. In this study, the potential of a DCNN-based SISR model, called enhanced super-resolution generative adversarial network (ESRGAN), to predict the spatial information degraded or lost in a hyper-spatial resolution unmanned aircraft system (UAS) RGB image set is investigated. ESRGAN model is trained over a limited number of original HR (50 out of 450 total images) and virtually-generated LR UAS images by downsampling the original HR images using a bicubic kernel with a factor × 4 . Quantitative and qualitative assessments of super-resolved images using standard image quality measures (IQMs) confirm that the DCNN-based SISR approach can be successfully applied on LR UAS imagery for spatial resolution enhancement. The performance of DCNN-based SISR approach for the UAS image set closely approximates performances reported on standard SISR image sets with mean peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) index values of around 28 dB and 0.85 dB, respectively. Furthermore, by exploiting the rigorous Structure-from-Motion (SfM) photogrammetry procedure, an accurate task-based IQM for evaluating the quality of the super-resolved images is carried out. Results verify that the interior and exterior imaging geometry, which are extremely important for extracting highly accurate spatial information from UAS imagery in photogrammetric applications, can be accurately retrieved from a super-resolved image set. The number of corresponding keypoints and dense points generated from the SfM photogrammetry process are about 6 and 17 times more than those extracted from the corresponding LR image set, respectively.

Adaptive deep residual network for single image super-resolution

In recent years, deep learning has achieved great success in the field of image processing. In the single image super-resolution (SISR) task, the convolutional neural network (CNN) extracts the features of the image through deeper layers, and has achieved impressive results. In this paper, we propose a single image super-resolution model based on Adaptive Deep Residual named as ADR-SR, which uses the Input Output Same Size (IOSS) structure, and releases the dependence of upsampling layers compared with the existing SR methods. Specifically, the key element of our model is the Adaptive Residual Block (ARB), which replaces the commonly used constant factor with an adaptive residual factor. The experiments prove the effectiveness of our ADR-SR model, which can not only reconstruct images with better visual effects, but also get better objective performances.

Learning Image Profile Enhancement and Denoising Statistics Priors for Single-Image Super-Resolution

Single-image super-resolution (SR) has been widely used in computer vision applications. The reconstruction-based SR methods are mainly based on certain prior terms to regularize the SR problem. However, it is very challenging to further improve the SR performance by the conventional design of explicit prior terms. Because of the powerful learning ability, deep convolutional neural networks (CNNs) have been widely used in single-image SR task. However, it is difficult to achieve further improvement by only designing the network architecture. In addition, most existing deep CNN-based SR methods learn a nonlinear mapping function to directly map low-resolution (LR) images to desirable high-resolution (HR) images, ignoring the observation models of input images. Inspired by the split Bregman iteration (SBI) algorithm, which is a powerful technique for solving the constrained optimization problems, the original SR problem is divided into two subproblems: 1) inversion subproblem and 2) denoising subproblem. Since the inversion subproblem can be regarded as an inversion step to reconstruct an intermediate HR image with sharper edges and finer structures, we propose to use deep CNN to capture low-level explicit image profile enhancement prior (PEP). Since the denoising subproblem aims to remove the noise in the intermediate image, we adopt a simple and effective denoising network to learn implicit image denoising statistics prior (DSP). Furthermore, the penalty parameter in SBI is adaptively tuned during the iterations for better performance. Finally, we also prove the convergence of our method. Thus, the deep CNNs are exploited to capture both implicit and explicit image statistics priors. Due to SBI, the SR observation model is also leveraged. Consequently, it bridges between two popular SR approaches: 1) learning-based method and 2) reconstruction-based method. Experimental results show that the proposed method achieves the state-of-the-art SR results.

Single Image Super Resolution Model Using Learnable Weight Factor in Residual Skip Connection

In single image super-resolution problems, the recent feed forward deep learning architectures use residual connections in order to preserve local features and carry them through the next layer. In a simple residual skip connection, all the features of the earlier layer are concatenated with the features of the current layer. A simple concatenation of the features does not exploit the fact that some features may be more useful than other features and vice versa. To overcome this limitation, we propose an extended architecture (baby neural network) which will have input as the features learned from the previous layer and output a multiplication factor. This multiplication factor will give importance to the given feature and thus help in training the current layer’s features more accurately. The proposed model clearly outperforms the existing works.

Super-resolution image reconstruction using fractional-order total variation and adaptive regularization parameters

Single-image super-resolution (SR) reconstruction aims to obtain a high-resolution (HR) image from a low-resolution (LR) image. In this paper, a hybrid single-image SR model integrated total variation (TV) and fractional-order TV (FOTV) is proposed to pursuit the adaptive reconstruction of the HR image. Specifically, fractional order in the proposed SR model is adaptively set according to the textural feature of the LR image firstly; then, the SR model is separated into two sub-models with each of them containing exactly one regularization parameter. These two sub-models are solved by using alternating direction multiplier method, and two regularization parameters are concurrently updated by using discrepancy principle. Finally, the solutions of two sub-models are interactively averaged to reconstruct HR image. The results of experiments indicated that the proposed hybrid SR model with adaptive regularization parameters has a comparative performance compared with state-of-the-art methods. Moreover, it would be potentially more adaptive for the condition of varied blurred kernels.