Single Image Super-resolution Methods Research Articles

Self-supervised cycle-consistent learning for scale-arbitrary real-world single image super-resolution

Whether conventional machine learning-based or current deep neural networks-based single image super-resolution (SISR) methods, they are generally trained and validated on synthetic datasets, in which low-resolution (LR) inputs are artificially produced by degrading high-resolution (HR) images based on a hand-crafted degradation model (e.g., bicubic downsampling). One of the main reasons for this is that it is challenging to build a realistic dataset composed of real-world LR–HR image pairs. However, a domain gap exists between synthetic and real-world data because the degradations in real scenarios are more complicated, limiting the performance in practical applications of SISR models trained with synthetic data. To address these problems, we propose a Self-supervised Cycle-consistent Learning-based Scale-Arbitrary Super-Resolution framework (SCL-SASR) for real-world images. Inspired by the Maximum a Posteriori estimation, our SCL-SASR consists of a Scale-Arbitrary Super-Resolution Network (SASRN) and an inverse Scale-Arbitrary Resolution-Degradation Network (SARDN). SARDN and SASRN restrain each other with the bidirectional cycle consistency constraints as well as image priors, making SASRN adapt to the image-specific degradation well. Meanwhile, considering the lack of targeted training images and the complexity of realistic degradations, SCL-SASR is designed to be online optimized solely with the LR input prior to the SR reconstruction. Benefitting from the flexible architecture and the self-supervised learning manner, SCL-SASR can easily super-resolve new images with arbitrary integer or non-integer scaling factors. Experiments on real-world images demonstrate the high flexibility and good applicability of SCL-SASR, which achieves better reconstruction performance than state-of-the-art self-supervised learning-based SISR methods as well as several external dataset-trained SISR models.

High-order cross-scale attention network for single image super-resolution

Deep convolutional neural networks (DCNN) have achieved remarkable performance for single image super-resolution (SISR). Although some recent works have introduced multi-scale representation into DCNN-based super-resolution, they neglect to explore the interdependencies across different scales. To address this issue, we propose a high-order cross-scale attention network (HOCSANet) for SISR reconstruction. Specifically, a novel high-order cross-scale attention block (HOCSABlock) is developed to integrate the multi-scale representation with attention mechanism, in which both in-scale and cross-scale feature correlations are exploited and utilized to adaptively rescale the multi-scale features. Moreover, we introduce a dual-nested residual block (DNRBlock), which combines plain residual block and wide-activated residual block and learns more informative features from the difference between them. Experimental results demonstrate the superiority of the proposed HOCSANet over several state-of-the-art SISR methods.

Adaptive feature denoising based deep convolutional network for single image super-resolution

Recently, the feature map recalibration (FMR) mechanism has been widely explored in single image super-resolution (SISR) and obtained remarkable performances. However, the existing FMR-based SISR methods directly incorporate the attention module into a deeper network structure (e.g. EDSR), while neglecting the differences between the low-level and high-level vision problems. In this paper, we design a low-level specific FMR mechanism for SISR task based on a new observation by examining current SISR methods, which all demonstrate a solid correlation between the SISR performance and the convolutional feature noise. Inspired by this, we extend the classic soft thresholding technique in the way of deep network, and develop an Adaptive Soft Thresholding (AST) module for feature noise suppression. Comparing to existing attention modules, AST is light-weighted and can be taken as an easy plug-in module in any SISR networks. To this end, we construct a adaptive Feature Denoising Super-Resolution (FDSR) network by combining the baseline EDSR and the proposed AST. Extensive experimental results show that the proposed FDSR network could achieve the state-of-the-art performances on SISR benchmarks, and significantly reduce the parameter (28.8% for EDSR, 74.6% for RCAN,s 76.0% for SAN) with respect to FMR module.

Lightweight Single Image Super-Resolution with Selective Channel Processing Network.

With the development of deep learning, considerable progress has been made in image restoration. Notably, many state-of-the-art single image super-resolution (SR) methods have been proposed. However, most of them contain many parameters, which leads to a significant amount of calculation consumption in the inference phase. To make current SR networks more lightweight and resource-friendly, we present a convolution neural network with the proposed selective channel processing strategy (SCPN). Specifically, the selective channel processing module (SCPM) is first designed to dynamically learn the significance of each channel in the feature map using a channel selection matrix in the training phase. Correspondingly, in the inference phase, only the essential channels indicated by the channel selection matrixes need to be further processed. By doing so, we can significantly reduce the parameters and the calculation consumption. Moreover, the differential channel attention (DCA) block is proposed, which takes into consideration the data distribution of the channels in feature maps to restore more high-frequency information. Extensive experiments are performed on the natural image super-resolution benchmarks (i.e., Set5, Set14, B100, Urban100, Manga109) and remote-sensing benchmarks (i.e., UCTest and RESISCTest), and our method achieves superior results to other state-of-the-art methods. Furthermore, our method keeps a slim size with fewer than 1 M parameters, which proves the superiority of our method. Owing to the proposed SCPM and DCA, our SCPN model achieves a better trade-off between calculation cost and performance in both general and remote-sensing SR applications, and our proposed method can be extended to other computer vision tasks for further research.

Involution-based network with contrastive learning for efficient image super-resolution

Single-image super-resolution (SISR) studies have achieved superior improvement with the development of convolution neural networks. However, most methods sink into the high computation cost. To tackle this issue, we propose an involution-based lightweight method with contrastive learning for efficient SISR. Unlike the original involution, we set the group number of involution operations to the input feature channels. This setting guarantees the spatial- and channel-specific peculiarity. Moreover, our implemented involution not only learns the weight but also the bias for convolution. Simultaneously, we rethink the kernel generation functions of involution. Instead, we utilize Sigmoid with reparameterized convolution. We additionally apply residual path to involution operation. Furthermore, contrastive learning is adopted during training to learn universal features. Compared with state-of-the-art efficient SISR methods, our proposed methods achieve the best performance with similar or fewer parameters.

Semantic Super-Resolution of Text Images via Self-Distillation

This research develops an effective single-image super-resolution (SR) method that increases the resolution of scanned text or document images and improves their readability. To this end, we introduce a new semantic loss and propose a semantic SR method that guides an SR network to learn implicit text-specific semantic priors through self-distillation. Experiments on the enhanced deep SR (EDSR) model, one of the most popular SR networks, confirmed that semantic loss can contribute to further improving the quality of text SR images. Although the improvement varied depending on image resolution and dataset, the peak signal-to-noise ratio (PSNR) value was increased by up to 0.3 dB by introducing the semantic loss. The proposed method outperformed an existing semantic SR method.

Single Image Super-Resolution Method Based on an Improved Adversarial Generation Network

Super-Resolution (SR) techniques for image restoration have recently been gaining attention due to their excellent performance. For powerful learning abilities, Generative Adversarial Networks (GANs) have been proven to have achieved great success. In this paper, we propose an Enhanced Generative Adversarial Network (EGAN) for improving its effects for a real-time Super-Resolution task. The main content of this paper are as follows: (1) We adopted the Laplacian pyramid framework as a pre-trained module, which is beneficial for providing multiscale features for our input. (2) At each feature block, a convolutional skip-connections network, which may contain some latent information, was significant for the generative model to reconstruct a plausible-looking image. (3) Considering that the edge details usually play an important role in image generation, a perceptual loss function was defined to train and seek the optimal parameters. Quantitative and qualitative evaluations were demonstrated so that our algorithm not only took full advantage of the Convolutional Neural Networks (CNNs) to improve the image quality, but also performed better than other algorithms in speed and performance for real-time Super-Resolution tasks.

SREFBN: Enhanced feature block network for single‐image super‐resolution

Deep learning has assisted the field of single-image super-resolution (SR) in achieving new heights. However, the task of restoring a high-resolution (HR) image from a highly degraded low-resolution (LR) image is sophisticated due to poor image restoration quality. A novel and effective lightweight SR method is presented as super-resolution via an enhanced feature block network (SREFBN) that successfully reconstructs an HR image using a corresponding LR image with a purposed deep residual block. In addition, a novel shared parameters approach in the top-down pathway among low-level feature maps is introduced. The experimental results prove that SREFBN achieves remarkable performance. The presented framework requires lower computational cost and outperforms many state-of-the-art methods. It is also highly adaptable with low-end devices, requiring lower multiplication and adding operations. A trade-off comparison between the number of parameters, execution time, and accuracies is given while also showing different variations of our approach to prove the effectiveness and reliability of the shared parameters. Most importantly, the results indicate that our framework has gained state-of-the-art performance on larger scales 3 and 4. Code is available at https://github.com/curzii23/SREFBN.

Lightweight single image super-resolution with attentive residual refinement network

In recent years, deep convolutional neural network (CNN) based single image super-resolution (SISR) methods have been demonstrated impressive performance in terms of quantitative metrics and visual effects. Most CNN-based SISR methods can learn the complex non-linear mapping between low-resolution (LR) images and their corresponding high-resolution (HR) images due to the powerful representation capabilities of deep convolutional neural networks. However, as the depth and width of the SISR networks increase, the parameters of SISR networks will increase dramatically, leading to huge computational cost and large memory consumption, making them impractical in real-world applications. To attack the above issues, we propose an accurate and lightweight deep convolutional neural network, named Attentive Residual Refinement Network (ARRFN), to recover the high-resolution image from the original low-resolution image directly for SISR. In general, our proposed ARRFN consists of three parts, a feature extraction block, a stack of attentive residual refinement blocks (ARRFB), and a multi-scale separable upscaling module (MSSU), respectively. Specifically, our ARRFB consists of two branches, a regular residual learning branch and an attentive residual refinement branch. The former conducts regular residual learning by two residual blocks while the latter refines the residual information from the two residual blocks of the former branch with an attentive residual mechanism to further enhance the representation capabilities of the network. Furthermore, a multi-scale separable upsampling module (MSSU) is proposed to replace the regular upsampling operation for better SR results. Extensive experiments on several standard benchmarks show that the proposed method outperforms state-of-the-art SR methods in terms of quantitative metrics, visual quality, memory footprint, and inference time.

APPLICATION OF PRE-TRAINED REAL-WORLD SUPER RESOLUTION MODELS TO OPTICAL SATELLITE IMAGE

Abstract. The single image super-resolution (SISR) technique refers to improving the resolution over the original image. In recent years, we use deep learning-based convolutional neural networks to improve the spatial resolution of images more reasonably. To train such deep learning models, we use training samples consisting of the original HR images and LR images obtained by bicubic downsampling. However, this method of training data using downsampling has a negative impact when applying the trained model to real images. That is because the downsampling function that occurs in the real image is unknown, and the hypothetically created LR image does not represent the resolution degradation that can realistically occur. Therefore, SISR methods that use realistic degradation called real-world super-resolution (RWSR) have been proposed. In this paper, we investigate how such RWSR methods using realistic degradation affect the SISR performance of satellite images. The results of applying the trained model to optical satellite images show that the RWSR method is not the most effective way to handle optical satellite images when compared to the deep learning method without modeling the degradation. In particular, we showed that the effect of RWSR with not only upsampling but also noise and blur removal is significant in the visibility of optical satellite images. Moreover, pre-trained RWSR models can be an aid in visually deciphering ground objects.

Dual Projection Fusion for Reference-Based Image Super-Resolution.

Reference-based image super-resolution (RefSR) methods have achieved performance superior to that of single image super-resolution (SISR) methods by transferring texture details from an additional high-resolution (HR) reference image to the low-resolution (LR) image. However, existing RefSR methods simply add or concatenate the transferred texture feature with the LR features, which cannot effectively fuse the information of these two independently extracted features. Therefore, this paper proposes a dual projection fusion for reference-based image super-resolution (DPFSR), which enables the network to focus more on the different information between feature sources through inter-residual projection operations, ensuring effective filling of detailed information in the LR feature. Moreover, this paper also proposes a novel backbone called the deep channel attention connection network (DCACN), which is capable of extracting valuable high-frequency components from the LR space to further facilitate the effectiveness of image reconstruction. Experimental results show that we achieve the best peak signal-to-noise ratio (PSNR) and structure similarity (SSIM) performance compared with the state-of-the-art (SOTA) SISR and RefSR methods. Visual results demonstrate that the proposed method in this paper recovers more natural and realistic texture details.

ENHANCED SUPER RESOLUTION FOR REMOTE SENSING IMAGERIES

Abstract. Single image super resolution (SISR) technology has been attracted much attention from remote sensing community due to its proven potentials in remote sensing applications. Existing SISR techniques varying from conventional interpolation methods to different network architectures. Generative adversarial networks (GANs) are one of the latest network architectures proven a greater potential as a SISR method whereas least attention has been given by the remote sensing community. Several studies have already been carried out on this context. However, yet there is no generalized GAN based approach to super resolve remote sensing imageries. Therefore, this study investigated the potentials of enhanced super resolution generative adversarial (ESRGAN) model to super resolve very high to medium resolution images from high to coarse resolution images for remote sensing applications. Two models were trained and Worldview-3 (WV3) images used as for very high resolution images. Whereas, down sampled WV3 and Sentinel-2(S2) were used as low resolution counterparts. Model performances were qualitatively and quantitatively analysed using standard metrics such as PSNR, SSIM, UIQI, CC, SAM, SID. Evaluation results emphasised super resolved images were preserved the original quality of the satellite images to a greater extent while improving its ground resolution.

DCT-FANet: DCT based frequency attention network for single image super-resolution

In single-image super-resolution (SISR) task, it is challenging to recover high-frequency details from a low-resolution (LR) image due to its ill-posed problem. Most existing CNN-based super-resolution (SR) methods pursue an end-to-end solution from the low-resolution image to the high-resolution image through deeper or wider networks. However, these methods treat high-frequency information and low-frequency information equally which hinder the recovery of high-frequency details and result in low visual comfort. In this paper, we propose a DCT based frequency attention network (DCT-FANet) to distinguish different frequency of LR images, and enhance the high-frequency information adaptively. Specially, a DCT spatial cube extraction (DCT-SCE) module is proposed to decompose LR images into multiple frequency subbands. Different from other DCT based SISR methods that use DCT coefficients as subbands, our module considers the feature difference between spatial domain and frequency domain, and remains spatial structure information of each DCT subband. Then, based on the characteristics of DCT subbands, we design a Gaussian based frequency selection (GFS) module to give more attention to the high-frequency information. To promote communication between each frequency subband, an adaptive non-local dual attention (ANDA) module is developed, leading to a further enhancement of high-frequency information and improvement of the final performance. Experimental results demonstrates that our DCT-FANet recovers more high-frequency details, and achieves excellent performance with fewer parameters.

Constrained and Unconstrained Inverse Potts Modelling for Joint Image Super-Resolution and Segmentation

In this work we consider two methods for joint single-image super-resolution and image partitioning. The proposed approaches rely on a constrained and on an unconstrained version of the inverse Potts model where an L0 regularization prior on the image gradient is used for promoting piecewise constant solutions. For the numerical solution of both models, we provide a unified implementation based on the Alternating Direction Method of Multipliers (ADMM). Upon suitable assumptions on both model operators and on the algorithmic parameters involved, we show that all the ADMM subproblems admit closed-form solutions, thus making the resulting algorithms computationally very cheap even when high-dimensional data are considered. Numerical details of the implementation of both models are given and several experiments are carried out on both synthetic and natural images to underline the accuracy and the computational efficiency of the models.

Deep iterative residual back-projection networks for single-image super-resolution

The process of single-image super-resolution (SR) has certain limitations, such as an insufficient utilization of high-frequency information in images and a network structure that is insufficiently flexible to reconstruct the feature information of different complexities. Therefore, deep iterative residual back-projection networks are proposed. Residual learning was used to ease the difficulty in training and fully discover the feature information of the image, and a back-projection method was applied to study the interdependence between high- and low-resolution images. In addition, the network structure reconstructs smooth-feature and high-frequency information of the image separately and transmits only the residual features among all residual blocks of the network structure. The experiment results show that compared with most single-frame image SR methods, the proposed approach not only achieves a significant improvement in objective indicators, but it also provides richer texture information in the reconstructed predicted image.

Non-linear perceptual multi-scale network for single image super-resolution

Recently, deep convolutional neural networks (CNNs) have been widely explored in single image super-resolution (SISR) and achieved remarkable progress. However, most of the existing CNN-based SISR networks with a single-stream structure fail to make full use of the multi-scale features of low-resolution (LR) image. While those multi-scale SR models often integrate the information with different receptive fields by means of linear fusion, which leads to the redundant feature extraction and hinders the reconstruction performance of the network. To address both issues, in this paper, we propose a non-linear perceptual multi-scale network (NLPMSNet) to fuse the multi-scale image information in a non-linear manner. Specifically, a novel non-linear perceptual multi-scale module (NLPMSM) is developed to learn more discriminative multi-scale feature correlation by using high-order channel attention mechanism, so as to adaptively extract image features at different scales. Besides, we present a multi-cascade residual nested group (MC-RNG) structure, which uses a global multi-cascade mechanism to organize multiple local residual nested groups (LRNG) to capture sufficient non-local hierarchical context information for reconstructing high-frequency details. LRNG uses a local residual nesting mechanism to stack NLPMSMs, which aims to form a more effective residual learning mechanism and obtain more representative local features. Experimental results show that, compared with the state-of-the-art SISR methods, the proposed NLPMSNet performs well in both quantitative metrics and visual quality with a small number of parameters.

Structure-Preserving and Color-Restoring Up-Sampling for Single Low-Light Image

Single image super-resolution (SR), as a basic computer vision task, has been widely studied. However, existing single image SR methods, applied in images with normal light, perform poorly on low-light images. To address this limitation, a method of structure-preserving and color-restoring up-sampling for single low-light image is proposed. Theoretical and experimental analysis indicates that existing methods suffer negative effects due to the suppressed color and the weakened texture in low-light image. Therefore, combined with the Retinex theory, the single low-light image up-sampling model is established for the first time, which avoids the conflict between color and texture by distinguishing reflectance and illumination. Further, we develop a structure-preserving and color-restoring up-sampling network for single low-light image SR. In the network, the reflectance and illumination components are obtained by decomposing the observed image, and then up-sampling of reflectance and enhancement of illumination are performed to complete the primary SR. In addition, the gradient information is reconstructed and fused into the up-sampling process to further enrich the SR texture. Experiments demonstrate that our method obtains competitive qualitative and quantitative evaluation on the produced dataset and real-world images.

Image Super-Resolution With Self-Similarity Prior Guided Network and Sample-Discriminating Learning

The nonlocal self-similarity in natural image provides an effective prior for single image super-resolution (SISR), which is beneficial to contextual information capture and performance improvement, as demonstrated by conventional SISR methods. However, it is little explored to utilize this property in deep neural networks. In this paper, we propose a self-similarity prior guided (SSPG) network to incorporate self-similarity-based nonlocal operation into deep neural network for SISR. Specifically, we design a cross-scale nearest-neighbor residual (CSNNR) block via introducing cross-scale <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> -nearest neighbors (KNN) matching into a residual block, which can be flexibly integrated into deep networks to capture long-range correlations among multi-scale and multi-level features. Meanwhile, by stacking a CSNNR block and a sequence of wide-activated residual blocks with a local skip-connection, a multi-level residual self-similarity (MRSS) module is developed to effectively employ local and nonlocal information for detail recovery. Thus, through cascading multiple MRSS modules, the proposed SSPG network performs both self-similarity-based nonlocal operation and convolution-based local operation on multi-level features to reconstruct informative features for accurate SISR. In addition, for pursuing visually pleasing results, we apply our SSPG network to the perception-oriented SISR field by following the framework of generative adversarial networks. In particular, we explore a sample-discriminating learning mechanism based on the statistical descriptions of training samples, and include it in optimization procedure to automatically tune the contributions of different samples according to their characteristics and then focus the network on creating realistic results. Extensive quantitative and qualitative evaluations on benchmark datasets illustrate the superiority of our proposed models over the state-of-the-art methods for both distortion-oriented and perception-oriented image super-resolution tasks.

Closed-Loop Residual Attention Network for Single Image Super-Resolution

Recent research on single image super-resolution (SISR) using convolutional neural networks (CNNs) with the utilization of residual structures and attention mechanisms to utilize image features has demonstrated excellent performance. However, previous SISR techniques mainly integrated extracted image features within a deep or wide network architecture, ignoring the interaction between multiscale features and the diversity of features. At the same time, SISR is also a typical ill-posed problem in that it allows for several predictions for a given LR image. These problems limit the great learning ability of CNNs. To solve these problems, we propose a closed-loop residual attention network (CLRAN) to extract and interact with all the available diversity of features features efficiently and limit the space of possible function solutions. Specifically, we design an enhanced residual attention block (ERA) to extract features, and it dynamically assigns weight to the internal attention branches. The ERA combines multi-scale block (MSB) and enhanced attention mechanism (EAM) base on the residual module. The MSB adaptively detects multiscale image features of different scales by using different 3 × 3 convolution kernels. The EAM combines multi-spectral channel attention (MSCA) and spatial attention (SA). Therefore, the EAM extracts different frequency component information and spatial information to utilize the diversity features. Furthermore, we apply the progressive network architecture and learn an additional map for model monitoring, which forms a closed-loop with the mapping already learned by the LR to HR function. Extensive experiments demonstrate that our CLRAN outperforms the state-of-the-art SISR methods on public datasets for both ×4 and ×8, proving its accuracy and visual perception.

Single MR image super-resolution via channel splitting and serial fusion network

In magnetic resonance imaging (MRI), spatial resolution is an important and critical imaging parameter that represents how much information is contained in a unit space. Acquiring high-resolution MRI data usually takes a long scanning time and is subject to motion artifacts due to hardware, physical, and physiological limitations. Single image super-resolution (SISR) based on deep learning is an effective and promising alternative technique to improve the native spatial resolution of magnetic resonance (MR) images. However, because of the simple diversity and single distribution of training samples, the effective training of deep models with medical training samples and improvement of the tradeoff between model performance and computing overhead are major challenges. In addition, deeper networks are more difficult to effectively train since the information is gradually weakened as the network deepens. In this paper, a novel channel splitting and serial fusion network (CSSFN) is presented for single MR image super-resolution. The proposed CSSFN splits hierarchical features into a series of subfeatures, which are then integrated together in a serial manner. Hence, the network becomes deeper and can discriminatively and reasonably deal with the subfeatures. Moreover, a dense global feature fusion (DGFF) is adopted to integrate the intermediate features, which further promotes the information flow in the network and helps to stabilize model training. Extensive experiments on several typical MR images show the superiority of our CSSFN models to other advanced SISR methods.