Single Image Super-resolution Methods Research Articles

Hybridisation of single‐image super‐resolution with edge‐aware multi‐focus image fusion for edge enrichment

To break the curtailment of digital imaging and retrieve appropriate information with different focused images, a novel edge-aware multi-focus image fusion is proposed by integrating the single-image super-resolution (SISR) method along with the edge-preserving filters. Initially, the multi-focus images are converted to high resolution images by estimating missing high frequency details from its blurred versions. With acquired high resolution images, smoothing by median and the anisotropic diffusion filters are performed to extract focused regions. An initial weight map is constructed by using maximum selection of pixel intensities of the difference images obtained with filtering. The precision of estimated weight map is further improved by exhibiting morphological operations and guided filter. Finally, the images are fused based on the optimised decision map. Simulation results of proposed fusion work are evaluated with seven metrics and the values are compared with different state-of-the-art methods. Both the quantitative and qualitative analyses showed the excellence of proposed work over other fusion methods.

Generative adversarial image super‐resolution network for multiple degradations

The existing single image super-resolution methods based on deep learning cannot handle multiple degradations well, and the generated image tends to be blurred and over-smoothed due to poor generalisation ability. In this study, the authors propose a method based on a generative adversarial network (GAN) to deal with multiple degradations. In the generator network, blur kernel and noise level are used as input through dimensionality stretching strategy preprocessing to make full use of prior knowledge. In addition, three discriminators with different scales are used in the discriminator network to pay attention to the reconstruction of image details while focusing on the global consistency of the image. For the problems of vanishing gradient and mode collapse existing in GAN-based methods, a gradient penalty term is added in the loss function. Extensive experiments demonstrate that the proposed method not only can handle multiple degradations to obtain state-of-the-art performance, but also deliver visually credible results in real scenes.

Single-image super-resolution based on local biquadratic spline with edge constraints and adaptive optimization in transform domain

This paper proposes a novel single-image super-resolution method based on local biquadratic spline with edge constraints and adaptive optimization in transform domain. The complex internal structure of the image makes the values of adjacent pixels often differ greatly. Using surface patches to interpolate image blocks can avoid large surface oscillation. Because the quadratic spline has better shape-preserving property, we construct the biquadratic spline surface on each image block to make the interpolation more flexible. The boundary conditions have great influence on the shape of local biquadratic spline surfaces and are the keys to constructing surfaces. Using edge information as a constraint to calculate them can reduce jagged and mosaic effects. To decrease the errors caused by surface fitting, we propose a new adaptive optimization model in transform domain. Compared with the traditional iterative back-projection, this model further improves the magnification accuracy by introducing SVD-based adaptive optimization. In the optimization, we convert similar block matrices to the transform domain by SVD. Then the contraction coefficients are calculated according to the non-local self-similarity, and the singular values are contracted. Experimental comparison with the other state-of-the-art methods shows that the proposed method has better performance in both visual effect and quantitative measurement.

An Image Super-Resolution Reconstruction Method with Single Frame Character Based on Wavelet Neural Network in Internet of Things

The application of the traditional single frame character image super-resolution reconstruction method has some problems, such as noise can not be removed completely and anti-interference performance is poor. A new method for the super-resolution reconstruction of single frame character image based on wavelet neural network is proposed. The structure and interface of image acquisition unit of solid state image sensor are designed. Combined with pinhole imaging model and camera self-calibration, image acquisition of Internet of Things is completed. An image degradation model was established to simulate the degradation process of ideal high-resolution image to low-resolution image. Wavelet threshold denoising method is used to remove the noise in a single frame character image and improve the anti-interference performance of the method. The wavelet neural network reflection model is used to reconstruct the single frame feature image and improve the resolution of the image. The experimental results show that the blur degree of the reconstructed image is always less than 5%. In the whole experiment, the accuracy of this method can be maintained at 80% ~ 90%. The image detail retention rate of the research method is relatively stable. With the increase of the number of experimental images, the retention rate of image details remains between 80% and 95%, indicating that the method is effective in practical application.

Single image super-resolution via Image Quality Assessment-Guided Deep Learning Network.

In recent years, deep learning (DL) networks have been widely used in super-resolution (SR) and exhibit improved performance. In this paper, an image quality assessment (IQA)-guided single image super-resolution (SISR) method is proposed in DL architecture, in order to achieve a nice tradeoff between perceptual quality and distortion measure of the SR result. Unlike existing DL-based SR algorithms, an IQA net is introduced to extract perception features from SR results, calculate corresponding loss fused with original absolute pixel loss, and guide the adjustment of SR net parameters. To solve the problem of heterogeneous datasets used by IQA and SR networks, an interactive training model is established via cascaded network. We also propose a pairwise ranking hinge loss method to overcome the shortcomings of insufficient samples during training process. The performance comparison between our proposed method with recent SISR methods shows that the former achieves a better tradeoff between perceptual quality and distortion measure than the latter. Extensive benchmark experiments and analyses also prove that our method provides a promising and opening architecture for SISR, which is not confined to a specific network model.

Leukocyte super-resolution via geometry prior and structural consistency.

.Significance: Researchers have made great progress in single-image super-resolution (SISR) using deep convolutional neural networks. However, in the field of leukocyte imaging, the performance of existing SISR methods is still limited as it fails to thoroughly explore the geometry and structural consistency of leukocytes. The inaccurate super-resolution (SR) results will hinder the pathological study of leukocytes, since the structure and cell lineage determine the types of leukocyte and will significantly affect the subsequent inspection.Aim: We propose a deep network that takes full use of the geometry prior and structural consistency of the leukocyte images. We establish and annotate a leukocyte dataset, which contains five main types of leukocytes (basophil, eosinophil, monocyte, lymphocyte, and neutrophil), for learning the structure and geometry information.Approach: Our model is composed of two modules: prior network and SR network. The prior network estimates the parsing map of the low-resolution (LR) image, and then the SR network takes both the estimated parsing map and LR image as input to predict the final high-resolution image.Result: Experiments show that the geometry prior and structural consistency in use obviously improves the SR performance of leukocyte images, enhancing the peak-signal-to-noise ratio (PSNR) by about 0.4 dB in our benchmark.Conclusion: As proved by our leukocyte SR benchmark, the proposed method significantly outperforms state-of-the-art SR methods. Our method not only improves the PSNR and structural similarity indices, but also accurately preserves the structural details of leukocytes. The proposed method is believed to have potential use in the wide-field cell prescreening by simply using a low-power objective.

Meta-USR: A Unified Super-Resolution Network for Multiple Degradation Parameters.

Recent research on single image super-resolution (SISR) has achieved great success due to the development of deep convolutional neural networks. However, most existing SISR methods merely focus on super-resolution of a single fixed integer scale factor. This simplified assumption does not meet the complex conditions for real-world images which often suffer from various blur kernels or various levels of noise. More importantly, previous methods lack the ability to cope with arbitrary degradation parameters (scale factors, blur kernels, and noise levels) with a single model. A few methods can handle multiple degradation factors, e.g., noninteger scale factors, blurring, and noise, simultaneously within a single SISR model. In this work, we propose a simple yet powerful method termed meta-USR which is the first unified super-resolution network for arbitrary degradation parameters with meta-learning. In Meta-USR, a meta-restoration module (MRM) is proposed to enhance the traditional upscale module with the capability to adaptively predict the weights of the convolution filters for various combinations of degradation parameters. Thus, the MRM can not only upscale the feature maps with arbitrary scale factors but also restore the SR image with different blur kernels and noise levels. Moreover, the lightweight MRM can be placed at the end of the network, which makes it very efficient for iteratively/repeatedly searching the various degradation factors. We evaluate the proposed method through extensive experiments on several widely used benchmark data sets on SISR. The qualitative and quantitative experimental results show the superiority of our Meta-USR.

Sparse Representation based Super-Resolution of MRI Images with Non-Local Total Variation Regularization

Diffusion-weighted (DW) and spectroscopic MR (MRS) images are found to be very helpful for diagnostic purposes as they provide complementary information to that provided by conventional MRI. These images are also acquired at a faster rate, but with low signal-to-noise ratio. This limitation can be overcome by applying image super-resolution techniques. In this paper, we propose sparse representation over a learned overcomplete dictionary based single-image super-resolution (SISR) technique for DW and MRS images. The proposed SISR method incorporates patch-wise sparsity constraint based on external HR information together with the non-local total variation (NLTV) as internal information to make the regularization problem more robust. Experiments are conducted for both DW and MRS test images and results are compared with some of the recent methods. Results indicate the potential of the proposed method for clinical MRI applications.

(SARN)spatial-wise attention residual network for image super-resolution

Recent research suggests that attention mechanism is capable of improving performance of deep learning-based single image super-resolution (SISR) methods. In this work, we propose a deep spatial-wise attention residual network (SARN) for SISR. Specifically, we propose a novel spatial attention block (SAB) to rescale pixel-wise features by explicitly modeling interdependencies between pixels on each feature map, encoding where (i.e., attentive spatial pixels in feature map) the visual attention is located. A modified patch-based non-local block can be inserted in SAB to capture long-distance spatial contextual information and relax the local neighborhood constraint. Furthermore, we design a bottleneck spatial attention module to widen the network so that more information is allowed to pass. Meanwhile, we adopt local and global residual connections in SISR to make the network focus on learning valuable high-frequency information. Extensive experiments show the superiority of the proposed SARN over the state-of-art methods on benchmark datasets in both accuracy and visual quality.

E-ProSRNet: An enhanced progressive single image super-resolution approach

In recent years, the convolutional neural networks (CNNs) have been successfully applied to single image super-resolution (SISR) task. However, most of the CNN based SISR methods obtain better performance with a huge amount of training parameters which increases the computational complexity of their SISR models. Such SR networks suffer from a heavy burden on computational assets and as a result, they are no longer appropriate for many real-world applications. Hence, in the computer vision community, it is an interest to endorse an SR approach which makes use of less number of training parameters with better SR performance. In this paper, we propose a computationally efficient SR approach called enhanced progressive super-resolution network i.e., E-ProSRNet. This approach is the enhanced version of our base proposed model called ProSRNet. In E-ProSRNet model, we propose a novel enhanced parallel densely connected residual network (E-PDRN) which helps to extract rich features in the low-resolution (LR) observation. The SR performance of proposed E-ProSRNet model is better than that of ProSRNet and it uses a less number of training parameters when compared to that of ProSRNet model. The experimental analysis on common testing benchmark datasets shows that the proposed E-ProSRNet sets new state-of-the-art performance on SISR task for upscaling factor ×4. The E-ProSRNet method obtains better SR performance when compared to that obtained using proposed ProSRNet as well as the other state-of-the-art methods with significant reduction in the computational complexity.

Deep ResNet Based Remote Sensing Image Super-Resolution Reconstruction in Discrete Wavelet Domain

We present a single-image super-resolution (SR) method for Remote Sensing Image based on deep learning within Discrete Wavelet Domain in this paper. Our method is inspired Residual Learning. Firstly, an input image is decomposed by single level 2D Discrete wavelet transform to get four sub-bands. The four sub-bands coefficients are feeding into the Deep Learning Residual Network to predict correspondingly residual images; Adding four sub-band images and residual images as the new sub-bands of 2D wavelet transform; Finally, uses the inverse 2D Discrete wavelet transform to get the final output Super Resolution HR image. Our proposed method performs better than existing methods in accuracy and visual improvements in our results are easily noticeable.

Residual network with detail perception loss for single image super-resolution

Recently, deep convolutional neural networks have demonstrated high-quality reconstruction for single image super-resolution. In this study, we present a network by using residual blocks with cascading simple blocks to improve the image resolution. Cascading simple blocks with a multi-layer perceptron are conducive to extract features and approximate a complex mapping with fewer parameters. Skip connections can help to alleviate the vanishing-gradient problem of deep networks. In addition, our network contains two pathways. One is to predict the high frequency information of the high resolution image and the other is to predict the low frequency information of the high resolution image. Then the information of two pathways is fused, and pixel-shuffle is used for upsampling. Moreover, to capture texture details of images, we introduce a novel loss function called detail perception loss, which is used to measure the difference of the wavelet coefficients from the reconstructed image and ground truth. By reducing detail perception loss, texture details of the reconstructed image are becoming more similar with texture details of ground truth. Extensive quantitative and qualitative experiments on four benchmark datasets show that our method achieves superior performance over typical single image super-resolution methods.

Remote-Sensing Image Superresolution Based on Visual Saliency Analysis and Unequal Reconstruction Networks

Remote-sensing images (RSIs) generally have strong spatial characteristics for surface features. Various ground objects, such as residential areas, roads, forests, and rivers, differ substantially. According to this visual attention characteristic, regions with complicated texture features require more realistic details to reflect a better description of the topography, while regions such as farmlands should be smooth and have less noise. However, most existing single-image superresolution (SISR) methods fail to fully utilize these properties and therefore apply a uniform reconstruction strategy to the whole image. In this article, we propose a novel saliency-driven unequal single-image reconstruction network in which the demands of various regions in the superresolution (SR) process are distinguished by saliency maps. First, we design a new gradient-based saliency analysis method to produce more accurate saliency maps with imagewise annotations. The method utilizes the superiority of a multireception field to extract both high-level features and low-level features. Second, we propose a novel saliency-driven gate conditional generative adversarial network, where the saliency map is regarded as a medium during the training procedure of the whole network. The saliency map is regarded as a pixelwise condition in a generator to enhance the training capability of the network. Additionally, we design a new loss function that combines normalized content loss, saliency-driven perceptual loss, and gate-control adversarial loss to further refine details of texture-complex areas for RSIs. We evaluate the performance of our algorithm and compare it with many other state-of-the-art SR methods using a remote-sensing data set. The experimental results show that our approach achieves the optimal outcome in salient areas. Our method attains the best effect on global quality and visual performance.

Single image super resolution via wavelet transform fusion and SRFeat network

Image super resolution is a vital research topic in the field of computer vision. It aims to reconstruct high resolution images from low resolution images. Although the conventional image super resolution methods have achieved good performance and effect, there are still have some issues, e.g., the high-frequency details information is insufficient, and the reconstruction process will bring additional noise, and most basic interpolation techniques produce blurry results. To settle the problems mentioned above, we consider combining the deep learning method with the frequency domain fusion method. In this paper, a novel single image super resolution method based on SRFeat network and wavelet fusion is proposed. First, the training image is taken as the input of the backbone SRFeat network, then the generative adversarial network training is carried out. Then, the up-sampling is utilized to obtain the coarse super resolved image. Finally, the output image after the network training is combined with the up-sampling image of the low-resolution image by Wavelet fusion to obtain the final result. Without increasing the depth of the network and the redundant parameters, the proposed method can achieve better reconstruct result. The experimental results show that the proposed method can not only reduce the probability of image distortion, but recover the global information of the reconstructed image and remove the noise brought by the reconstruction process. The PSNR value of the proposed method is improved 0.3 dB, and the SSIM is improved 0.02.

Deep residual networks with a fully connected reconstruction layer for single image super-resolution

Recently, deep neural networks have achieved impressive performance in terms of both reconstruction accuracy and efficiency for single image super-resolution (SISR). However, the network model of these methods is a fully convolutional neural network, which is limited to exploit the differentiated contextual information over the global region in the input image because of the weight sharing in convolution height and width extent. In this paper, we discuss a new SISR method where features are extracted in the low-resolution (LR) space, and then we use a fully connected layer which learns an array of upsampling weights to reconstruct the desired high-resolution (HR) image from the final obtained LR features. By doing so, we effectively exploit the differentiated contextual information over the input image, whilst maintaining the low computational complexity for the overall SR operations. In addition, we introduce an edge difference constraint into our loss function to preserve edges and restore textures. Extensive experiments validate that our method outperforms the existing state-of-the-art SISR methods.

Lightweight image super-resolution with feature enhancement residual network

Recently, single image super-resolution (SR) methods based on deep convolutional neural network (CNN) have demonstrated remarkable progress. The essence of most CNN-based models is to learn the non-linear mapping between low-resolution patches and corresponding high-resolution ones. However, numerous convolutions are applied to implement this mapping, which directly contributes to large model sizes and huge graphics memory consumption. In this paper, we propose a lightweight feature enhancement residual network (FERN) to achieve prominent performance by incorporating lightweight non-local operations into the residual block. By taking advantage of utilizing this non-locally enhanced residual block, the proposed model can capture long-range dependencies. For further improving performance, we design the structure-aware channel attention layer that explicitly boosts feature maps with more structural and textural details. Extensive experiments suggest that the proposed approach performs favorably against the state-of-the-art SR algorithms in terms of visual quality and inference time.

Reference Image Guided Super-Resolution via Progressive Channel Attention Networks

In recent years, the convolutional neural networks (CNNs) for single image super-resolution (SISR) are becoming more and more complex, and it is more challenging to improve the SISR performance. In contrast, the reference image guided super-resolution (RefSR) is an effective strategy to boost the SR (super-resolution) performance. In RefSR, the introduced high-resolution (HR) references can facilitate the high-frequency residual prediction process. According to the best of our knowledge, the existing CNN-based RefSR methods treat the features from the references and the low-resolution (LR) input equally by simply concatenating them together. However, the HR references and the LR inputs contribute differently to the final SR results. Therefore, we propose a progressive channel attention network (PCANet) for RefSR. There are two technical contributions in this paper. First, we propose a novel channel attention module (CAM), which estimates the channel weighting parameter by weightedly averaging the spatial features instead of using global averaging. Second, considering that the residual prediction process can be improved when the LR input is enriched with more details, we perform super-resolution progressively, which can take advantage of the reference images in multi-scales. Extensive quantitative and qualitative evaluations on three benchmark datasets, which represent three typical scenarios for RefSR, demonstrate that our method is superior to the state-of-the-art SISR and RefSR methods in terms of PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity).

MAMNet: Multi-path adaptive modulation network for image super-resolution

In recent years, single image super-resolution (SR) methods based on deep convolutional neural networks (CNNs) have made significant progress. However, due to the non-adaptive nature of the convolution operation, they cannot adapt to various characteristics of images, which limits their representational capability and, consequently, results in unnecessarily large model sizes. To address this issue, we propose a novel multi-path adaptive modulation network (MAMNet). Specifically, we propose a multi-path adaptive modulation block (MAMB), which is a lightweight yet effective residual block that adaptively modulates residual feature responses by fully exploiting their information via three paths. The three paths model three types of information suitable for SR: 1) channel-specific information (CSI) using global variance pooling, 2) inter-channel dependencies (ICD) based on the CSI, 3) and channel-specific spatial dependencies (CSD) via depth-wise convolution. We demonstrate that the proposed MAMB is effective and parameter-efficient for image SR than other feature modulation methods. In addition, experimental results show that our MAMNet outperforms most of the state-of-the-art methods with a relatively small number of parameters.

Remote Sensing Single-Image Resolution Improvement Using A Deep Gradient-Aware Network with Image-Specific Enhancement

Super-resolution (SR) is able to improve the spatial resolution of remote sensing images, which is critical for many practical applications such as fine urban monitoring. In this paper, a new single-image SR method, deep gradient-aware network with image-specific enhancement (DGANet-ISE) was proposed to improve the spatial resolution of remote sensing images. First, DGANet was proposed to model the complex relationship between low- and high-resolution images. A new gradient-aware loss was designed in the training phase to preserve more gradient details in super-resolved remote sensing images. Then, the ISE approach was proposed in the testing phase to further improve the SR performance. By using the specific features of each test image, ISE can further boost the generalization capability and adaptability of our method on inexperienced datasets. Finally, three datasets were used to verify the effectiveness of our method. The results indicate that DGANet-ISE outperforms the other 14 methods in the remote sensing image SR, and the cross-database test results demonstrate that our method exhibits satisfactory generalization performance in adapting to new data.

Deep quantification down-plain-upsampling residual learning for single image super-resolution

Deep convolutional neural networks have been widely used in single image super-resolution (SISR) with great success. However, the performance and efficiency of such models need to be improved for practical applications. In this paper, a novel deep quantification down-plain-upsampling (QDPU) network for SISR is proposed. In the framework, a down-plain-upsampling (DPU) residual block based on U-Net is firstly designed to reduce the computational cost by transforming the spatial scale of feature maps without sacrificing the reconstruction performance. Then, to better transmit low-level features to the reconstruction layer, we construct quantification skip-connection modules to integrate the outputs of the DPU residual blocks. Finally, QDPU is developed by stacking the DPU residual blocks with multiple skip-connections to reconstruct high-resolution images and reduce the computational burden. Quantitative and qualitative evaluations of the reconstruction results on four benchmark datasets show that the proposed method can achieve better performance compared with several state-of-the-art SISR methods.