Single Image Super-resolution Methods Research Articles

Enhancement of Acoustic Microscopy Lateral Resolution: A Comparison Between Deep Learning and Two Deconvolution Methods.

Scanning acoustic microscopy (SAM) provides high-resolution images of biological tissues. Since higher transducer frequencies limit penetration depth, image resolution enhancement techniques could help in maintaining sufficient lateral resolution without sacrificing penetration depth. Compared with existing SAM research, this work introduces two novelties. First, deep learning (DL) is used to improve lateral resolution of 180-MHz SAM images, comparing it with two deconvolution-based approaches. Second, 316-MHz images are used as ground truth in order to quantitatively evaluate image resolution enhancement. The samples used were mouse and rat brain sections. The results demonstrate that DL can closely approximate ground truth (NRMSE = 0.056 and PSNR = 28.4 dB) even with a relatively limited training set (four images, each smaller than 1 mm ×1 mm). This study suggests the high potential of using DL as a single image superresolution method in SAM.

Remote Sensing Single-Image Superresolution Based on a Deep Compendium Model

This letter introduces a novel remote sensing single-image superresolution (SR) architecture based on a deep efficient compendium model. The current deep learning-based SR trend stands for using deeper networks to improve the performance. However, this practice often results in the degradation of visual results. To address this issue, the proposed approach harmonizes several different improvements on the network design to achieve state-of-the-art performance when superresolving remote sensing imagery. On the one hand, the proposal combines residual units and skip connections to extract more informative features on both local and global image areas. On the other hand, it makes use of parallelized $1\times 1$ convolutional filters (network in network) to reconstruct the superresolved result while reducing the information loss through the network. Our experiments, conducted using seven different SR methods over the well-known UC Merced remote sensing data set, and two additional GaoFen-2 test images, show that the proposed model is able to provide competitive advantages.

Learning Image Profile Enhancement and Denoising Statistics Priors for Single-Image Super-Resolution

Single-image super-resolution (SR) has been widely used in computer vision applications. The reconstruction-based SR methods are mainly based on certain prior terms to regularize the SR problem. However, it is very challenging to further improve the SR performance by the conventional design of explicit prior terms. Because of the powerful learning ability, deep convolutional neural networks (CNNs) have been widely used in single-image SR task. However, it is difficult to achieve further improvement by only designing the network architecture. In addition, most existing deep CNN-based SR methods learn a nonlinear mapping function to directly map low-resolution (LR) images to desirable high-resolution (HR) images, ignoring the observation models of input images. Inspired by the split Bregman iteration (SBI) algorithm, which is a powerful technique for solving the constrained optimization problems, the original SR problem is divided into two subproblems: 1) inversion subproblem and 2) denoising subproblem. Since the inversion subproblem can be regarded as an inversion step to reconstruct an intermediate HR image with sharper edges and finer structures, we propose to use deep CNN to capture low-level explicit image profile enhancement prior (PEP). Since the denoising subproblem aims to remove the noise in the intermediate image, we adopt a simple and effective denoising network to learn implicit image denoising statistics prior (DSP). Furthermore, the penalty parameter in SBI is adaptively tuned during the iterations for better performance. Finally, we also prove the convergence of our method. Thus, the deep CNNs are exploited to capture both implicit and explicit image statistics priors. Due to SBI, the SR observation model is also leveraged. Consequently, it bridges between two popular SR approaches: 1) learning-based method and 2) reconstruction-based method. Experimental results show that the proposed method achieves the state-of-the-art SR results.

Single image super-resolution reconstruction based on multi-scale feature mapping adversarial network

Single image super-resolution (SISR) aims to reconstruct a high-resolution image from a degraded low-resolution image. In recent years, the super-resolution methods based on convolutional neural network (CNN) have achieved promising performance on SISR task, indicating that CNN is a viable approach to image super-resolution reconstruction. The one limitation of the current SISR methods is that many methods use the pixel-wise loss. It is well known that the pixel-wise loss cannot well recover high-frequency details even if the high peak signal-to-noise ratio (PSNR) can be obtained. Some other methods purely focus on restoring more details, which resulted in poor PSNR score and high-frequency noise. In this paper, we proposed a multi-component loss function based on pixel-wise loss, perceptual loss and adversarial loss for a multi-scale feature mapping generator network for SISR image reconstruction model. We evaluated our method on commonly used benchmarks and compared it with other SISR methods. The results showed that our method could achieve the better balance between the high-frequency detail and stable spatial structure generation.

Superresolution of Single Gaofen-4 Visible-Light and Near-Infrared (VNIR) Image Based on Texture Image Extraction

Superresolution (SR) is an important compensatory mean for geostationary orbit (GEO) satellites because of their low spatial resolution caused by orbital height. The multiple-image SR (MISR) methods generally have better reconstruction effects than single-image SR (SISR) methods. However, MISR methods require multiple images of the same place, which are susceptible to geometric distortion and radiation differences. Gaofen-4 (GF-4) is a GEO satellite equipped with a visible-light and near-infrared (VNIR) sensor and provides images with five bands. Based on the similarity of spatial information between different bands, this paper proposes an SISR method suitable for GF-4 VNIR images. The radiation differences are eliminated by extracting texture images. The high-resolution (HR) texture image is reconstructed by projections onto convex sets. The final SR result is fused by HR texture image and upsampled VNIR image. Tested by real GF-4 data, the reconstructed results of this method have better sharpness and more abundant information than comparable methods.

Generating High-Quality Air-Coupled Ultrasonic Images for Wooden Material Characterization by Single Image Super-Resolution

As the practical applications in other fields, high-resolution images are usually expected to provide a more accurate assessment for the air-coupled ultrasonic (ACU) characterization of wooden materials. This paper investigated the feasibility of applying single image super-resolution (SISR) methods to recover high-quality ACU images from the raw observations that were constructed directly by the on-the-shelf ACU scanners. Four state-of-the-art SISR methods were applied to the low-resolution ACU images of wood products. The reconstructed images were evaluated by visual assessment and objective image quality metrics, including peak signal-to-noise-ratio and structural similarity. Both qualitative and quantitative evaluations indicated that the substantial improvement of image quality can be yielded. The results of the experiments demonstrated the superior performance and high reproducibility of the method for generating high-quality ACU images. Sparse coding based super-resolution and super-resolution convolutional neural network (SRCNN) significantly outperformed other algorithms. SRCNN has the potential to act as an effective tool to generate higher resolution ACU images due to its flexibility.

Image super-resolution using TV priori guided convolutional network

We propose a TV priori guided deep learning method for single image super-resolution (SR). The new up-sample method based on TV priori information, new learning algorithm and neural networks architecture are embraced in our TV Priori Guided Convolutional Neural Network which directly learns an end-to-end mapping between the low/high-resolution images. There are three aspects of the innovation of our algorithm. First, unlike most super-resolution algorithms, an input image is up-sampled via Moms interpolation before they fed into the network. Then, we use a set of discrete TV council's templates to extract priori information. Second, we put priori information into non-local regression frame, and propose a non-local TV interpolation; Third, non-local TV interpolated images are fed into the network. We conduct a number of experiments to evaluate the effectiveness of TV priori guided CNN.

A Convolutional Neural Network for 250-MHz Quantitative Acoustic-microscopy Resolution Enhancement.

Quantitative acoustic microscopy (QAM) permits the formation of quantitative two-dimensional (2D) maps of acoustic and mechanical properties of soft tissues at microscopic resolution. The 2D maps formed using our custom SAM systems employing a 250-MHz and a 500-MHz single-element transducer have a nominal resolution of 7 μm and 4μm, respectively. In a previous study, the potential of single-image super-resolution (SR) image post-processing to enhance the spatial resolution of 2D SAM maps was demonstrated using a forward model accounting for blur, decimation, and noise. However, results obtained when the SR method was applied to soft tissue data were not entirely satisfactory because of the limitation of the convolution model considered and by the difficulty of estimating the system point spread function and designing the appropriate regularization term. Therefore, in this study, a machine learning approach based on convolutional neural networks was implemented. For training, data acquired on the same samples at 250 and 500 MHz were used. The resulting trained network was tested on 2D impedance maps (2DZMs) of human lymph nodes acquired from breast-cancer patients. Visual inspection of the reconstructed enhanced 2DZMs were found similar to the 2DZMs obtained at 500 MHz which were used as ground truth. In addition, the enhanced 250-MHz 2DZMs obtained from the proposed method yielded better peak signal to noise ratio and normalized mean square error than those obtained with the previous SR method. This improvement was also demonstrated by the statistical analyses. This pioneering work could significantly reduce challenges and costs associated with current very high-frequency SAM systems while providing enhanced spatial resolution.

Deep Inception-Residual Laplacian Pyramid Networks for Accurate Single-Image Super-Resolution

With exploiting contextual information over large image regions in an efficient way, the deep convolutional neural network has shown an impressive performance for single-image super-resolution (SR). In this paper, we propose a new deep convolutional network by cascading multiple well-designed inception-residual blocks within the deep Laplacian pyramid framework to progressively restore the missing high-frequency details in the low-resolution images. By optimizing our network structure, the trainable depth of our proposed network gains a significant improvement, which in turn improves super-resolving accuracy. However, the saturation and degradation of training accuracy remains a critical problem. With regard to this, we propose an effective two-stage training strategy, in which we first use the images downsampled from the ground-truth high-resolution (HR) images to pretrain the inception-residual blocks on each pyramid level with an extremely high learning rate enabled by gradient clipping, and then the original ground-truth HR images are used to fine-tune all the pretrained inception-residual blocks for obtaining our final SR models. Furthermore, we present a new loss function operating in both image space and local rank space to optimize our network for exploiting the contextual information among different output components. Extensive experiments on benchmark data sets validate that the proposed method outperforms the existing state-of-the-art SR methods in terms of the objective evaluation as well as the visual quality.

Light Field Super-Resolution By Jointly Exploiting Internal and External Similarities

Light field images taken by plenoptic cameras often have a tradeoff between spatial and angular resolutions. In this paper, we propose a novel spatial super-resolution approach for light field images by jointly exploiting internal and external similarities. The internal similarity refers to the correlations across the angular dimensions of the 4D light field itself, while the external similarity refers to the cross-scale correlations learned from an external light field dataset. Specifically, we advance the classic projection-based method that exploits the internal similarity by introducing the intensity consistency checking criterion and a back-projection refinement, while the external correlation is learned by a CNN-based method which aggregates all warped high-resolution sub-aperture images upsampled from the low-resolution input using a single image super-resolution method. By analyzing the error distributions of the above two methods and investigating the upperbound of combining them, we find that the internal and external similarities are complementary to each other. Accordingly, we further propose a pixel-wise adaptive fusion network to take advantage of both their merits by learning a weighting matrix. Experimental results on both synthetic and real-world light field datasets validate the superior performance of the proposed approach over the state-of-the-arts.

Channel Splitting Network for Single MR Image Super-Resolution.

High resolution magnetic resonance (MR) imaging is desirable in many clinical applications due to its contribution to more accurate subsequent analyses and early clinical diagnoses. Single image super-resolution (SISR) is an effective and cost efficient alternative technique to improve the spatial resolution of MR images. In the past few years, SISR methods based on deep learning techniques, especially convolutional neural networks (CNNs), have achieved the state-of-the-art performance on natural images. However, the information is gradually weakened and training becomes increasingly difficult as the network deepens. The problem is more serious for medical images because lacking high quality and effective training samples makes deep models prone to underfitting or overfitting. Nevertheless, many current models treat the hierarchical features on different channels equivalently, which is not helpful for the models to deal with the hierarchical features discriminatively and targetedly. To this end, we present a novel channel splitting network (CSN) to ease the representational burden of deep models. The proposed CSN model divides the hierarchical features into two branches, i.e., residual branch and dense branch, with different information transmissions. The residual branch is able to promote feature reuse, while the dense branch is beneficial to the exploration of new features. Besides, we also adopt the merge-and-run mapping to facilitate information integration between different branches. The extensive experiments on various MR images, including proton density (PD), T1, and T2 images, show that the proposed CSN model achieves superior performance over other state-of-the-art SISR methods.

SISRSet: Single image super-resolution subjective evaluation test and objective quality assessment

With the outstanding performance of deep learning based single image super-resolution (SISR) methods, some widely used quality evaluation metrics such as peak signal to noise ratio (PSNR) and structure similarity index (SSIM) have become more and more difficult to meet the assessment requirements of SISR methods, especially in terms of their consistency with subjective visual perception. To deal with this super-resolution (SR) image quality assessment (IQA) issue, it calls for a specialized objective evaluation metric based on the visual characteristics of SR images and the human visual system (HVS). Notwithstanding, there is a lack of practical databases for analysis. In this paper, the SISR subjective evaluation tests are conducted to build an SISRSet database, which requires a large number of reconstructed images generated by various SR algorithms. The statistical analyses of SISRSet reveal that high-frequency texture details play an important role in the performance evaluation of SR algorithms. Inspired by the origination selectivity mechanism (OSM) and the internal generative mechanism (IGM) of the HVS, a visual content prediction model (VCPM) is proposed to measure different visible structural contents of SR images, especially the texture details. Finally, a novel SISR quality assessment metric is devised based on the VCPM similarity comparison between the references and SR images. The experimental results demonstrate that the proposed SISR-IQA metric does well in the performance evaluation of SISR methods and well correlates with those by the subjective evaluation.

Accurate single image super‐resolution using cascading dense connections

In recent years, deep learning methods, especially deep convolutional neural network, have been successfully applied to the single-image super-resolution (SISR) task. In this Letter, the authors propose an accurate SISR method by introducing cascading dense connections in a very deep network. In detail, they construct the cascading dense network (CDN) to fully make use of the hierarchical features from all the convolutional layers, which implements a cascading mechanism upon the dense connected convolutional layers. The cascading dense connection in the CDN enables short and long paths to be built directly from the output to each layer, alleviating the vanishing-gradient problem of very deep networks. Extensive experiments show that CDN achieves state-of-the-art performance on traditional metrics (PSNR and SSIM). Also, they introduce object recognition as the additional evaluation metric for SISR, which further demonstrates the effectiveness of the authors’ method.

A Sparse Analysis-Based Single Image Super-Resolution

In the current study, we were inspired by sparse analysis signal representation theory to propose a novel single-image super-resolution method termed “sparse analysis-based super resolution” (SASR). This study presents and demonstrates mapping between low and high resolution images using a coupled sparse analysis operator learning method to reconstruct high resolution (HR) images. We further show that the proposed method selects more informative high and low resolution (LR) learning patches based on image texture complexity to train high and low resolution operators more efficiently. The coupled high and low resolution operators are used for high resolution image reconstruction at a low computational complexity cost. The experimental results for quantitative criteria peak signal to noise ratio (PSNR), root mean square error (RMSE), structural similarity index (SSIM) and elapsed time, human observation as a qualitative measure, and computational complexity verify the improvements offered by the proposed SASR algorithm.

Single image super-resolution via multi-scale residual channel attention network

Recently, various convolutional neural networks (CNNs) based single image super-resolution (SR) methods have been vigorously explored, and a lot of impressive results have emerged. However, more or less unfortunately, most of the methods mainly focused on increasing the depth of network to improve reconstruction performance. As a matter of fact, deeper depth of network usually means an increase in parameters and computations, or worse still, the increase in parameters or computations often results in the difficulty to train the network. This paper develops a new SR approach called multi-scale residual channel attention network (MSRCAN), which is comparative shallow two-stage neural network structure, and can extract more details to effectively ameliorate the quality of SR. Specifically, a multi-scale residual channel attention block (MSRCAB) is designed to plenarily exploit the image features with convolutional kernels of different sizes. At the same time, a channel attention mechanism is introduced to recalibrate the channel significance of feature mappings adaptively. Furthermore, multiple short skip connections and a long skip connection are presented in each MSRCAB to complement information loss. Moreover, the two-stage design contributes to fully uncover low-level and high-level information. Evaluation on the benchmark data set indicates that the proposed method can rival the state-of-the-art convolutional methods.

Speedy and accurate image super‐resolution via deeply recursive CNN with skip connection and network in network

The single image super-resolution (SISR) methods based on the deep convolutional neural network (CNN) have recently achieved significant improvements in accuracy, advancing the state of the art. However, these deeper models are computationally expensive and require a large number of parameters. Accordingly, they demand more memory and are unsuitable for on-chip devices. In this study, a novel SISR method using a deeply recursive CNN with skip connections and a network in network structure is proposed. The deeply recursive CNN with skip connections is adopted for the image feature extraction at both local and global levels. Parallelised 1 × 1 CNNs, usually called a network in network structure, are adopted for image reconstruction. Specifically, recursive learning is utilised to control the number of model parameters needed and residual learning is used to ease the difficulty of training. The proposed method performs favourably against the state-of-the-art methods in terms of computational speed and accuracy. It significantly outperforms the previous methods by a large margin, while demanding far fewer parameters. This model requires less memory and is friendly to on-chip devices.

Collaborative Representation Cascade for Single-Image Super-Resolution

Most recent learning-based single-image super-resolution methods first interpolate the low-resolution (LR) input, from which overlapped LR features are then extracted to reconstruct their high-resolution (HR) counterparts and the final HR image. However, most of them neglect to take advantage of the intermediate recovered HR image to enhance image quality further. We conduct principal component analysis (PCA) to reduce LR feature dimension. Then we find that the number of principal components after conducting PCA in the LR feature space from the reconstructed images is larger than that from the interpolated images by using bicubic interpolation. Based on this observation, we present an unsophisticated yet effective framework named collaborative representation cascade (CRC) that learns multilayer mapping models between LR and HR feature pairs. In particular, we extract the features from the intermediate recovered image to upscale and enhance LR input progressively. In the learning phase, for each cascade layer, we use the intermediate recovered results and their original HR counterparts to learn single-layer mapping model. Then, we use this single-layer mapping model to super-resolve the original LR inputs. And the intermediate HR outputs are regarded as training inputs for the next cascade layer, until we obtain multilayer mapping models. In the reconstruction phase, we extract multiple sets of LR features from the LR image and intermediate recovered. Then, in each cascade layer, mapping model is utilized to pursue HR image. Our experiments on several commonly used image SR testing datasets show that our proposed CRC method achieves state-of-the-art image SR results, and CRC can also be served as a general image enhancement framework.

Deep recursive up-down sampling networks for single image super-resolution

Single image super-resolution (SISR) technology can reconstruct a high-resolution (HR) image from the corresponding low-resolution (LR) image. The emergence of deep learning pushes SISR to a new level. The successful application of the recursive network motivates us to explore a more efficient SISR method. In this paper, we propose the deep recursive up-down sampling networks (DRUDN) for SISR. In DRUDN, an original LR image is directly fed without extra interpolation. Then, we use the sophisticated recursive up-down sampling blocks (RUDB) to learn the complex mapping between the LR image and the HR image. At the reconstruction part, the feature map is up-scaled to the ideal size by a de-convolutional layer. Extensive experiments demonstrate that DRUDN outperforms the state-of-the-art methods in both subjective effects and objective evaluation.

Data-adaptive low-rank modeling and external gradient prior for single image super-resolution

Image super-resolution (SR) is a challenging task which aims to recover the high-resolution (HR) images from the degraded low-resolution (LR) observations. To address this ill-posed problem, properly exploiting the image prior is of great importance. In this paper, we propose a data-adaptive low-rank (DLR) model. Rather than directly assuming that the rank of a group of similar patches is low, the DLR model imposes the low-rank property on the residual of the grouped patches. In addition, the shape of the patches in our DLR model is adapted to the contents of images, so that the dissimilar pixels in a group of patches can be largely reduced. In order to further boost the performance, an external gradient prior (EGP), which is learned externally to capture gradient information, is combined with DLR to form a joint prior. When solving the DLR-based and the joint-prior-based minimization problems, the split Bregman method is adopted to speed up the convergence. The extensive experimental results show that our algorithms outperform many state-of-the-art single image SR methods in terms of both objective and subjective qualities.

Enhanced Non-Local Total Variation Model and Multi-Directional Feature Prediction Prior for Single Image Super Resolution.

It is widely acknowledged that single image super-resolution (SISR) methods play a critical role in recovering the missing high-frequencies in an input low-resolution image. As SISR is severely ill-conditioned, image priors are necessary to regularize the solution spaces and generate the corresponding high-resolution image. In this paper, we propose an effective SISR framework based on the enhanced non-local similarity modeling and learning-based multi-directional feature prediction (ENLTV-MDFP). Since both the modeled and learned priors are exploited, the proposed ENLTV-MDFP method benefits from the complementary properties of the reconstruction-based and learning-based SISR approaches. Specifically, for the non-local similarity-based modeled prior [enhanced non-local total variation, (ENLTV)], it is characterized via the decaying kernel and stable group similarity reliability schemes. For the learned prior [multi-directional feature prediction prior, (MDFP)], it is learned via the deep convolutional neural network. The modeled prior performs well in enhancing edges and suppressing visual artifacts, while the learned prior is effective in hallucinating details from external images. Combining these two complementary priors in the MAP framework, a combined SR cost function is proposed. Finally, the combined SR problem is solved via the split Bregman iteration algorithm. Based on the extensive experiments, the proposed ENLTV-MDFP method outperforms many state-of-the-art algorithms visually and quantitatively.