Deep Learning-based Super-resolution Methods Research Articles

A deep learning-based super-resolution method for building height estimation at 2.5 m spatial resolution in the Northern Hemisphere

Building height is an important indicator for assessing the level of urban development along the vertical dimension. Existing large-scale building height estimation studies focus on coarse spatial resolution (e.g., 10, 500, and 1000 m), which cannot reveal height variations across buildings in urban areas. High-resolution images (e.g., < 5 m resolution) can support building-scale height estimation, but they usually have small spatial coverage and are not openly accessible. In this context, we proposed a deep learning-based super-resolution method to generate building height maps at a spatial resolution of 2.5 m using Sentinel-1/2 images. The proposed method consisted of two parts: 1) a super-resolution module (SR) for learning high-resolution features; and 2) a height stratification estimation module (HS) for guiding the network to learn different height levels to mitigate the imbalanced distribution of height values. We created an open building height dataset with 45,000 samples covering multiple urban areas in the Northern Hemisphere, including China, the conterminous United States (CONUS), and Europe. Experimental results showed that for height estimation at the pixel level, the proposed method obtained a root mean square error of 10.318 m in China, 5.654 m in CONUS, and 4.113 m in Europe, respectively. Predicted results provided rich spatial details, due to the inclusion of the super-resolution module, which was heavily missed by existing large-scale studies. Moreover, we calculated the mean and standard deviation of building height in 301 urban centers, each having at least a population of 500,000, and found that the buildings in China were the highest (11.353 m ± 9.543 m), followed by CONUS (8.487 m ± 6.202 m) and Europe (8.136 m ± 5.020 m). Ablation studies indicated that the joint use of Sentinel-1/2 images and the proposed modules (SR and HS) can effectively improve the performance of building height estimation. The building dataset we generated provides great potential in high-resolution database updating, urban planning, and natural disaster assessment, and indeed, a new perspective of how we can utilize cutting-edge satellite imaging technology in urban observation, measurement, monitoring, and management. The dataset and code of this study will be available at: https://github.com/lauraset/Super-resolution-building-height-estimation.

Unsupervised stepwise extraction of offshore aquaculture ponds using super-resolution hyperspectral images

Despite offshore aquaculture brings great economic benefits, it also has destructive effects on the ecological environment of coastal regions. Therefore, the accurate monitoring of offshore aquaculture areas is vital. Existing methods for extracting aquaculture ponds are still limited by the lack of high-resolution hyperspectral imagery and inadequate feature extraction. In this paper, we proposed an unsupervised aquaculture ponds extraction method based on hyperspectral imagery super-resolution, feature fusion and stepwise extraction strategy. First, the resolution of the original hyperspectral imagery is enhanced via a deep learning-based super-resolution method. Then we introduce a feature fusion method by combining multi-dimensional features including elevation, reflectance, biochemistry, principal component analysis, and fast-Fourier transform features to enhance the feature sensitivity to the aquaculture ponds. The aquaculture ponds are finally extracted via a step-wise extraction strategy. Experiments on ZH-1 and GF-7 imageries are conducted, and the experimental results show that our method achieves an OA of 97.9% on the aquaculture pond extraction and that the method can be generalized for the object extraction on multispectral datasets. Ablation experiments show that all three of our innovative modules can effectively improve the extraction accuracy and confirm the generalization of the method.

Super-resolution method for SEM images based on pixelwise weighted loss function.

Scanning electron microscopy (SEM) has realized high-throughput defect monitoring of semiconductor devices. As miniaturization and complexification of semiconductor circuit patterns increase in recent years, so has the number of defects. There is thus a great need to further increase the throughput of SEM defect monitoring. Toward this end, we propose a deep learning-based super-resolution method that reproduces high-resolution (HR) images from corresponding low-resolution images. Image quality factors such as pattern contrast and sharpness are important in SEM HR images in order to evaluate the quality of printed circuit patterns. Our proposed method meets various image quality requirements by changing the loss calculation method pixelwise based on the pattern in the image. It realizes super-resolved images that compare favorably with actual HR images and can improve SEM throughput by 100% or more.

Accurate and lightweight MRI super-resolution via multi-scale bidirectional fusion attention network.

High-resolution magnetic resonance (MR) imaging has attracted much attention due to its contribution to clinical diagnoses and treatment. However, because of the interference of noise and the limitation of imaging equipment, it is expensive to generate a satisfactory image. Super-resolution (SR) is a technique that enhances an imaging system's resolution, which is effective and cost-efficient for MR imaging. In recent years, deep learning-based SR methods have made remarkable progress on natural images but not on medical images. Most existing medical images SR algorithms focus on the spatial information of a single image but ignore the temporal correlation between medical images sequence. We proposed two novel architectures for single medical image and sequential medical images, respectively. The multi-scale back-projection network (MSBPN) is constructed of several different scale back-projection units which consist of iterative up- and down-sampling layers. The multi-scale machine extracts different scale spatial information and strengthens the information fusion for a single image. Based on MSBPN, we proposed an accurate and lightweight Multi-Scale Bidirectional Fusion Attention Network(MSBFAN) that combines temporal information iteratively. That supplementary temporal information is extracted from the adjacent image sequence of the target image. The MSBFAN can effectively learn both the spatio-temporal dependencies and the iterative refinement process with only a lightweight number of parameters. Experimental results demonstrate that our MSBPN and MSBFAN are outperforming current SR methods in terms of reconstruction accuracy and parameter quantity of the model.

Deep locally linear embedding network

Recent years have seen great progress in many deep learning-based super-resolution methods that learn global responses through the non-local strategy. However, improper linear fitting tends to produce unsatisfactory results with notable artifacts. To relieve the improper linear fitting and reuse the hierarchical features, we propose an end-to-end locally linear embedding super-resolution network, termed as LLE-Net, based on the assumption that the local geometric relationship in the low-resolution manifold space also exists in high-level feature manifold space. The proposed LLE-Net includes two major components: 1) the local linear embedding block (LLEB) and 2) the hierarchical non-local block (HNB). In particular, LLEB searches sparse and similar feature maps and embeds the geometric relationship into the high-level feature space, allowing more attention to be paid to objects’ textural details. To promote the representation ability, the proposed HNB is able to explore layer- and pixel-level interdependencies. We conduct extensive experiments to evaluate the superiority of LLE-Net via two groups of experiments: (1) super-resolution tasks on two satellite image datasets and a satellite video image dataset, and (2) the subsequent high-level image processing tasks (i.e., satellite semantic segmentation). The proposed method achieves an 0.7 dB improvement in PSNR value on the Draper dataset and 0.07% segmentation accuracy improvement. The source code of the proposed LLE-Net is publicly available.

Effects of the deep learning-based super-resolution method on thermal image classification applications

Effects of the deep learning-based super-resolution method on thermal image classification applications

Deep-Learning-Based Super-Resolution of Video Satellite Imagery by the Coupling of Multiframe and Single-Frame Models

Image super-resolution (SR) is an effective sol-ution to the limitation of the spatial resolution of video satellite images, which is caused by the degradation and compression in the imaging phase. For the processing of satellite videos, the commonly employed deep learning based single-frame SR (SFSR) framework have limited performance without using complementary information between the video frames. On the other side, the multi-frame SR (MFSR) can utilize temporal sub-pixel information to super-resolve the high-resolution (HR) imagery. However, although deeper and wider deep learning network provide powerful feature representations for SR methods, it has always been a challenge to accurately recon-struct the boundaries of ground objects in video satellite images. In this paper, to address these issues, we propose an edge-guided video super-resolution (EGVSR) framework for video satellite image SR, which couple MFSR model and edge-SFSR (E-SFSR) model in a unified network. The EGVSR framework is composed of an MFSR branch and an edge branch. The MFSR branch is used to extract the complementary features from the consecutive video frames. Concurrently, the edge branch acts as an SFSR model to translate the edge maps from the low-resolution modality to the HR one. At the final SR stage, the DBFM is built to focus on the promising inner repre-sentations of the features of the two branches and fuse them. Extensive experiments on video satellite imagery show that the proposed EGVSR method can achieve a superior performance when compared to the representative deep learning based SR methods.

A Deep Multi-Frame Super-Resolution Network for Dynamic Scenes

The multi-frame super-resolution techniques have been prosperous over the past two decades. However, little attention has been paid to the combination of deep learning and multi-frame super-resolution. One reason is that most deep learning-based super-resolution methods cannot handle variant numbers of input frames. Another reason is that it is hard to capture accurate temporal and spatial information because of the misalignment of input images. To solve these problems, we propose an optical-flow-based multi-frame super-resolution framework, which is capable of dealing with various numbers of input frames. This framework enables to make full use of the input frames, allowing it to obtain better performance. In addition, we use a spatial subpixel alignment module for more accurate subpixel-wise spatial alignment and introduce a dual weighting module to generate weights for temporal fusion. Both two modules lead to more effective and accurate temporal fusion. We compare our method with other state-of-the-art methods and conduct ablation studies on our method. The results of qualitative and quantitative analyses show that our method achieves state-of-the-art performances, demonstrating the advantage of the designed framework and the necessity of proposed modules.

Remote sensing image super-resolution using cascade generative adversarial nets

Image super-resolution (SR) is a widely used and cost-effective technology in remote sensing image processing. Deep learning-based SR methods have shown promising performance, but they are prone to losing texture details. Instead, generative adversarial nets (GAN)-based methods can generate more visually acceptable results. However, GAN-based SR methods are suffering from scene variance and uncontrollable performance of discriminators as well as unstable training. Besides, both these two methods cannot yield arbitrary high-time SR images. To solve these issues, we propose a novel SR method for remote sensing images using Cascade Generative Adversarial Nets (CGAN) with introduction of content fidelity and scene constraint, which can achieve arbitrary high-time high-quality SR image. More specifically, the scene-constraint item is incorporated to constrain generated feature for avoiding the risk of scene change. Then, content fidelity is introduced to stabilize the training and avoid gradient vanishing. Besides, an edge enhancement module is designed to preserve edge detail and suppress the noise. CGAN with these three components has achieved higher quality SR results than other recent state-of-the-art methods. Compared with these methods, our proposed method outperformed average increments of 7.3% SSIM, 7.3% FSIM and 6.0% MSIM on WHU-RS19 and NWPU-RESISC45 datasets. In addition, the evaluation of GAN-train and GAN-test gained average increments of 6.3% and 4.5% on the WHU-RS19 and AID datasets, respectively.

Deep Learning-based Super Resolution Method Using Combination of Channel Attention and Spatial Attention

Deep Learning-based Super Resolution Method Using Combination of Channel Attention and Spatial Attention

Using super-resolution generative adversarial network models and transfer learning to obtain high resolution digital periapical radiographs

Periapical Radiographs are commonly used to detect several anomalies, like caries, periodontal, and periapical diseases. Even considering that digital imaging systems used nowadays tend to provide high-quality images, external factors, or even system limitations can result in a vast amount of radiographic images with low quality and resolution. Commercial solutions offer tools based on interpolation methods to increase image resolution. However, previous literature shows that these methods may create undesirable effects in the images affecting the diagnosis accuracy. One alternative is using deep learning-based super-resolution methods to achieve better high-resolution images. Nevertheless, the amount of data for training such models is limited, demanding transfer learning approaches. In this work, we propose the use of super-resolution generative adversarial network (SRGAN) models and transfer learning to achieve periapical images with higher quality and resolution. Moreover, we evaluate the influence of using the transfer learning approach and the datasets selected for it in the final generated images. For that, we performed an experiment comparing the performance of the SRGAN models (with and without transfer learning) with other super-resolution methods. Considering Mean Square Error (MSE), Peak Signal to Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Mean Opinion Score (MOS), the results of SRGAN models using transfer learning were better on average. This superiority was also verified statistically using the Wilcoxon paired test. In the visual analysis, the high quality achieved by the SRGAN models, in general, is visible, resulting in more defined edges details and fewer blur effects.

A trusted medical image super-resolution method based on feedback adaptive weighted dense network.

High-resolution (HR) medical images are preferred in clinical diagnoses and subsequent analysis. However, the acquisition of HR medical images is easily affected by hardware devices. As an effective and trusted alternative method, the super-resolution (SR) technology is introduced to improve the image resolution. Compared with traditional SR methods, the deep learning-based SR methods can obtain more clear and trusted HR images. In this paper, we propose a trusted deep convolutional neural network-based SR method named feedback adaptive weighted dense network (FAWDN) for HR medical image reconstruction. Specifically, the proposed FAWDN can transmit the information of the output image to the low-level features by a feedback connection. To explore advanced feature representation and reduce the feature redundancy in dense blocks, an adaptive weighted dense block (AWDB) is introduced to adaptively select the informative features. Experimental results demonstrate that our FAWDN outperforms the state-of-the-art image SR methods and can obtain more clear and trusted medical images than comparative methods.

Deep learning-based super-resolution for small-angle neutron scattering data: attempt to accelerate experimental workflow

The authors propose an alternative route to circumvent the limitation of neutron flux using the recent deep learning super-resolution technique. The feasibility of accelerating data collection has been demonstrated by using small-angle neutron scattering (SANS) data collected from the EQ-SANS instrument at Spallation Neutron Source (SNS). Data collection time can be reduced by increasing the size of binning of the detector pixels at the sacrifice of resolution. High-resolution scattering data is then reconstructed by using a deep learning-based super-resolution method. This will allow users to make critical decisions at a much earlier stage of data collection, which can accelerate the overall experimental workflow.

MRI Super-Resolution With Ensemble Learning and Complementary Priors

Magnetic resonance imaging (MRI) is a widely used medical imaging modality. However, due to the limitations in hardware, scan time, and throughput, it is often clinically challenging to obtain high-quality MR images. The super-resolution approach is potentially promising to improve MR image quality without any hardware upgrade. In this paper, we propose an ensemble learning and deep learning framework for MR image super-resolution. In our study, we first enlarged low resolution images using 5 commonly used super-resolution algorithms and obtained differentially enlarged image datasets with complementary priors. Then, a generative adversarial network (GAN) is trained with each dataset to generate super-resolution MR images. Finally, a convolutional neural network is used for ensemble learning that synergizes the outputs of GANs into the final MR super-resolution images. According to our results, the ensemble learning results outcome any one of GAN outputs. Compared with some state-of-the-art deep learning-based super-resolution methods, our approach is advantageous in suppressing artifacts and keeping more image details.

Progressive Sub-Band Residual-Learning Network for MR Image Super Resolution.

High-resolution (HR) magnetic resonance images (MRI) provide more detailed information for clinical application. However, HR MRI is less available because of the longer scan time and lower signal-to-noise ratio. Spatial resolution is one of the key parameters of MRI. The image post-processing technique super-resolution (SR) is an alternative approach to improve the spatial resolution of MR images. Inspired by advanced deep learning based SR methods, we propose an MRI SR model named progressive sub-band residual learning SR network (PSR-SRN). The proposed model contains two parallel progressive learning streams, where one stream learns on missed high-frequency residuals by sub-band residual learning unit (ISRL) and the other focuses on reconstructing refined MR image. These two streams complement each other and enable to learn complex mappings between "Low-" and "High-" resolution MR images. Besides, we introduce brain-like mechanisms (in-depth supervision and local feedback mechanism) and progressive sub-band learning strategy to emphasize variant textures of MRI. Compared with traditional and deep learning MRI SR methods, our PSR-SRN model shows superior performance.