Super-resolution Deep Learning Research Articles

Recursive Deep Prior Video: A super resolution algorithm for time-lapse microscopy of organ-on-chip experiments.

Biological experiments based on organ-on-chips (OOCs) exploit light Time-Lapse Microscopy (TLM) for a direct observation of cell movement that is an observable signature of underlying biological processes. A high spatial resolution is essential to capture cell dynamics and interactions from recorded experiments by TLM. Unfortunately, due to physical and cost limitations, acquiring high resolution videos is not always possible. To overcome the problem, we present here a new deep learning-based algorithm that extends the well-known Deep Image Prior (DIP) to TLM Video Super Resolution without requiring any training. The proposed Recursive Deep Prior Video method introduces some novelties. The weights of the DIP network architecture are initialized for each of the frames according to a new recursive updating rule combined with an efficient early stopping criterion. Moreover, the DIP loss function is penalized by two different Total Variation-based terms. The method has been validated on synthetic, i.e., artificially generated, as well as real videos from OOC experiments related to tumor-immune interaction. The achieved results are compared with several state-of-the-art trained deep learning Super Resolution algorithms showing outstanding performances.

COMPARISON OF MULTI-IMAGES DEEP LEARNING SUPER RESOLUTION FOR PASSIVE MICROWAVE IMAGES OF ARCTIC SEA ICE

Abstract. The observation of Arctic sea ice is of great significance to monitoring of the polar environment, research on global climate change and application of Arctic navigation. Compared to optical imagery and SAR imagery, passive microwave images can be obtained for all-sky conditions with high time resolution. However, the spatial resolution of passive microwave images is relatively low (6.25 km – 25 km) for the observation of detailed sea ice characteristics and small-scale sea ice geographical phenomena. Therefore, in this paper, considering the suitability of different alignment and fusion strategies to the characteristics of passive microwave images of sea ice, two multi-images deep learning super-resolution (SR) algorithms, Recurrent Back-Projection Network (RBPN) and network of Temporal Group Attention (TGA), are selected to test the effects of SR technique for passive microwave images of sea ice. Both qualitative and quantitative comparisons are provided for the SR results oriented from two algorithms. Overall, the SR performance of TGA algorithm outperforms RBPN algorithm for the passive microwave images of sea ice.

DL-MRI: A Unified Framework of Deep Learning-Based MRI Super Resolution.

Magnetic resonance imaging (MRI) is widely used in the detection and diagnosis of diseases. High-resolution MR images will help doctors to locate lesions and diagnose diseases. However, the acquisition of high-resolution MR images requires high magnetic field intensity and long scanning time, which will bring discomfort to patients and easily introduce motion artifacts, resulting in image quality degradation. Therefore, the resolution of hardware imaging has reached its limit. Based on this situation, a unified framework based on deep learning super resolution is proposed to transfer state-of-the-art deep learning methods of natural images to MRI super resolution. Compared with the traditional image super-resolution method, the deep learning super-resolution method has stronger feature extraction and characterization ability, can learn prior knowledge from a large number of sample data, and has a more stable and excellent image reconstruction effect. We propose a unified framework of deep learning -based MRI super resolution, which has five current deep learning methods with the best super-resolution effect. In addition, a high-low resolution MR image dataset with the scales of ×2, ×3, and ×4 was constructed, covering 4 parts of the skull, knee, breast, and head and neck. Experimental results show that the proposed unified framework of deep learning super resolution has a better reconstruction effect on the data than traditional methods and provides a standard dataset and experimental benchmark for the application of deep learning super resolution in MR images.

Deep convolutional neural network for P-band spaceborne synthetic aperture radar imaging through the ionosphere

The dispersion characteristics of the background ionosphere and the random fluctuations of the ionospheric irregularities are an important source of phase error that seriously damages the quality of radar images. To mitigate the ionospheric distortions of P-band spaceborne synthetic aperture radar (SAR) images, a super-resolution deep learning method is proposed in this paper. Different from the traditional imaging method based on the prior knowledge of imaging, the method proposed in this paper directly trains the ionospheric implicit imaging model of spaceborne SAR without complicated iterative processes. First, the phase errors, which are caused by the dispersion and the scintillation in the range and azimuth directions, respectively, are analyzed. Second, an improved u-net structure that embeds the residual network between the encoder and decoder is briefly proposed. Finally, the range doppler algorithm is used to preprocess radar image as the input of the convolutional neural network (CNN) and is compare with the predicted output of the CNN. Experimental results prove the effectiveness of the proposed method in radar image focusing.

Achieving Higher Resolution Lake Area from Remote Sensing Images Through an Unsupervised Deep Learning Super-Resolution Method

Lakes have been identified as an important indicator of climate change and a finer lake area can better reflect the changes. In this paper, we propose an effective unsupervised deep gradient network (UDGN) to generate a higher resolution lake area from remote sensing images. By exploiting the power of deep learning, UDGN models the internal recurrence of information inside the single image and its corresponding gradient map to generate images with higher spatial resolution. The gradient map is derived from the input image to provide important geographical information. Since the training samples are only extracted from the input image, UDGN can adapt to different settings per image. Based on the superior adaptability of the UDGN model, two strategies are proposed for super-resolution (SR) mapping of lakes from multispectral remote sensing images. Finally, Landsat 8 and MODIS (moderate-resolution imaging spectroradiometer) images from two study areas on the Tibetan Plateau in China were used to evaluate the performance of UDGN. Compared with four unsupervised SR methods, UDGN obtained the best SR results as well as lake extraction results in terms of both quantitative and visual aspects. The experiments prove that our approach provides a promising way to break through the limitations of median-low resolution remote sensing images in lake change monitoring, and ultimately support finer lake applications.

Widefield light sheet microscopy using an Airy beam combined with deep-learning super-resolution

Imaging across length scales and in depth has been an important pursuit of widefield optical imaging. This promises to reveal fine cellular detail within a widefield snapshot of a tissue sample. Current advances often sacrifice resolution through selective sub-sampling to provide a wide field of view in a reasonable time scale. We demonstrate a new avenue for recovering high-resolution images from sub-sampled data in light sheet microscopy using deep-learning super-resolution. We combine this with the use of a widefield Airy beam to achieve high-resolution imaging over extended fields of view and depths. We characterise our method on fluorescent beads as test targets. We then demonstrate improvements in imaging amyloid plaques in a cleared brain from a mouse model of Alzheimer’s disease, and in excised healthy and cancerous colon and breast tissues. This development can be widely applied in all forms of light sheet microscopy to provide a two-fold increase in the dynamic range of the imaged length scale. It has the potential to provide further insight into neuroscience, developmental biology, and histopathology.

1. Deep Learning Super-resolution in Medical Imaging: What Is It and How to Use It

1. Deep Learning Super-resolution in Medical Imaging: What Is It and How to Use It

Super-Resolution Model for High-Precision In Vivo Proton Range Verification Using a Stereo Gamma Camera: A Feasibility Study

The feasibility of a deep-learning-based super-resolution (SR) model to improve the fiducial marker-tracking accuracy of a stereo portable gamma camera (SPGC) system over the range of an in vivo proton beam was verified in a Monte-Carlo (MC) simulation using the Geometry and Tracking 4 (Geant4) package. The SPGC system is capable of measuring the three-dimensional (3D) position of excited gold markers by detecting proton-induced X-ray emissions (PIXEs) generated by the interactions between the gold marker and a proton beam. The SPGC system was modeled using Geant4 according to manufacturer’s specifications. The original image (Io) acquired by using the SPGC system, which was comprised of 32 × 32 arrays over an area of 104 × 104 mm2, was subjected to resolution enhancement to produce an SR-enhanced image (ISR) (128 × 128 arrays) through a fully trained SR model based on a convolutional neural network (CNN). In virtual experiments, two portable gamma cameras were positioned perpendicular to each other. Next, a pair of Io’s were acquired by detecting the radiations from the exited gold marker positioned in a water phantom. Then, the fully trained SR model improved the quality of the Io’s by converting those to ISR’s. The 3D position of the radiation source was calculated by using Anger logic and 3D vector calculations. Virtual experiments for in vivo proton range verification using the SPGC system were performed by irradiating to a gold marker in a water phantom with a proton beam. A gold marker was placed at five different positions along the Bragg curve of a 100.0-MeV proton beam, which had a range of 74.5 mm in water. The proton beam was irradiated to deliver 20.0 Gy to the gold marker when it was positioned at the center of the Bragg peak; then, the PIXEs were measured by using the SPGC system. When a gold marker was at a different position, it was irradiated with the same dose for a quantitative comparison. Then 3D position of the gold marker was calculated for the original image (Io) and for the high-resolution image (ISR) to compare the detection accuracy. The averaged root-mean-square errors of the five positions between the reference and calculation for Io and ISR were 9.127 mm and 3.991 mm, respectively. In conclusion, the feasibility of using a deep-learning SR model for improving the image resolution of Io and therefore, the tracking accuracy of the SPGC system was validated in MC simulations. The SR model can be applicable to diverse areas of research using gamma camera.

Spatial aliasing removal using deep learning super-resolution

Spatial aliasing removal using deep learning super-resolution

Mastcam Image Resolution Enhancement with Application to Disparity Map Generation for Stereo Images with Different Resolutions

In this paper, we introduce an in-depth application of high-resolution disparity map estimation using stereo images from Mars Curiosity rover’s Mastcams, which have two imagers with different resolutions. The left Mastcam has three times lower resolution as that of the right. The left Mastcam image’s resolution is first enhanced with three methods: Bicubic interpolation, pansharpening-based method, and a deep learning super resolution method. The enhanced left camera image and the right camera image are then used to estimate the disparity map. The impact of the left camera image enhancement is examined. The comparative performance analyses showed that the left camera enhancement results in getting more accurate disparity maps in comparison to using the original left Mastcam images for disparity map estimation. The deep learning-based method provided the best performance among the three for both image enhancement and disparity map estimation accuracy. A high-resolution disparity map, which is the result of the left camera image enhancement, is anticipated to improve the conducted science products in the Mastcam imagery such as 3D scene reconstructions, depth maps, and anaglyph images.

An enhanced fault-detection method based on adaptive spectral decomposition and super-resolution deep learning

Coherence is widely used for detecting faults in reservoir characterization. However, faults detected from coherence may be contaminated by some other discontinuities (e.g., noise and stratigraphic features) that are unrelated to faults. To further improve the accuracy and efficiency of coherence, preprocessing or postprocessing techniques are required. We developed an enhanced fault-detection method with adaptive scale highlighting and high resolution, by combining adaptive spectral decomposition and super-resolution (SR) deep learning into coherence calculation. As a preprocessing technique, adaptive spectral decomposition is first proposed and applied on seismic data to get a dominant-frequency-optimized amplitude spectrum, which has features of scale focus and multiple resolution. Eigenstructure-based coherence with dip correction is then calculated to delineate fault discontinuities. Following the remarkable success of SR deep learning in image reconstruction, a convolutional neural network (CNN) model is built and it then takes fault-detection images as the input to achieve enhanced results. The effectiveness of our proposed method is validated on a seismic survey acquired from Eastern China. Examples demonstrate that coherence from adaptive amplitude spectrum without dip correction is comparable to the dip-corrected one from seismic amplitude data at a certain degree, and they even highlight the specific scale of fault targets. Comparing fault detections from adaptive spectrum and some specific-frequency components, it can be concluded that adaptive spectral-based coherence highlights the primary scale of faults at various depths with only one single volume of data, thus improving the interpretation efficiency and reducing storage cost. Furthermore, with the trained CNN model, the resolution and signal-to-noise ratio of coherence images are effectively improved and the continuity of detected fault is promisingly enhanced.

High-throughput, high-resolution deep learning microscopy based on registration-free generative adversarial network.

We combine a generative adversarial network (GAN) with light microscopy to achieve deep learning super-resolution under a large field of view (FOV). By appropriately adopting prior microscopy data in an adversarial training, the neural network can recover a high-resolution, accurate image of new specimen from its single low-resolution measurement. Its capacity has been broadly demonstrated via imaging various types of samples, such as USAF resolution target, human pathological slides, fluorescence-labelled fibroblast cells, and deep tissues in transgenic mouse brain, by both wide-field and light-sheet microscopes. The gigapixel, multi-color reconstruction of these samples verifies a successful GAN-based single image super-resolution procedure. We also propose an image degrading model to generate low resolution images for training, making our approach free from the complex image registration during training data set preparation. After a well-trained network has been created, this deep learning-based imaging approach is capable of recovering a large FOV (~95 mm2) enhanced resolution of ~1.7 μm at high speed (within 1 second), while not necessarily introducing any changes to the setup of existing microscopes.

Super-resolution for biometrics: A comprehensive survey

The lack of resolution of imaging systems has critically adverse impacts on the recognition and performance of biometric systems, especially in the case of long range biometrics and surveillance such as face recognition at a distance, iris recognition and gait recognition. Super-resolution, as one of the core innovations in computer vision, has been an attractive but challenging solution to address this problem in both general imaging systems and biometric systems. However, a fundamental difference exists between conventional super-resolution motivations and those required for biometrics. The former aims to enhance the visual clarity of the scene while the latter, more significantly, aims to improve the recognition accuracy of classifiers by exploiting specific characteristics of the observed biometric traits. This paper comprehensively surveys the state-of-the-art super-resolution approaches proposed for four major biometric modalities: face (2D+3D), iris, fingerprint and gait. We approach the super-resolution problem in biometrics from several different perspectives, including from the spatial and frequency domains, single and multiple input images, learning-based and reconstruction-based approaches. Especially, we highlight two special categories: feature-domain super-resolution which performs super-resolution directly on the feature space to purposely improve the recognition performance, and deep-learning super-resolution which discusses the most recent advances in deep learning for the super-resolution task. Finally, we discuss the current and open research challenges and provide recommendations into the future for the improved use of super-resolution with biometrics.