Image Super-resolution Method Research Articles

Deep learning-based image super-resolution restoration for mobile infrared imaging system

Mobile infrared imaging system (MIIS) has important applications in the areas of the military affairs, medical diagnosis, and industrial detection. However, MIIS usually suffers the problems of the low resolution, low contrast, and noise interference. To solve these problems, we propose a deep learning-based image super-resolution restoration method for the MIIS. We design a multiscale feature distillation residual network to retain image features at various stages during the training process. The network uses dilated convolution to expand the receptive field and outputs the super-resolution (SR) infrared image with a sub-pixel method. Many infrared images that are captured by a MIIS are used for experiments. Experimental results indicate that the proposed method can realize the infrared image SR restoration and performs outstandingly against the four recently published SR methods both in visual quality and indicator performance. The proposed method is helpful for the practical applications of the MIIS.

Infrared image super-resolution method based on dual-branch deep neural network

Infrared image super-resolution method based on dual-branch deep neural network

Rethinking 3D-CNN in Hyperspectral Image Super-Resolution

Recently, CNN-based methods for hyperspectral image super-resolution (HSISR) have achieved outstanding performance. Due to the multi-band property of hyperspectral images, 3D convolutions are natural candidates for extracting spatial–spectral correlations. However, pure 3D CNN models are rare to see, since they are generally considered to be too complex, require large amounts of data to train, and run the risk of overfitting on relatively small-scale hyperspectral datasets. In this paper, we question this common notion and propose Full 3D U-Net (F3DUN), a full 3D CNN model combined with the U-Net architecture. By introducing skip connections, the model becomes deeper and utilizes multi-scale features. Extensive experiments show that F3DUN can achieve state-of-the-art performance on HSISR tasks, indicating the effectiveness of the full 3D CNN on HSISR tasks, thanks to the carefully designed architecture. To further explore the properties of the full 3D CNN model, we develop a 3D/2D mixed model, a popular kind of model prior, called Mixed U-Net (MUN) which shares a similar architecture with F3DUN. Through analysis on F3DUN and MUN, we find that 3D convolutions give the model a larger capacity; that is, the full 3D CNN model can obtain better results than the 3D/2D mixed model with the same number of parameters when it is sufficiently trained. Moreover, experimental results show that the full 3D CNN model could achieve competitive results with the 3D/2D mixed model on a small-scale dataset, suggesting that 3D CNN is less sensitive to data scaling than what people used to believe. Extensive experiments on two benchmark datasets, CAVE and Harvard, demonstrate that our proposed F3DUN exceeds state-of-the-art HSISR methods both quantitatively and qualitatively.

Benchmark for Building Segmentation on Up-Scaled Sentinel-2 Imagery

Currently, we can solve a wide range of tasks using computer vision algorithms, which reduce manual labor and enable rapid analysis of the environment. The remote sensing domain provides vast amounts of satellite data, but it also poses challenges associated with processing this data. Baseline solutions with intermediate results are available for various tasks, such as forest species classification, infrastructure recognition, and emergency situation analysis using satellite data. Despite these advances, two major issues with high-performing artificial intelligence algorithms remain in the current decade. The first issue relates to the availability of data. To train a robust algorithm, a reasonable amount of well-annotated training data is required. The second issue is the availability of satellite data, which is another concern. Even though there are a number of data providers, high-resolution and up-to-date imagery is extremely expensive. This paper aims to address these challenges by proposing an effective pipeline for building segmentation that utilizes freely available Sentinel-2 data with 10 m spatial resolution. The approach we use combines a super-resolution (SR) component with a semantic segmentation component. As a result, we simultaneously consider and analyze SR and building segmentation tasks to improve the quality of the infrastructure analysis through medium-resolution satellite data. Additionally, we collected and made available a unique dataset for the Russian Federation covering area of 1091.2 square kilometers. The dataset provides Sentinel-2 imagery adjusted to the spatial resolution of 2.5 m and is accompanied by semantic segmentation masks. The building footprints were created using OpenStreetMap data that was manually checked and verified. Several experiments were conducted for the SR task, using advanced image SR methods such as the diffusion-based SR3 model, RCAN, SRGAN, and MCGR. The MCGR network produced the best result, with a PSNR of 27.54 and SSIM of 0.79. The obtained SR images were then used to tackle the building segmentation task with different neural network models, including DeepLabV3 with different encoders, SWIN, and Twins transformers. The SWIN transformer achieved the best results, with an F1-score of 79.60.

Direct localization and delineation of human pedunculopontine nucleus based on a self-supervised magnetic resonance image super-resolution method.

The pedunculopontine nucleus (PPN) is a small brainstem structure and has attracted attention as a potentially effective deep brain stimulation (DBS) target for the treatment of Parkinson's disease (PD). However, the in vivo location of PPN remains poorly described and barely visible on conventional structural magnetic resonance (MR) images due to a lack of high spatial resolution and tissue contrast. This study aims to delineate the PPN on a high-resolution (HR) atlas and investigate the visibility of the PPN in individual quantitative susceptibility mapping (QSM) images. We combine a recently constructed Montreal Neurological Institute (MNI) space unbiased QSM atlas (MuSus-100), with an implicit representation-based self-supervised image super-resolution (SR) technique to achieve an atlas with improved spatial resolution. Then guided by a myelin staining histology human brain atlas, we localize and delineate PPN on the atlas with improved resolution. Furthermore, we examine the feasibility of directly identifying the approximate PPN location on the 3.0-T individual QSM MR images. The proposed SR network produces atlas images with four times the higher spatial resolution (from 1 to 0.25 mm isotropic) without a training dataset. The SR process also reduces artifacts and keeps superb image contrast for further delineating small deep brain nuclei, such as PPN. Using the myelin staining histological atlas as guidance, we first identify and annotate the location of PPN on the T1-weighted (T1w)-QSM hybrid MR atlas with improved resolution in the MNI space. Then, we relocate and validate that the optimal targeting site for PPN-DBS is at the middle-to-caudal part of PPN on our atlas. Furthermore, we confirm that the PPN region can be identified in a set of individual QSM images of 10 patients with PD and 10 healthy young adults. The contrast ratios of the PPN to its adjacent structure, namely the medial lemniscus, on images of different modalities indicate that QSM substantially improves the visibility of the PPN both in the atlas and individual images. Our findings indicate that the proposed SR network is an efficient tool for small-size brain nucleus identification. HR QSM is promising for improving the visibility of the PPN. The PPN can be directly identified on the individual QSM images acquired at the 3.0-T MR scanners, facilitating a direct targeting of PPN for DBS surgery.

A single image super resolution method based on cross residual network and wavelet transform

In this paper, we propose a method to realize single image super resolution using a network composed of residual blocks with a cross architecture and wavelet transform. For the high quality of the reconstructed high resolution image, the brightness change must be sharp at the edge and gentle at the flat region. In single image super resolution, it is important to accurately predict the high-frequency components of the image. Hence, we propose a method to realize single image super resolution by estimating the high-frequency component in the wavelet domain using neural network composed of residual blocks with two crossing skip connections.

Local super-resolution imaging of foveated areas in super-oscillating optical fields.

Herein, we propose a super-oscillation optical field foveated local super-resolution imaging method. Firstly, the post-diffraction integral equation of the foveated modulation device is constructed, the objective function and constraints are established, and the structural parameters of the amplitude modulation device are optimally solved by using genetic algorithm. Secondly, the solved data have been input into the software for point diffusion function analysis. We have studied the super-resolution performance of different ring band amplitude types, and find the 8-ring 0-1 amplitude type has the best super-resolution performance. Finally, the principle experimental device is built according to the simulation parameters, and the super-oscillatory device parameters is loaded onto the amplitude type spatial light modulator for the principle experiments, in which the super-oscillation foveated local super-resolution imaging system is able to perform high image contrast imaging in the whole field of view and super-resolution imaging in the foveated field of view area. As a result, this method achieves the 1.25 times super-resolution magnification in the foveated field of view area, which realizes the super-resolutio n imaging of local field while keeping the resolution of other fields unchanged. Experiments verify the feasibility and effectiveness of our system.

Terrain Self-Similarity-Based Transformer for Generating Super Resolution DEMs

High-resolution digital elevation models (DEMs) are important for relevant geoscience research and practical applications. Compared with traditional hardware-based methods, super-resolution (SR) reconstruction techniques are currently low-cost and feasible methods used for obtaining high-resolution DEMs. Single-image super-resolution (SISR) techniques have become popular in DEM SR in recent years. However, DEM super-resolution has not yet utilized reference-based image super-resolution (RefSR) techniques. In this paper, we propose a terrain self-similarity-based transformer (SSTrans) to generate super-resolution DEMs. It is a reference-based image super-resolution method that automatically acquires reference images using terrain self-similarity. To verify the proposed model, we conducted experiments on four distinct types of terrain and compared them to the results from the bicubic, SRGAN, and SRCNN approaches. The experimental results show that the SSTrans method performs well in all four terrains and has outstanding advantages in complex and uneven surface terrains.

Image Super-Resolution Via a Convolutional Neural Network

We proposed a deep learning method for single image super-resolution (SR). Our method directly learns an end- to- end mapping between the low/high-resolution images. The mapping is represented as a deep convolutional neural network (CNN) that takes the low-resolution image as the input and outputs the high-resolution one. We further show that traditional sparse-coding-based SR methods can also be viewed as a deep convolutional network. But unlike traditional methods that handle each component separately, our method jointly optimizes all layers. Our deep CNN has a lightweight structure, yet demonstrates state-of-the-art restoration quality, and achieves fast speed for practical on-line usage. We explore different network structures and parameter settings to achieve tradeoffs between performance and speed. Moreover, we extend our network to cope with three color channels simultaneously, and show better overall reconstruction quality. keywords:Super-resolution, Deep Convolutional Neural Network, Sparse codin.

Visualization of the three-dimensional structure of the human centromere in mitotic chromosomes by superresolution microscopy.

The human centromere comprises large arrays of repetitive α-satellite DNA at the primary constriction of mitotic chromosomes. In addition, centromeres are epigenetically specified by the centromere-specific histone H3 variant CENP-A that supports kinetochore assembly to enable chromosome segregation. Because CENP-A is bound to only a fraction of the α-satellite elements within the megabase-sized centromere DNA, correlating the three-dimensional (3D) organization of α-satellite DNA and CENP-A remains elusive. To visualize centromere organization within a single chromatid, we used a combination of the centromere chromosome orientation fluorescence in situ hybridization (Cen-CO-FISH) technique together with structured illumination microscopy. Cen-CO-FISH allows the differential labeling of the sister chromatids without the denaturation step used in conventional FISH that may affect DNA structure. Our data indicate that α-satellite DNA is arranged in a ring-like organization within prometaphase chromosomes, in the presence or absence of spindle's microtubules. Using expansion microscopy, we found that CENP-A organization within mitotic chromosomes follows a rounded pattern similar to that of α-satellite DNA, often visible as a ring thicker at the outer surface oriented toward the kinetochore-microtubule interface. Collectively, our data provide a 3D reconstruction of α-satellite DNA along with CENP-A clusters that outlines the overall architecture of the mitotic centromere.

Feature similarity rank-based information distillation network for lightweight image superresolution

To explore a fast and accurate image superresolution (SR) algorithm for real-world scenarios, information distillation is used to compress the number of feature channels, which has become one of the most commonly used tools in the field of lightweight image SR. However, existing information distillation-based image SR methods simply distill the first (distilled rate) channels in which many channels with unique features, i.e., low-redundancy features, are distilled as well. Hence, these methods lead to suboptimal SR performance since low-redundancy features are indispensable for image SR reconstruction. To alleviate this issue, this paper proposes a feature similarity ranking algorithm, which sorts the extracted feature channels according to the degree of channel redundancy. Then, it utilizes the ranked information distillation block (RIDB) for distilling high-redundancy features. In addition, considering that feature channels have already been sorted in the channel dimension, it involves a gated nonlocal attention module (GNLM) to capture the long-distance feature correlation in the spatial dimension. GNLM greatly reduces the computational consumption through the partitioned nonlocal module as well as the gate unit for evaluating the importance of each module, which is more suitable for lightweight image SR. Extensive experiments demonstrate that the proposed method performs favorably in terms of SR performance compared with not only information distillation-based image SR methods but also other state-of-the-art lightweight SR methods.

Image Super-Resolution Reconstruction Based on Residual Convolution and Double Attention Mechanism

Currently, single-image super-resolution reconstruction based on deep learning has achieved good results. To address the problems that most networks will have long training time, weak image learning ability and not fully utilizing the high frequency information of images, an image super-resolution reconstruction method based on residual convolution and double attention mechanism is proposed. The model performs deep feature extraction of the image by cascading deep convolutional networks, introduces local residual blocks to solve the model degradation brought by too many network parameters, and embeds the dual attention mechanism module in the residual blocks for adaptive calibration to adjust the feature map weights of each channel and the spatial correlation between features, so as to obtain deep texture detail information and reconstruct the feature image by sub-pixel convolutional layers to up sampling to reconstruct the high-resolution image. In the test sets of Set5 and Set14, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used as evaluation indexes, while comparing SRCNN, FSRCNN and VDSR methods all reconstructed images with better results. The experimental results show that the method can effectively improve the utilization of high-frequency feature information and can increase the reconstruction capability of the images to a certain extent.

Towards cloud-enabled fully automated end-to-end single molecule imaging.

Single Molecule Super-resolution imaging has revolutionised understanding biological processes on the sub-100 nm scale. Our group developed Fluorophore Localisation Imaging with Photobleaching (FLImP), a super-resolution imaging method that can achieve <4 nm resolution. In concert with molecular dynamics simulations, FLImP was used to reveal the in-situ atomic resolution structures of ligand-bound EGFR oligomers at the plasma membrane of cells (Needham et al., Nat Comms 2016), and those of ligand-unbound dimers and oligomers on cells (Zanetti-Domingues et al., Nat Comms 2018), shedding new light on the mechanisms of EGFR auto-inhibition, phosphorylation and signalling.

Imaging Simulation Method for Novel Rotating Synthetic Aperture System Based on Conditional Convolutional Neural Network

The novel rotating synthetic aperture (RSA) is a new optical imaging system that uses the method of rotating the rectangular primary mirror for dynamic imaging. It has the advantage of being lightweight, with no need for splicing and real-time surface shape maintenance on orbit. The novel imaging method leads to complex image quality degradation characteristics. Therefore, it is vital to use the image quality improvement method to restore and improve the image quality to meet the application requirements. For the RSA system, a new system that has not been applied in orbit, it is difficult to construct suitable large datasets. Therefore, it is necessary to study and establish the dynamic imaging characteristic model of the RSA system, and on this basis provide data support for the corresponding image super resolution and restoration method through simulation. In this paper, we first analyze the imaging characteristics and mathematically model the rectangular rotary pupil of the RSA system. On this basis, combined with the analysis of the physical interpretation of the blur kernel, we find that the optimal blur kernel is not the point spread function (PSF) of the imaging system. Therefore, the simulation method of convolving the input image directly with the PSF is flawed. Furthermore, the weights of a convolutional neural network (CNN) are the same for each input. This means that the normal convolutional layer is not only difficult to accurately estimate the time-varying blur kernel, but also difficult to adapt to the change in the length–width ratio of the primary mirror. To that end, we propose a blur kernel estimation conditional convolutional neural network (CCNN) that is equivalent to multiple normal CNNs. We extend the CNN to a conditional model by taking an encoding as an additional input and using conditionally parameterized convolutions instead of normal convolutions. The CCNN can simulate the imaging characteristics of the rectangular pupil with different length–width ratios and different rotation angles in a controllable manner. The results of semi-physical experiments show that the proposed simulation method achieves a satisfactory performance, which can provide data and theoretical support for the image restoration and super-resolution method of the RSA system.

SIT-SR 3D: Self-supervised slice interpolation via transfer learning for 3D volume super-resolution

We present SIT-SR 3D, a novel self-supervised method for 3D single image super-resolution (SISR). Scaling 2D SISR networks to 3D SISR requires code redesign, high computing resources, and 3D ground-truth. However, we circumvent this by (1) using a pre-trained 2D SISR for indirect supervision and (2) using a novel consistency loss to learn frame interpolation. Any pre-trained state of the art 2D SISR method can replace the 2D SISR used in SIT-SR 3D, thus transferring the merits of 2D to 3D and ensuring modularity. We trained two end-to-end 3D baselines in a supervised setting; a 3D RRDBNet trained only with L1 loss and a 3D ESRGAN trained with adversarial and perceptual loss. We show that the proposed pipeline's self-supervised version is qualitatively better than the baselines. When trained in a supervised setting, SIT-SR 3D achieves better PSNR than its counterparts. Furthermore, our pipeline uses fewer parameters compared to the baselines. We demonstrate our results on an open-source digital rock CT dataset. Our code and pre-trained models will be made publicly available.

Dual-channel structured illumination super-resolution quantitative fluorescence resonance energy transfer imaging

The Structured illumination (SI)-based super resolution fluorescence resonance energy transfer (SR-FRET) imaging technique, known as SISR-FRET, enables the investigation of molecular structures and functions in cellular organelles by resolving sub-diffraction FRET signals within living cells. The FRET microscopy offers unique advantages for quantitatively detecting dynamic interactions and spatial distribution of biomolecules within living cells. The spatial resolution of conventional FRET microscopy is limited by the diffraction limit, and it can only capture the average behavior of these events within the resolution limits of conventional fluorescence microscopy. The SISR-FRET performs sequential linear reconstruction of the three-channel SIM images followed by FRET quantitative analysis by using a common localization mask-based filtering approach. This two-step process ensures the fidelity of the reconstructed SR-FRET signals while effectively removing false-positive FRET signals caused by SIM artifacts. However, the slow imaging speed resulting from the switching of excitation-emission channels in SISR-FRET imaging limits its application in fast imaging scenarios. To address this issue, this study proposes a dual-channel structured illumination super-resolution quantitative FRET imaging system and method. By incorporating an FRET dual-channel imaging and registration module into the imaging pathway, the spatial switching and channel multiplexing of the SISR-FRET excitation-emission channels are achieved. Combining the image reconstruction algorithm with channel sub-pixel registration correction, the dual-channel SISR-FRET technique enhances the temporal resolution by 3.5 times while preserving the quantitative super-resolution FRET analysis. Experimental results are obtained by using a multi-color SIM system to perform super-resolution imaging of living cells expressing mitochondria outer membrane FRET standard plasmids. These experiments validate the improved spatial and temporal resolution of dual-channel SISR-FRET and the fidelity of FRET quantification analysis. In summary, this research presents a novel dual-channel structured illumination super-resolution FRET imaging system and method. It overcomes the limitations of slow imaging speed in SISR-FRET by realizing the spatial switching and channel multiplexing of excitation-emission channels. The proposed technique enhances the temporal resolution while maintaining quantitative analysis of super-resolution FRET. Experimental validation demonstrates the increased spatial and temporal resolution of dual-channel SISR-FRET and the accuracy of FRET quantification analysis. This advancement contributes to the study of molecular structures and functions in cellular organelles, providing valuable insights into the intricate mechanisms of living cells.

Spatial resolution enhancement in photon-starved STED imaging using deep learning-based fluorescence lifetime analysis.

As a super-resolution imaging method, stimulated emission depletion (STED) microscopy has unraveled fine intracellular structures and provided insights into nanoscale organizations in cells. Although image resolution can be further enhanced by continuously increasing the STED-beam power, the resulting photodamage and phototoxicity are major issues for real-world applications of STED microscopy. Here we demonstrate that, with 50% less STED-beam power, the STED image resolution can be improved up to 1.45-fold using the separation of photons by a lifetime tuning (SPLIT) scheme combined with a deep learning-based phasor analysis algorithm termed flimGANE (fluorescence lifetime imaging based on a generative adversarial network). This work offers a new approach for STED imaging in situations where only a limited photon budget is available.

RZSR: Reference-Based Zero-Shot Super-Resolution With Depth Guided Self-Exemplars

Recent methods for single image super-resolution (SISR) have demonstrated outstanding performance in generating high-resolution (HR) images from low-resolution (LR) images. However, most of these methods show their superiority using synthetically generated LR images, and their generalizability to real-world images is often not satisfactory. In this paper, we pay attention to two well-known strategies developed for robust super-resolution (SR), i.e., reference-based SR (RefSR) and zero-shot SR (ZSSR), and propose an integrated solution, called reference-based zero-shot SR (RZSR). Following the principle of ZSSR, we train an image-specific SR network at test time using training samples extracted only from the input image itself. To advance ZSSR, we obtain reference image patches with rich textures and high-frequency details which are also extracted only from the input image using cross-scale matching. To this end, we construct an internal reference dataset and retrieve reference image patches from the dataset using depth information. Using LR patches and their corresponding HR reference patches, we train a RefSR network that is embodied with a non-local attention module. Experimental results demonstrate the superiority of the proposed RZSR compared to the previous ZSSR methods and robustness to unseen images compared to other fully supervised SISR methods.

High-Fidelity Reconstruction of Structured Illumination Microscopy by an Amplitude-Phase Channel Attention Network With Multitemporal Information

Structured illumination microscopy (SIM) has been a popular method for live-cell super-resolution (SR) imaging due to its excellent photon efficiency. However, SIM is always marred by artifacts in the reconstructed SR images. To address this problem, we develop an end-to-end amplitude-phase channel attention network (APCAN) for SIM reconstruction based on a-priori frequency and temporal knowledge. The APCAN reinforces both amplitude and phase information of the raw images to guide network reconstruction and attains SR images with fewer artifacts. Moreover, inspired by the continuity knowledge in the traditional method, we design a temporal processing module in APCAN to utilize multiple time points data. Trained with the video dataset imaged from our setup, APCAN can reconstruct an artifact-minimized SR image, achieving a reduction of 38% in reconstruction errors. Finally, we demonstrate that the APCAN reconstructed images have great resistance to noise and photobleaching, achieving the best fidelity among all methods tested.

Learning Detail-Structure Alternative Optimization for Blind Super-Resolution

Existing convolutional neural networks (CNN) based image super-resolution (SR) methods have achieved impressive performance on bicubic kernel, which is not valid to handle unknown degradations in real-world applications. Recent blind SR methods suggest to reconstruct SR images relying on blur kernel estimation. However, their results still remain visible artifacts and detail distortion due to the estimation errors. To alleviate these problems, in this paper, we propose an effective and kernel-free network, namely DSSR, which enables recurrent detail-structure alternative optimization without blur kernel prior incorporation for blind SR. Specifically, in our DSSR, a detail-structure modulation module (DSMM) is built to exploit the interaction and collaboration of image details and structures. The DSMM consists of two components: a detail restoration unit (DRU) and a structure modulation unit (SMU). The former aims at regressing the intermediate HR detail reconstruction from LR structural contexts, and the latter performs structural contexts modulation conditioned on the learned detail maps at both HR and LR spaces. Besides, we use the output of DSMM as the hidden state and design our DSSR architecture from a recurrent convolutional neural network (RCNN) view. In this way, the network can alternatively optimize the image details and structural contexts, achieving co-optimization across time. Moreover, equipped with the recurrent connection, our DSSR allows low- and high-level feature representations complementary by observing previous HR details and contexts at every unrolling time. Extensive experiments on synthetic datasets and real-world images demonstrate that our method achieves the state-of-the-art against existing methods. The source code can be found at https://github.com/Arcananana/DSSR.