High-quality Super-resolution Images Research Articles

Deep learning-based spectroscopic single-molecule localization microscopy.

Spectroscopic single-molecule localization microscopy (sSMLM) takes advantage of nanoscopy and spectroscopy, enabling sub-10nm resolution as well as simultaneous multicolor imaging of multi-labeled samples. Reconstruction of raw sSMLM data using deep learning is a promising approach for visualizing the subcellular structures at the nanoscale. Develop a novel computational approach leveraging deep learning to reconstruct both label-free and fluorescence-labeled sSMLM imaging data. We developed a two-network-model based deep learning algorithm, termed DsSMLM, to reconstruct sSMLM data. The effectiveness of DsSMLM was assessed by conducting imaging experiments on diverse samples, including label-free single-stranded DNA (ssDNA) fiber, fluorescence-labeled histone markers on COS-7 and U2OS cells, and simultaneous multicolor imaging of synthetic DNA origami nanoruler. For label-free imaging, a spatial resolution of 6.22nm was achieved on ssDNA fiber; for fluorescence-labeled imaging, DsSMLM revealed the distribution of chromatin-rich and chromatin-poor regions defined by histone markers on the cell nucleus and also offered simultaneous multicolor imaging of nanoruler samples, distinguishing two dyes labeled in three emitting points with a separation distance of 40nm. With DsSMLM, we observed enhanced spectral profiles with 8.8% higher localization detection for single-color imaging and up to 5.05% higher localization detection for simultaneous two-color imaging. We demonstrate the feasibility of deep learning-based reconstruction for sSMLM imaging applicable to label-free and fluorescence-labeled sSMLM imaging data. We anticipate our technique will be a valuable tool for high-quality super-resolution imaging for a deeper understanding of DNA molecules' photophysics and will facilitate the investigation of multiple nanoscopic cellular structures and their interactions.

SwinIBSR: Towards real-world infrared image super-resolution

Traditional infrared image super-resolution (SR) methods often fall short in real-world scenarios, particularly when dealing with images degraded by various forms of noise and artifacts. Addressing this gap, this paper presents SwinIBSR, a blind SR approach optimized for real-world infrared imaging conditions. SwinIBSR introduces a practical degradation model, specifically designed to simulate the complex degradation processes encountered in real-world infrared imaging. Additionally, a mixed training strategy using both infrared and visible light datasets greatly enhances the network’s generalization capabilities, leading to the production of high-quality SR images. Building upon the Transformer-based architecture of SwinIR, SwinIBSR is adept at learning intricate mappings from low-resolution to high-resolution images. Our comprehensive experiments validate the effectiveness of SwinIBSR, showcasing its outstanding performance in processing real-world infrared images and its notable superiority over existing methods regarding visual quality. This work contributes to the field of infrared image SR, offering insights in enhancing SR techniques for practical applications.

Image Super-Resolution With Content-Aware Feature Processing

Image super-resolution (SR) is currently a very active research topic with applications spanning from computer vision to videos and graphic industries. The top performers in SR field usually employ deep or wide convolutional neural networks (CNNs) to restore the lost textures from low-resolution images. However, most of these methods adopt pixel-shuffle, and deconvolution as their up-sampling techniques and often generate conspicuous artifacts in the reconstructed image. Additionally, the ongoing trend of directly portraying the degraded low-resolution image to high-resolution via complex deep CNNs improves the reconstruction performance, but at the cost of high computational complexity. In this work, we propose a multi-level bi-cubic up-sampler network (MBUp-Net) for reconstructing high-quality SR image with restricted number of parameters. A novel Content-Aware Feature Difference (CAFD) block is presented to reform the network by focusing on contextual information. The proposed CAFD block consists of four multi-level attention blocks for better extraction of low-level features at different scales. Furthermore, we design an innovative up-sampling layer that consistently outperforms the traditional up-sampling methods. These components collaboratively endow our proposed network with a great performance boost, helping it achieve state-of-the-art accuracy on five synthetic benchmark datasets, both qualitatively and quantitatively. In addition, a detailed ablation study has been accomplished to scrutinize the improvements obtained by different modules in the proposed method.

Super-resolution biomedical imaging via reference-free statistical implicit neural representation

Objective. Supervised deep learning for image super-resolution (SR) has limitations in biomedical imaging due to the lack of large amounts of low- and high-resolution image pairs for model training. In this work, we propose a reference-free statistical implicit neural representation (INR) framework, which needs only a single or a few observed low-resolution (LR) image(s), to generate high-quality SR images. Approach. The framework models the statistics of the observed LR images via maximum likelihood estimation and trains the INR network to represent the latent high-resolution (HR) image as a continuous function in the spatial domain. The INR network is constructed as a coordinate-based multi-layer perceptron, whose inputs are image spatial coordinates and outputs are corresponding pixel intensities. The trained INR not only constrains functional smoothness but also allows an arbitrary scale in SR imaging. Main results. We demonstrate the efficacy of the proposed framework on various biomedical images, including computed tomography (CT), magnetic resonance imaging (MRI), fluorescence microscopy, and ultrasound images, across different SR magnification scales of 2×, 4×, and 8×. A limited number of LR images were used for each of the SR imaging tasks to show the potential of the proposed statistical INR framework. Significance. The proposed method provides an urgently needed unsupervised deep learning framework for numerous biomedical SR applications that lack HR reference images.

Super-resolution image restoration for microlens array imaging system

Microlens array (MLA) has been widely applied in optical imaging fields. However, the existing MLA imaging system (MLAIS) often suffers the problems of low imaging resolutions and luminance contrast due to the limitation of the sensor size and microlens aperture. To address these problems, we propose a super-resolution (SR) image restoration method for the microlens array imaging system (MLAIS). The proposed method uses an Otsu-Kmeans gravity-based center extraction method to segment the image captured by MLAIS into many sub-images and an enhanced correlation coefficient-based approach to align the sub-images. A maximum a posteriori probability algorithm is developed to recover the SR image from the aligned sub-images. A two-step gray transformation algorithm is designed to enhance the SR image. Numerical simulations and practical experiments are implemented to verify the effectiveness of the proposed method, and its SR restoration performance is evaluated by comparing the results with classical and state-of-the-art methods. The results indicate that the proposed method can achieve high-quality SR image restoration for the MLAIS, and its SR performance is better than the comparison methods. The proposed method may find applications in multispectral SR imaging, remote sensing SR imaging, muti-sensor SR imaging, and so on.

Deep neural network based on multi-level wavelet and attention for structured illumination microscopy

Structured illumination microscopy (SIM) is a popular and powerful super-resolution (SR) technique in biomedical research. However, the conventional reconstruction algorithm for SIM heavily relies on the accurate prior knowledge of illumination patterns and signal-to-noise ratio (SNR) of raw images. To obtain high-quality SR images, several raw images need to be captured under high fluorescence level, which further restricts SIM’s temporal resolution and its applications. Deep learning (DL) is a data-driven technology that has been used to expand the limits of optical microscopy. In this study, we propose a deep neural network based on multi-level wavelet and attention mechanism (MWAM) for SIM. Our results show that the MWAM network can extract high-frequency information contained in SIM raw images and accurately integrate it into the output image, resulting in superior SR images compared to those generated using wide-field images as input data. We also demonstrate that the number of SIM raw images can be reduced to three, with one image in each illumination orientation, to achieve the optimal tradeoff between temporal and spatial resolution. Furthermore, our MWAM network exhibits superior reconstruction ability on low-SNR images compared to conventional SIM algorithms. We have also analyzed the adaptability of this network on other biological samples and successfully applied the pretrained model to other SIM systems.

The miEye: Bench-top super-resolution microscope with cost-effective equipment

Commercial super-resolution (SR) imaging systems require a high budget, while current more affordable open source microscopy systems lack modularity and sometimes are too complex or lack reliability. We present miEye – a cost-effective microscope designed for high-resolution wide-field fluorescence imaging. The build is constructed using a CNC milled aluminum microscope body and commercially available optomechanics, with open-source Python-based microscope control, data visualization, and analysis software integration. The data acquisition software works robustly with commonly used industrial-grade complementary metal oxide semiconductor (iCMOS) cameras, performs IR beam back-reflection-based automatic focus stabilization, and allows for laser control via an Arduino-based laser relay. The open-source nature of the design is aimed to facilitate adaptation by the community. The build can be constructed for a cost of roughly 50 k €. It contains SM-fiber and MM-fiber excitation paths that are easy to interchange and an adaptable emission path. Also, it ensures <5 nm/min stability of the sample on all axes, and allows achieving <30 nm lateral resolution for dSTORM and DNA-PAINT single-molecule localization microscopy (SMLM) experiments. Thus it serves as a cost-effective and adaptable addition to the open source microscopy community and potentially will allow high-quality SR imaging even for limited-budget research groups.

3D reconstruction of porous media by combining scaling transformation and multi-scale discrimination using generative adversarial networks

The modeling and characterization of porous media is quite significant to explore and develop oil and natural gas resources . Traditional numerical simulation methods obtain reconstruction results through the statistical probability in training images (TIs), while the reconstruction process is lengthy and the probability information cannot be reused. The development of deep learning has provided reliable support for the reconstruction of porous media due to its strong ability of learning and extracting characteristics from TIs and the reuse of network parameters. As a common deep learning method, generative adversarial networks (GANs) can obtain images highly similar to the structural characteristics of the TI through adversarial training between the generator and the discriminator . Based on GAN, a three-dimensional multi-scale pattern generative adversarial network (3D-MSPGAN) that can achieve 3D super-resolution reconstruction of porous media is proposed in this paper. The use of scaling transformation allows 3D-MSPGAN to reconstruct super-resolution images of porous media that are much larger than the TI, while the multi-scale discrimination in the discriminator provides a guarantee for the generation of high-quality super-resolution images, by which the discriminator can retain the global structure and local characteristics of the TI simultaneously during the reconstruction process. In addition, due to the multi-scale patch information used in the discriminator, only a single TI is needed to complete the reconstruction of porous media. Experimental comparison with some typical methods proves that 3D-MSPGAN can achieve the reconstruction of 3D porous media with faster speed and higher quality. • Our method can achieve super-resolution reconstruction of porous media. • Our method only needs a single training image for the reconstruction. • Our method can complete large-quantity reconstructions quickly.

LASSR: Effective super-resolution method for plant disease diagnosis

The collection of high-resolution training data is crucial in building robust plant disease diagnosis systems, since such data have a significant impact on diagnostic performance. However, they are very difficult to obtain and are not always available in practice. Deep learning-based techniques, and particularly generative adversarial networks (GANs), can be applied to generate high-quality super-resolution images, but these methods often produce unexpected artifacts that can lower the diagnostic performance. In this paper, we propose a novel artifact-suppression super-resolution method that is specifically designed for diagnosing leaf disease, called Leaf Artifact-Suppression Super-Resolution (LASSR). Thanks to its own artifact removal module that detects and suppresses artifacts to a considerable extent, LASSR can generate much more pleasing, high-quality images compared to the state-of-the-art ESRGAN model. Experiments based on a five-class cucumber disease (including healthy) discrimination model show that training with data generated by LASSR significantly boosts the performance on an unseen test dataset by over 21% compared with the baseline, and that our approach is more than 2% better than a model trained with images generated by ESRGAN.

Translation position extracting in incoherent pattern-illumination Fourier ptychographic photography

Incoherent Pattern-illuminated Fourier ptychography (IFP) is a newly developed super-resolution method, where accurate prior knowledge of translation positions is essential for image reconstruction. To release this limitation, we propose a pre-processing algorithm capable of extracting translation positions of the structure light directly from raw images of IFP, termed translation position extracting (TPE). TPE mainly involves two steps. First, the speckle parts mixed in the acquired raw images, in which the illumination motion is encoded, are isolated by intensity averaging and division. Then the cross-correlations of the speckle dataset are computed to determine the shift positions. TPE-IFP improves the previous IFP by removal of the requirement for prior knowledge of translation positions. Its effectiveness is demonstrated by obtaining high-quality super-resolution images in absence of location information in both simulations and experiments. By further relaxing the practical conditions, the proposed TPE may accelerate the applications of IFP. What's more, as a pre-processing approach, TPE might also contribute to the estimation of pattern positions for the similar speckle-based imaging.

Multi-color structured illumination microscopy for live cell imaging based on the enhanced image recombination transform algorithm

Structured illumination microscopy (SIM) has attracted considerable interest in super-resolution, live-cell imaging because of its low light dose and high imaging speed. Obtaining a high-quality reconstruction image in SIM depends on the precise determination of the parameters of the fringe illumination pattern. The image recombination transform (IRT) algorithm is superior to other algorithms in obtaining the precise initial phase without any approximation, which is promising to provide a considerable solution to address the difficulty of initial phase estimation at low-modulation-depth conditions. However, the IRT algorithm only considers a phase shift of π∕2, which limits its applications in general scenarios. In this letter, we present a general form of IRT algorithm suitable for arbitrary phase shifts, providing a powerful tool for parameter estimation in low signal-to-noise cases. To demonstrate the effectiveness of the enhanced IRT algorithm, we constructed a multicolor, structured illumination microscope and studied at super-resolution, the cargo traffic in HRPE cells, and monitored the movement of mitochondrial structures and microtubules in COS-7 cells. The custom SIM system using the enhanced IRT algorithm allows multicolor capability and a low excitation intensity fluorescence imaging less than 1 W/cm2. High-quality super-resolution images are obtained, which demonstrates the utility of this approach in imaging in the life sciences.

Remote sensing image super-resolution using cascade generative adversarial nets

Image super-resolution (SR) is a widely used and cost-effective technology in remote sensing image processing. Deep learning-based SR methods have shown promising performance, but they are prone to losing texture details. Instead, generative adversarial nets (GAN)-based methods can generate more visually acceptable results. However, GAN-based SR methods are suffering from scene variance and uncontrollable performance of discriminators as well as unstable training. Besides, both these two methods cannot yield arbitrary high-time SR images. To solve these issues, we propose a novel SR method for remote sensing images using Cascade Generative Adversarial Nets (CGAN) with introduction of content fidelity and scene constraint, which can achieve arbitrary high-time high-quality SR image. More specifically, the scene-constraint item is incorporated to constrain generated feature for avoiding the risk of scene change. Then, content fidelity is introduced to stabilize the training and avoid gradient vanishing. Besides, an edge enhancement module is designed to preserve edge detail and suppress the noise. CGAN with these three components has achieved higher quality SR results than other recent state-of-the-art methods. Compared with these methods, our proposed method outperformed average increments of 7.3% SSIM, 7.3% FSIM and 6.0% MSIM on WHU-RS19 and NWPU-RESISC45 datasets. In addition, the evaluation of GAN-train and GAN-test gained average increments of 6.3% and 4.5% on the WHU-RS19 and AID datasets, respectively.

Optimized Stochastic Optical Reconstruction Microscopy for Imaging Chromatin Structure in Pathological Tissue.

Direct visualization of higher-order chromatin structure at the molecular scale is of great importance for understanding the impact of chromatin organization on gene expression in many biological processes. Understanding the changes in chromatin structure during pathological processes requires the use of in vivo models and clinical samples, and formalin-fixed, paraffin-embedded (FFPE) tissue is the most widespread form of preservation. Here we describe the details of PathSTORM, an optimized stochastic optical reconstruction microscopy (STORM) protocol for high-quality super-resolution imaging of densely packed higher-order chromatin organization in pathological tissue. We discuss detailed methods for fluorescence staining of DNA and histone proteins, as well as the key technical factors for obtaining high-quality STORM images in pathological tissue samples. © 2020 Wiley Periodicals LLC Basic Protocol 1: Fluorescence staining of chromatin in pathological tissue Basic Protocol 2: STORM data processing Support Protocol 1: Drift correction Support Protocol 2: Image reconstruction Support Protocol 3: Hematoxylin & eosin (H&E) staining.

Enhanced high-quality super-resolution imaging in air using microsphere lens groups.

Most microsphere-assisted super-resolution imaging experiments require a high-refractive-index microsphere to be immersed in a liquid to improve the super-resolution. However, samples are inevitably polluted by residuals in the liquid. This Letter presents a novel (to the best of our knowledge) method employing a microsphere lens group (MLG) that can easily achieve high-quality super-resolution imaging in air. The performance of this method is at par or better than that of the high-refractive-index microspheres immersed in liquid. In addition, the MLG generates a real image that is closely related to the photonic nanojet position of the microsphere super-lens. This imaging method is beneficial in microsphere imaging applications where liquids are impractical.

A hybrid convolutional neural network for super-resolution reconstruction of MR images.

Spatial resolution is an important parameter for magnetic resonance imaging (MRI). High-resolution MR images provide detailed information and benefit subsequent image analysis. However, higher resolution MR images come at the expense of longer scanning time and lower signal-to-noise ratios (SNRs). Using algorithms to improve image resolution can mitigate these limitations. Recently, some convolutional neural network (CNN)-based super-resolution (SR) algorithms have flourished on MR image reconstruction. However, most algorithms usually adopt deeper network structures to improve the performance. In this study, we propose a novel hybrid network (named HybridNet) to improve the quality of SR images by increasing the width of the network. Specifically, the proposed hybrid block combines a multipath structure and variant dense blocks to extract abundant features from low-resolution images. Furthermore, we fully exploit the hierarchical features from different hybrid blocks to reconstruct high-quality images. All SR algorithms are evaluated using three MR image datasets and the proposed HybridNet outperformed the comparative methods with peak a signal-to-noise ratio (PSNR) of 42.12±0.92dB, 38.60±2.46dB, 35.17±2.96dB and a structural similarity index (SSIM) of 0.9949±0.0015, 0.9892±0.0034, 0.9740±0.0064, respectively. Besides, our proposed network can reconstruct high-quality images on an unseen MR dataset with PSNR of 33.27±1.56 and SSIM of 0.9581±0.0068. The results demonstrate that HybridNet can reconstruct high-quality SR images from degraded MR images and has good generalization ability. It also can be leveraged to assist the task of image analysis or processing.

Bright ligand-activatable fluorescent protein for high-quality multicolor live-cell super-resolution microscopy

We introduce UnaG as a green-to-dark photoswitching fluorescent protein capable of high-quality super-resolution imaging with photon numbers equivalent to the brightest photoswitchable red protein. UnaG only fluoresces upon binding of a fluorogenic metabolite, bilirubin, enabling UV-free reversible photoswitching with easily controllable kinetics and low background under Epi illumination. The on- and off-switching rates are controlled by the concentration of the ligand and the excitation light intensity, respectively, where the dissolved oxygen also promotes the off-switching. The photo-oxidation reaction mechanism of bilirubin in UnaG suggests that the lack of ligand-protein covalent bond allows the oxidized ligand to detach from the protein, emptying the binding cavity for rebinding to a fresh ligand molecule. We demonstrate super-resolution single-molecule localization imaging of various subcellular structures genetically encoded with UnaG, which enables facile labeling and simultaneous multicolor imaging of live cells. UnaG has the promise of becoming a default protein for high-performance super-resolution imaging.

Soft-edge Assisted Network for Single Image Super-Resolution.

The task of single image super-resolution (SISR) is a highly ill-posed inverse problem since reconstructing the highfrequency details from a low-resolution image is challenging. Most previous CNN-based super-resolution (SR) methods tend to directly learn the mapping from the low-resolution image to the high-resolution image through some complex convolutional neural networks. However, the method of blindly increasing the depth of the network is not the best choice because the performance improvement of such methods is marginal but the computational cost is huge. A more efficient method is to integrate the image prior knowledge into the model to assist the image reconstruction. Indeed, the soft-edge has been widely applied in many computer vision tasks as the role of an important image feature. In this paper, we propose a Soft-edge assisted Network (SeaNet) to reconstruct the high-quality SR image with the help of image soft-edge. The proposed SeaNet consists of three sub-nets: a rough image reconstruction network (RIRN), a soft-edge reconstruction network (Edge-Net), and an image refinement network (IRN). The complete reconstruction process consists of two stages. In Stage-I, the rough SR feature maps and the SR soft-edge are reconstructed by the RIRN and Edge-Net, respectively. In Stage-II, the outputs of the previous stages are fused and then feed to the IRN for high-quality SR image reconstruction. Extensive experiments show that our SeaNet converges rapidly and achieves excellent performance under the assistance of image soft-edge. The code is available at https://gitlab.com/junchenglee/seanet-pytorch.

Imaging Higher-order Chromatin Structures in Single Cells Using Stochastic Optical Reconstruction Microscopy.

Higher-order chromatin organization shaped by epigenetic modifications influence the chromatin environment and subsequently regulate gene expression. Direct visualization of the higher-order chromatin structure at their epigenomic states is of great importance for understanding chromatin compaction and its subsequent effect on gene expression and various cellular processes. With the recent advances in super-resolution microscopy, the higher-order chromatin structure can now be directly visualized in situ down to the scale of ~30 nm. This protocol provides detailed description of super-resolution imaging of higher-order chromatin structure using stochastic optical reconstruction microscopy (STORM). We discussed fluorescence staining methods of DNA and histone proteins and crucial technical factors to obtain high-quality super-resolution images.

Single image super-resolution under multi-frame method

The multi-frame image super-resolution method utilizes a series of low-resolution images from the same scene to reconstruct the corresponding high-quality super-resolution image. However, the insufficient input low-resolution images make it incapable of reconstructing the high-resolution image from a single low-resolution image. In this work, we extend the framework of multi-frame image super-resolution to handle the single low-resolution image via rolling guidance filtering. And an improved version of diffusion-driven regularizer-based multi-frame image super-resolution algorithm is proposed and applied on passive millimeter-wave (PMMW) image super-resolution. Specifically, the joint filtering is first exploited to suppress the noise of single low-resolution noisy image. The rolling guidance method is exploited to generate the structurally multi-scale low-resolution images forming the basis of multi-frame image super-resolution. The generated image sequences are then fed to the nonlinear diffusion regularizer-based super-resolution algorithm. The two-directional total variation de-convolution is finally employed to remove the blur, producing a sharp and clear high-resolution image. Experiments demonstrate the effectiveness of the proposed method and show its superiority for the natural and PMMW images.

X10 expansion microscopy enables 25-nm resolution on conventional microscopes.

Expansion microscopy is a recently introduced imaging technique that achieves super‐resolution through physically expanding the specimen by ~4×, after embedding into a swellable gel. The resolution attained is, correspondingly, approximately fourfold better than the diffraction limit, or ~70 nm. This is a major improvement over conventional microscopy, but still lags behind modern STED or STORM setups, whose resolution can reach 20–30 nm. We addressed this issue here by introducing an improved gel recipe that enables an expansion factor of ~10× in each dimension, which corresponds to an expansion of the sample volume by more than 1,000‐fold. Our protocol, which we termed X10 microscopy, achieves a resolution of 25–30 nm on conventional epifluorescence microscopes. X10 provides multi‐color images similar or even superior to those produced with more challenging methods, such as STED, STORM, and iterative expansion microscopy (iExM). X10 is therefore the cheapest and easiest option for high‐quality super‐resolution imaging currently available. X10 should be usable in any laboratory, irrespective of the machinery owned or of the technical knowledge.