Stochastic Super-resolution Research Articles

A flow-based multi-scale learning network for single image stochastic super-resolution

Single image super-resolution (SISR) is still an important while challenging task. Existing methods usually ignore the diversity of generated Super-Resolution (SR) images. The fine details of the corresponding high-resolution (HR) images cannot be confidently recovered due to the degradation of detail in low-resolution (LR) images. To address the above issue, this paper presents a flow-based multi-scale learning network (FMLnet) to explore the diverse mapping spaces for SR. First, we propose a multi-scale learning block (MLB) to extract the underlying features of the LR image. Second, the introduced pixel-wise multi-head attention allows our model to map multiple representation subspaces simultaneously. Third, by employing a normalizing flow module for a given LR input, our approach generates various stochastic SR outputs with high visual quality. The trade-off between fidelity and perceptual quality can be controlled. Finally, the experimental results on five datasets demonstrate that the proposed network outperforms the existing methods in terms of diversity, and achieves competitive PSNR/SSIM results. Code is available at https://github.com/qianyuwu/FMLnet.

Stochastic Super-Resolution for Downscaling Time-Evolving Atmospheric Fields With a Generative Adversarial Network

Generative adversarial networks (GANs) have been recently adopted for super-resolution, an application closely related to what is referred to as "downscaling" in the atmospheric sciences: improving the spatial resolution of low-resolution images. The ability of conditional GANs to generate an ensemble of solutions for a given input lends itself naturally to stochastic downscaling, but the stochastic nature of GANs is not usually considered in super-resolution applications. Here, we introduce a recurrent, stochastic super-resolution GAN that can generate ensembles of time-evolving high-resolution atmospheric fields for an input consisting of a low-resolution sequence of images of the same field. We test the GAN using two datasets, one consisting of radar-measured precipitation from Switzerland, the other of cloud optical thickness derived from the Geostationary Earth Observing Satellite 16 (GOES-16). We find that the GAN can generate realistic, temporally consistent super-resolution sequences for both datasets. The statistical properties of the generated ensemble are analyzed using rank statistics, a method adapted from ensemble weather forecasting; these analyses indicate that the GAN produces close to the correct amount of variability in its outputs. As the GAN generator is fully convolutional, it can be applied after training to input images larger than the images used to train it. It is also able to generate time series much longer than the training sequences, as demonstrated by applying the generator to a three-month dataset of the precipitation radar data. The source code to our GAN is available at https://github.com/jleinonen/downscaling-rnn-gan.

Super-resolved Measurement of Piconewton Receptor Forces via Tension-PAINT

Piconewton receptor forces exerted by cellular receptors regulate critical cellular processes, including development, migration, immune recognition, and hemostasis. Despite the vital role of mechanical forces in biology, no current technique can image cellular force organization with sub-100 nm resolution. Here, we present tension-PAINT, which leverages DNA-based molecular tension probes in combination with the stochastic super-resolution microscopy technique DNA-Points Accumulation for Imaging in Nanoscale Topography (DNA-PAINT), to enable super-resolved receptor tension measurements. DNA-PAINT relies on the transient binding of fluorophore-labeled oligonucleotide “imager” sequences to complementary “docking” sequences to produce the single-molecule fluorescence required for stochastic super-resolution imaging. We reasoned that encoding a cryptic docking sequence within DNA-based molecular tension probes would allow these probes to function like receptor force-triggered switches. These DNA based molecular force probes reveal a single-stranded DNA docking site only under the influence of pN receptor forces. The resulting technique, which we have named tension-PAINT, provides maps of receptor tension with up to 25 nm spatial resolution and pN force sensitivity. Here, we present variants of tension-PAINT capable of real-time super-resolved tension measurement in living cells, and of integrating cellular force history over time. We apply tension-PAINT to map the integrin forces of living platelets, revealing that platelets exert integrin mediated forces in a 100-200 nm wide spreading “ring” pattern at the lamellipodial edge. Additionally, we reveal the dynamics of fibroblast focal-adhesion and filopodial retraction forces. Due to its compatibility with widely available, unmodified fluorescence microscopes, we anticipate that tension-PAINT will provide a valuable, widely accessible tool to quantify the role of the nanoscale distribution of receptor forces in biological function.

Super-resolution Mapping of Enhanced Emission by Collective Plasmonic Resonances.

Plasmonic particle arrays have remarkable optical properties originating from their collective behavior, which results in resonances with narrow line widths and enhanced electric fields extending far into the surrounding medium. Such resonances can be exploited for applications in strong light-matter coupling, sensing, light harvesting, nonlinear nanophotonics, lasing, and solid-state lighting. However, as the lattice constants associated with plasmonic particle arrays are on the order of their resonance wavelengths, mapping the interaction between point dipoles and plasmonic particle arrays cannot be done with diffraction-limited methods. Here, we map the enhanced emission of single fluorescent molecules coupled to a plasmonic particle array with ∼20 nm in-plane resolution by using stochastic super-resolution microscopy. We find that extended lattice resonances have minimal influence on the spontaneous decay rate of an emitter but instead can be exploited to enhance the outcoupling and directivity of the emission. Our results can guide the rational design of future optical devices based on plasmonic particle arrays.

Trade-offs between structural integrity and acquisition time in stochastic super-resolution microscopy techniques.

The applicability of widefield stochastic microscopy, such as PALM or STORM, is limited by their long acquisition times. Images are produced from the accumulation of a large number of frames that each contain a scarce number of super-resolved localizations. We show that the random and uneven distribution of localizations leads to a specific type of trade-off between the spatial and temporal resolutions. We derive analytical predictions for the minimal time required to obtain a reliable image at a given spatial resolution. We find that the image completion time scales logarithmically with the ratio of the image size to the spatial resolution volume, with second order corrections due to spurious localization within the background noise. We validate our predictions against experimental localization sequences of labeled microtubule filaments obtained by STORM. Our theoretical framework makes it possible to compare the efficiency of emitters, define optimal labeling strategies, and allow implementation of a stopping criterion for data acquisitions that can be performed using real-time monitoring algorithms.

Super-resolution microscopy with DNA-PAINT.

Super-resolution techniques have begun to transform biological and biomedical research by allowing researchers to observe structures well below the classic diffraction limit of light. DNA points accumulation for imaging in nanoscale topography (DNA-PAINT) offers an easy-to-implement approach to localization-based super-resolution microscopy, owing to the use of DNA probes. In DNA-PAINT, transient binding of short dye-labeled ('imager') oligonucleotides to their complementary target ('docking') strands creates the necessary 'blinking' to enable stochastic super-resolution microscopy. Using the programmability and specificity of DNA molecules as imaging and labeling probes allows researchers to decouple blinking from dye photophysics, alleviating limitations of current super-resolution techniques, making them compatible with virtually any single-molecule-compatible dye. Recent developments in DNA-PAINT have enabled spectrally unlimited multiplexing, precise molecule counting and ultra-high, molecular-scale (sub-5-nm) spatial resolution, reaching ∼1-nm localization precision. DNA-PAINT can be applied to a multitude of in vitro and cellular applications by linking docking strands to antibodies. Here, we present a protocol for the key aspects of the DNA-PAINT framework for both novice and expert users. This protocol describes the creation of DNA origami test samples, in situ sample preparation, multiplexed data acquisition, data simulation, super-resolution image reconstruction and post-processing such as drift correction, molecule counting (qPAINT) and particle averaging. Moreover, we provide an integrated software package, named Picasso, for the computational steps involved. The protocol is designed to be modular, so that individual components can be chosen and implemented per requirements of a specific application. The procedure can be completed in 1-2 d.

Storm-Lite: An Inexpensive Tool to Demonstrate Stochastic Super Resolution Microscopy Techniques in the Classroom

Studying sub-cellular structures is central to biophysics research. However, diffraction limits the resolution in light microscopy to a few hundred nanometers and the finest structures of interest can no longer be resolved. To address this obstacle, a variety of “super-resolution” microscopy techniques have been developed. These techniques make use of the photophysics of fluorescent molecules to increase the resolution to several tens of nanometers.We describe here a device that will allow classrooms worldwide to easily simulate super-resolution microscopy techniques such as photoactivation localization microscopy (PALM) and stochastic reconstruction microscopy (STORM). These techniques are extremely expensive and cumbersome to demonstrate in a classroom setting, limiting students’ exposure to them. By contrast, our device is inexpensive, lightweight, and portable, using only an LED array to simulate the fluorophores, frosted glass to simulate the resolution limit, and a camera and user interface developed for the Raspberry Pi to process the images. All aspects of the device and software design will be made publically available in the hopes that others will build and use this setup in their teaching of super resolution techniques at colleges and universities worldwide.

Superresolution imaging of the local density of states in plasmon lattices

Periodic plasmonic nanostructures have been found promising in controlling photoluminescence directivity and efficiency for a wide variety of applications. Because of the inhomogeneous spatial distribution of the photonic resonances of periodic plasmonic nanostructures, their influence on emission is strongly dependent on the position of emitters relative to the nanostructures. Therefore, mapping the local dependence of directivity, efficiency, and emission rate enhancements is key to understanding and optimizing the devices. We introduce a method of mapping the local enhancement of spontaneous emission rates of emitters coupled to periodic nanostructures based on stochastic superresolution imaging. As an example, we show superresolved measurements of the local density of states (LDOS) at 605 nm induced by a hexagonal lattice of aluminum nanoantennas with a spatial resolution of 40 nm, defined by the size of the colloidal nanosources we use as randomly dispersed probes. We demonstrate that our method is superior to near-field mapping of emission rates. Comparison with electrodynamic simulations indicates that the variation of the decay rate of the emitters in the investigated sample is hardly influenced by the lattice modes and mainly governed by single-particle LDOS variations and nearest-neighbor interactions.

Optical super-resolution microscopy and its applications in nano-catalysis

The resolution of conventional optical microscopy is only ∼200 nm, which is becoming less and less sufficient for a variety of applications. In order to surpass the diffraction limited resolution, super-resolution microscopy (SRM) has been developed to achieve a high resolution of one to tens of nanometers. The techniques involved in SRM can be assigned into two broad categories, namely “true” super-resolution techniques and “functional” super-resolution techniques. In “functional” super-resolution techniques, stochastic super-resolution microscopy (SSRM) is widely used due to its low expense, simple operation, and high resolution. The principle process in SSRM is to accumulate the coordinates of many diffraction-limited emitters (e.g., single fluorescent molecules) on the object by localizing the centroids of the point spread functions (PSF), and then reconstruct the image of the object using these coordinates. When the diffraction-limited emitters take part in a catalytic reaction, the activity distribution and kinetic information about the catalysis by nanoparticles can be obtained by SSRM. SSRM has been applied and exhibited outstanding advantages in several fields of catalysis, such as metal nanoparticle catalysis, molecular sieve catalysis, and photocatalysis. Since SSRM is able to resolve the catalytic activity within one nanoparticle, it promises to accelerate the development and discovery of new and better catalysts. This review will present a brief introduction to SRM, and a detailed description of SSRM and its applications in nano-catalysis.

Chapter 10 - Imaging the Assembly, Structure, and Function of the Nuclear Pore Inside Cells

The nuclear pore complex (NPC) mediates selective transport across the nuclear envelope (NE) and plays crucial roles in several additional cellular functions. In higher eukaryotes, the NPC and the NE disassemble and reassemble during cell division and live-cell imaging has been a powerful tool to analyze these dynamic processes. Here, we present a method for the kinetic analysis of postmitotic NPC assembly and reestablishment of transport competence in intact cells by multicolor 4D imaging and photoswitching. By applying the methods we have established previously using normal rat kidney to HeLa cells, we demonstrate the conservation of NPC assembly in different mammalian cells. We recently showed that the molecular organization of the NPC can be studied by combining stochastic super-resolution microscopy with single-particle averaging and present this method here in detail.

Dynamic Stochastic Superresolution of sparsely observed turbulent systems

Real-time capture of the relevant features of the unresolved turbulent dynamics of complex natural systems from sparse noisy observations and imperfect models is a notoriously difficult problem. The resulting lack of observational resolution and statistical accuracy in estimating the important turbulent processes, which intermittently send significant energy to the large-scale fluctuations, hinders efficient parameterization and real-time prediction using discretized PDE models. This issue is particularly subtle and important when dealing with turbulent geophysical systems with an vast range of interacting spatio-temporal scales and rough energy spectra near the mesh scale of numerical models. Here, we introduce and study a suite of general Dynamic Stochastic Superresolution (DSS) algorithms and show that, by appropriately filtering sparse regular observations with the help of cheap stochastic exactly solvable models, one can derive stochastically ‘superresolved’ velocity fields and gain insight into the important characteristics of the unresolved dynamics, including the detection of the so-called black swans. The DSS algorithms operate in Fourier domain and exploit the fact that the coarse observation network aliases high-wavenumber information into the resolved waveband. It is shown that these cheap algorithms are robust and have significant skill on a test bed of turbulent solutions from realistic nonlinear turbulent spatially extended systems in the presence of a significant model error. In particular, the DSS algorithms are capable of successfully capturing time-localized extreme events in the unresolved modes, and they provide good and robust skill for recovery of the unresolved processes in terms of pattern correlation. Moreover, we show that DSS improves the skill for recovering the primary modes associated with the sparse observation mesh which is equally important in applications. The skill of the various DSS algorithms depends on the energy spectrum of the turbulent signal and the observation time relative to the decorrelation time of the turbulence at a given spatial scale in a fashion elucidated here. The DSS technique exploiting a simple Gaussian closure of the nonlinear stochastic forecast model emerges as the most suitable trade-off between the superresolution skill and computational complexity associated with estimating the cross-correlations between the aliasing modes of the sparsely observed turbulent signal. Such techniques offer a promising and efficient approach to constraining unresolved turbulent fluxes through stochastic superparameterization and a subsequent improvement in coarse-grained filtering and prediction of the next generation atmosphere–ocean system (AOS) models.

Quantifying Actin mRNA Expression with Super Resolution Fluorescence Microscopy

The copy number of a specific mRNA and its subcellular localization in cells is a mode by which cells can regulate protein synthesis. Beta-actin mRNA is transported to filopodia upon serum stimulation, regions of cell growth and movement where actin polymerization is essential (1). The copy number of mRNA molecules directly translates to the amount of actin monomers that are synthesized. Quantifying the actin mRNA copy numbers provides a measure of how polymerization of actin is regulated.To this extent, we apply stochastic super-resolution microscopy with photoswitchable fluorescent probes combined with single-molecule localization (2,3) for quantitative studies of mRNA concentration and localization in eukaryotic cells. For this purpose, we apply various labelling approaches for mRNA, including well-established methods such as fluorescence in-situ hybridization (FISH) (4). Focussing on actin mRNA, we derive quantitative information on copy numbers, and we apply coordinate-based algorithms to analyze our single-molecule super-resolution data.References(1) Ben-Ari et al. (2010) J Cell Sci, 123, 1761-1774.(2) Betzig et al. (2006) Science, 313, 1642-1645.(3) Heilemann et al. (2008) Angew. Chemie, 47, 6172-6176.(4) Levskey, J. M.; Singer, R. (2003) Journal of Cell Science, 116, 2833-2838.

Stochastic super-resolution image reconstruction

The objective of super-resolution (SR) imaging is to reconstruct a single higher-resolution image based on a set of lower-resolution images that were acquired from the same scene to overcome the limitations of image acquisition process for facilitating better visualization and content recognition. In this paper, a stochastic Markov chain Monte Carlo (MCMC) SR image reconstruction approach is proposed. First, a Bayesian inference formulation, which is based on the observed low-resolution images and the prior high-resolution image model, is mathematically derived. Second, to exploit the MCMC sample-generation technique for the stochastic SR image reconstruction, three fundamental issues are observed as follows. First, since the hyperparameter value of the prior image model controls the degree of regularization and intimately affects the quality of the reconstructed high-resolution image, how to determine an optimal hyperparameter value for different low-resolution input images becomes a very challenging task. Rather than exploiting the exhaustive search, an iterative updating approach is developed in this paper by allowing the value of hyperparameter being simultaneously updated in each sample-generation iteration. Second, the samples generated during the so-called burn-in period (measured in terms of the number of samples initially generated) of the MCMC-based sample-generation process are considered unreliable and should be discarded. To determine the length of the burn-in period for each set of low-resolution input images, a time-period bound in closed form is mathematically derived. Third, image artifacts could be incurred in the reconstructed high-resolution image, if the number of samples (counting after the burn-in period) generated by the MCMC-based sample-generation process is insufficient. For that, a variation-sensitive bilateral filter is proposed as a ‘complementary’ post-processing scheme, to improve the reconstructed high-resolution image quality, when the number of samples is insufficient. Extensive simulation results have clearly shown that the proposed stochastic SR image reconstruction method consistently yields superior performance.