Point Spread Function Engineering Research Articles

Research Progress of Double-Helix Point Spread Function Engineering and Its Application

Research Progress of Double-Helix Point Spread Function Engineering and Its Application

Three-Dimensional Multifocal Two-Photon Laser Scanning Microscopy Based on Double-Helix Point Spread Function Engineering

Three-Dimensional Multifocal Two-Photon Laser Scanning Microscopy Based on Double-Helix Point Spread Function Engineering

High-axial-resolution single-molecule localization under dense excitation with a multi-channel deep U-Net.

Single-molecule localization microscopy (SMLM) can bypass the diffraction limit of optical microscopes and greatly improve the resolution in fluorescence microscopy. By introducing the point spread function (PSF) engineering technique, we can customize depth varying PSF to achieve higher axial resolution. However, most existing 3D single-molecule localization algorithms require excited fluorescent molecules to be sparse and captured at high signal-to-noise ratios, which results in a long acquisition time and precludes SMLM's further applications in many potential fields. To address this problem, we propose a novel 3D single-molecular localization method based on a multi-channel neural network based on U-Net. By leveraging the deep network's great advantages in feature extraction, the proposed network can reliably discriminate dense fluorescent molecules with overlapped PSFs and corrupted by sensor noise. Both simulated and real experiments demonstrate its superior performance in PSF engineered microscopes with short exposure and dense excitations, which holds great potential in fast 3D super-resolution microscopy.

Simultaneous orientation and 3D localization microscopy with a Vortex point spread function

Estimating the orientation and 3D position of rotationally constrained emitters with localization microscopy typically requires polarization splitting or a large engineered Point Spread Function (PSF). Here we utilize a compact modified PSF for single molecule emitter imaging to estimate simultaneously the 3D position, dipole orientation, and degree of rotational constraint from a single 2D image. We use an affordable and commonly available phase plate, normally used for STED microscopy in the excitation light path, to alter the PSF in the emission light path. This resulting Vortex PSF does not require polarization splitting and has a compact PSF size, making it easy to implement and combine with localization microscopy techniques. In addition to a vectorial PSF fitting routine we calibrate for field-dependent aberrations which enables orientation and position estimation within 30% of the Cramér-Rao bound limit over a 66 μm field of view. We demonstrate this technique on reorienting single molecules adhered to the cover slip, λ-DNA with DNA intercalators using binding-activated localization microscopy, and we reveal periodicity on intertwined structures on supercoiled DNA.

Computational Recovery of Engineered Point Spread Functions in Single Molecule Localization Microscopy using the Double Helix 3DTRAX Software

Computational Recovery of Engineered Point Spread Functions in Single Molecule Localization Microscopy using the Double Helix 3DTRAX Software

Deep-Learning-Based Virtual Refocusing of Images Using an Engineered Point-Spread Function

We present a virtual image refocusing method over an extended depth of field (DOF) enabled by cascaded neural networks and a double-helix point-spread function (DH-PSF). This network model, referred to as W-Net, is composed of two cascaded generator and discriminator network pairs. The first generator network learns to virtually refocus an input image onto a user-defined plane, while the second generator learns to perform a cross-modality image transformation, improving the lateral resolution of the output image. Using this W-Net model with DH-PSF engineering, we extend the DOF of a fluorescence microscope by ~20-fold. This approach can be applied to develop deep learning-enabled image reconstruction methods for localization microscopy techniques that utilize engineered PSFs to improve their imaging performance, including spatial resolution and volumetric imaging throughput.

Heterochromatic Nonlinear Optical Responses in Upconversion Nanoparticles for Super-Resolution Nanoscopy.

Point spread function (PSF) engineering by an emitter's response can code higher-spatial-frequency information of an image for microscopy to achieve super-resolution. However, complexed excitation optics or repetitive scans are needed, which explains the issues of low speed, poor stability, and operational complexity associated with the current laser scanning microscopy approaches. Here, the diverse emission responses of upconversion nanoparticles (UCNPs) are reported for super-resolution nanoscopy to improve the imaging quality and speed. The method only needs a doughnut-shaped scanning excitation beam at an appropriate power density. By collecting the four-photon emission of single UCNPs, the high-frequency information of a super-resolution image can be resolved through the doughnut-emission PSF. Meanwhile, the two-photon state of the same nanoparticle is oversaturated, so that the complementary lower-frequency information of the super-resolution image can be simultaneously collected by the Gaussian-like emission PSF. This leads to a method of Fourier-domain heterochromatic fusion, which allows the extended capability of the engineered PSFs to cover both low- and high-frequency information to yield optimized image quality. This approach achieves a spatial resolution of 40nm, 1/24th of the excitation wavelength. This work suggests a new scope for developing nonlinear multi-color emitting probes in super-resolution nanoscopy.

3D Particle Localization in Living Cells by Deep Learning

Point spread function engineering techniques have extended the axial range accessible to fluorescence microscopy by simple modifications to the optical path of a microscope. A particularly promising engineered PSF is the double helix PSF (DHPSF), which provides an axial range of over 2 μm. This extended depth allows sub-diffraction localization of point sources in 3D across an axial range approaching the typical thickness of mammalian adherent cells. Thus, DHPSF is not only attractive for PALM-style superresolution experiments but also for tracking small punctate structures in cells. Still, data analysis of DHPSF images presents a challenge because no analytic form of the DHPSF exists. This difficulty has been addressed previously by using heuristic functions for fitting or interpolations of calibration scans of fluorescent beads. However, this approach is limiting for cellular particle tracking applications of DHPSF because of the heterogeneous signal background, high particle densities, and large variation in intensities of the target particles encountered in such experiments. Here, we present a data-driven and fast method based on deep learning (DL) to address these challenges. Specifically, we use the DL approach to analyze DHPSF data without the need to introduce heuristic fitting functions or interpolate calibration scans. We demonstrate the accuracy of the method on localizing simulated data in 3D, and test its efficacy in the presence of noisy, heterogeneous backgrounds. Finally, we localize and track in 3D puncta of human T-cell leukemia virus type 1 (HTLV-1) in live cells and present preliminary data on the mobilities of these particles measured in this noisy environment. This work has been supported by grants from the National Institutes of Health (R01 GM098550 and RO1 GM124279).

Research Progress of Computational Microscopy Imaging Based on Point Spread Function Engineering

Research Progress of Computational Microscopy Imaging Based on Point Spread Function Engineering

Joint Estimation of Trajectory and Model Parameters for Single Particle Tracking of 3D Confined Diffusion Using the Double-Helix Point Spread Function.

Single particle tracking plays a significant role in biophysics through its ability to reveal dynamic mechanisms and physical properties of biological macromolecules inside living cells. The motion of these molecules can often be modeled as a confined diffusion. The standard paradigm in the biophysics community is to first estimate the trajectory of a particle and then use a technique such as the Mean Square Displacement or the Maximum Likelihood Estimation (MLE) to determine model parameters. These approaches, however, ignore the fact that localization and parameter estimation problems are coupled. We have previously introduced a framework based on optimal estimation theory to simultaneously do localization and parameter estimation. Here we build upon that work by expanding it to include a recent advance in imaging three dimensional motion, namely the Double-Helix (DH) engineered Point Spread Function (PSF). The DH-PSF encodes the axial position of the particle directly into the 2D image acquired by the camera mounted to the microscope. Our approach uses Expectation Maximization (EM) and Sequential Monte Carlo (SMC) to handle the nonlinearities in the observation and motion models. In this paper, we also improve upon the computational complexity of this scheme, using a Gaussian Particle Filter and Backward Simulation Particle Smoother in the SMC elements of the algorithm. We compare our scheme through simulation to state of the art methods based on localization using Gaussian fitting followed by MLE of the model parameters. These results show that our method outperforms GF-MLE at the low signal intensity levels common to biophysical experiments.

Mutifocal image scanning microscopy based on double-helix point spread function engineering

Confocal laser scanning microscopy (CLSM) is a powerful imaging tool providing high resolution and optical sectioning. In its standard optical configuration, a pair of confocal pinholes is used to reject out-of-focus light. The diffraction limited resolution can be broken by reducing the confocal pinhole size. But this comes at the cost of extremely low signal-to-noise ratio (SNR). The limited SNR problem can be solved by image scanning microscopy (ISM), in which the single-point detector of a regular point-scanning confocal microscopy is substituted with an array detector such as CCD or CMOS, thus the two-fold super-resolution imaging can be achieved by pixel reassignment and deconvolution. However, the practical application of ISM is challenging due to its limited image acquisition speed. Here, we present a hybrid microscopy technique, named multifocal refocusing after scanning using helical phase engineering microscopy (MRESCH), which combines the double-helix point spread function (DH-PSF) engineering with multifocal structured illumination to dramatically improve the image acquisition speed. In the illumination path, sparse multifocal illumination patterns are generated by a digital micromirror device for parallel imaging information acquisition. In the detection path, a phase mask is introduced to modulate the conventional PSF to the DH-PSF, which provides volumetric information, and meanwhile, we also present a digital refocusing strategy for processing the collected raw data to recover the wild-filed image from different sample layers. To demonstrate imaging capabilities of MRESCH, we acquire the images of mitochondria in live HeLa cells and make a detailed comparison with those from the wide-field microscopy. In contrast to the conventional wide-field approach, the MRESCH can expand the imaging depth in a range from –1 μm to 1 μm. Next, we sample the F-actin of bovine pulmonary artery endothelial cells to characterize the lateral resolution of the MRESCH. The results show that the MRESCH has a better resolution capability than the conventional wide-field illumination microscopy. Finally, the proposed image scanning microscopy can record three-dimensional specimen information from a single multi-spot two-dimensional scan, which ensures faster data acquisition and larger field of view than ISM.

Recent advances in point spread function engineering and related computational microscopy approaches: from one viewpoint.

This personal hybrid review piece, written in light of my recipience of the UIPAB 2020 young investigator award, contains a mixture of my scientific biography and work so far. This paper is not intended to be a comprehensive review, but only to highlight my contributions to computation-related aspects of super-resolution microscopy, as well as their origins and future directions.

Integrating engineered point spread functions into the phasor-based single-molecule localization microscopy framework.

In single-molecule localization microscopy (SMLM), the use of engineered point spread functions (PSFs) provides access to three-dimensional localization information. The conventional approach of fitting PSFs with a single 2-dimensional Gaussian profile, however, often falls short in analyzing complex PSFs created by placing phase masks, deformable mirrors or spatial light modulators in the optical detection pathway. Here, we describe the integration of PSF modalities known as double-helix, saddle-point or tetra-pod into the phasor-based SMLM (pSMLM) framework enabling fast CPU based localization of single-molecule emitters with sub-pixel accuracy in three dimensions. For the double-helix PSF, pSMLM identifies the two individual lobes and uses their relative rotation for obtaining z-resolved localizations. For the analysis of saddle-point or tetra-pod PSFs, we present a novel phasor-based deconvolution approach entitled circular-tangent pSMLM. Saddle-point PSFs were experimentally realized by placing a deformable mirror in the Fourier plane and modulating the incoming wavefront with specific Zernike modes. Our pSMLM software package delivers similar precision and recall rates to the best-in-class software package (SMAP) at signal-to-noise ratios typical for organic fluorophores and achieves localization rates of up to 15kHz (double-helix) and 250kHz (saddle-point/tetra-pod) on a standard CPU. We further integrated pSMLM into an existing software package (SMALL-LABS) suitable for single-particle imaging and tracking in environments with obscuring backgrounds. Taken together, we provide a powerful hardware and software environment for advanced single-molecule studies.

Single-molecule orientation localization microscopy for resolving structural heterogeneities between amyloid fibrils.

Simultaneous measurements of single-molecule positions and orientations provide critical insight into a variety of biological and chemical processes. Various engineered point spread functions (PSFs) have been introduced for measuring the orientation and rotational diffusion of dipole-like emitters, but the widely used Cramér-Rao bound (CRB) only evaluates performance for one specific orientation at a time. Here, we report a performance metric, termed variance upper bound (VUB), that yields a global maximum CRB for all possible molecular orientations, thereby enabling the measurement performance of any PSF to be computed efficiently (~1000× faster than calculating average CRB). Our VUB reveals that the simple polarized standard PSF provides robust and precise orientation measurements if emitters are near a refractive index interface. Using this PSF, we measure the orientations and positions of Nile red (NR) molecules transiently bound to amyloid aggregates. Our super-resolved images reveal the main binding mode of NR on amyloid fiber surfaces, as well as structural heterogeneities along amyloid fibrillar networks, that cannot be resolved by single-molecule localization alone.

Visualizable detection of nanoscale objects using anti-symmetric excitation and non-resonance amplification

Why can we not see nanoscale objects under a light microscope? The textbook answers are that their relative signals are weak and their separation is smaller than Abbe’s resolution limit. Thus, significant effort has gone into developing ultraviolet imaging, oil and solid immersion objectives, nonlinear methods, fluorescence dyes, evanescent wave tailoring, and point-spread function engineering. In this work, we introduce a new optical sensing framework based on the concepts of electromagnetic canyons and non-resonance amplification, to directly view on a widefield microscope λ/31-scale (25-nm radius) objects in the near-field region of nanowire-based sensors across a 726-μm × 582-μm field of view. Our work provides a simple but highly efficient framework that can transform conventional diffraction-limited optical microscopes for nanoscale visualization. Given the ubiquity of microscopy and importance of visualizing viruses, molecules, nanoparticles, semiconductor defects, and other nanoscale objects, we believe our proposed framework will impact many science and engineering fields.

End-to-end Learned, Optically Coded Super-resolution SPAD Camera

Single Photon Avalanche Photodiodes (SPADs) have recently received a lot of attention in imaging and vision applications due to their excellent performance in low-light conditions, as well as their ultra-high temporal resolution. Unfortunately, like many evolving sensor technologies, image sensors built around SPAD technology currently suffer from a low pixel count. In this work, we investigate a simple, low-cost, and compact optical coding camera design that supports high-resolution image reconstructions from raw measurements with low pixel counts. We demonstrate this approach for regular intensity imaging, depth imaging, as well transient imaging. Our method uses an end-to-end framework to simultaneously optimize the optical design and a reconstruction network for obtaining super-resolved images from raw measurements. The optical design space is that of an engineered point spread function (implemented with diffractive optics), which can be considered an optimized anti-aliasing filter to preserve as much high-resolution information as possible despite imaging with a low pixel count, low fill-factor SPAD array. We further investigate a deep network for reconstruction. The effectiveness of this joint design and reconstruction approach is demonstrated for a range of different applications, including high-speed imaging, and time of flight depth imaging, as well as transient imaging. While our work specifically focuses on low-resolution SPAD sensors, similar approaches should prove effective for other emerging image sensor technologies with low pixel counts and low fill-factors.

Three-Dimensional Single Particle Tracking and Its Applications in Confined Environments.

Single particle tracking (SPT) has proven to be a powerful technique in studying molecular dynamics in complicated systems. We review its recent development, including three-dimensional (3D) SPT and its applications in probing nanostructures and molecule-surface interactions that are important to analytical chemical processes. Several frequently used 3D SPT techniques are introduced. Especially of interest are those based on point spread function engineering, which are simple in instrumentation and can be easily adapted and used in analytical labs. Corresponding data analysis methods are briefly discussed. We present several important case studies, with a focus on probing mass transport and molecule-surface interactions in confined environments. The presented studies demonstrate the great potential of 3D SPT for understanding fundamental phenomena in confined space, which will enable us to predict basic principles involved in chemical recognition, separation, and analysis, and to optimize mass transport and responses by structural design and optimization.

Real-Time Point Spread Function Engineering for iSCAT

Real-Time Point Spread Function Engineering for iSCAT

Accurate and rapid background estimation in single-molecule localization microscopy using the deep neural network BGnet

Background fluorescence, especially when it exhibits undesired spatial features, is a primary factor for reduced image quality in optical microscopy. Structured background is particularly detrimental when analyzing single-molecule images for 3-dimensional localization microscopy or single-molecule tracking. Here, we introduce BGnet, a deep neural network with a U-net-type architecture, as a general method to rapidly estimate the background underlying the image of a point source with excellent accuracy, even when point-spread function (PSF) engineering is in use to create complex PSF shapes. We trained BGnet to extract the background from images of various PSFs and show that the identification is accurate for a wide range of different interfering background structures constructed from many spatial frequencies. Furthermore, we demonstrate that the obtained background-corrected PSF images, for both simulated and experimental data, lead to a substantial improvement in localization precision. Finally, we verify that structured background estimation with BGnet results in higher quality of superresolution reconstructions of biological structures.

A Proposed Method for Optimizing the Spectral Discernibility of Engineered Point-spread Functions for Localization Microscopy

A Proposed Method for Optimizing the Spectral Discernibility of Engineered Point-spread Functions for Localization Microscopy