3D Point Spread Function Research Articles

Accurate 3D single-molecule localization via vectorial in situ point spread function retrieval and aberration assessment

Single-molecule localization microscopy (SMLM) gradually plays an important role in deep tissue imaging. However, current SMLM methods primarily rely on fiducial marks, neglecting aberrations introduced by thick samples, thereby resulting in decreased image quality in deep tissues. Here, we introduce vectorial in situ point spread function (PSF) retrieval (VISPR), a method that retrieves a precise PSF model considering both system- and sample-induced aberrations under SMLM conditions. By employing the theory of vectorial PSF model and maximum likelihood estimation (MLE) phase retrieval, VISPR is capable of reconstructing an accurate in situ 3D PSF model achieving the theoretically minimum uncertainty and accurately reflecting three-dimensional information of single molecules. This capability enables accurate 3D super-resolution reconstruction in deep regions away from the coverslips. Additionally, VISPR demonstrates applicability in low signal-to-noise ratio scenarios and compatibility with various SMLM microscope modalities. From both simulations and experiments, we verified the superiority and effectiveness of VISPR. We anticipate that VISPR will become a pivotal tool for advancing deep tissue SMLM imaging.

Quantitative microbiology with widefield microscopy: navigating optical artefacts for accurate interpretations

Time-resolved live-cell imaging using widefield microscopy is instrumental in quantitative microbiology research. It allows researchers to track and measure the size, shape, and content of individual microbial cells over time. However, the small size of microbial cells poses a significant challenge in interpreting image data, as their dimensions approache that of the microscope’s depth of field, and they begin to experience significant diffraction effects. As a result, 2D widefield images of microbial cells contain projected 3D information, blurred by the 3D point spread function. In this study, we employed simulations and targeted experiments to investigate the impact of diffraction and projection on our ability to quantify the size and content of microbial cells from 2D microscopic images. This study points to some new and often unconsidered artefacts resulting from the interplay of projection and diffraction effects, within the context of quantitative microbiology. These artefacts introduce substantial errors and biases in size, fluorescence quantification, and even single-molecule counting, making the elimination of these errors a complex task. Awareness of these artefacts is crucial for designing strategies to accurately interpret micrographs of microbes. To address this, we present new experimental designs and machine learning-based analysis methods that account for these effects, resulting in accurate quantification of microbiological processes.

Assessment of intraocular foreign body using high resolution 3D ultrasound imaging

Ocular trauma often involves intraocular foreign bodies (IOFBs) that pose challenges in accurate diagnosis due to their size, shape, and material composition. In this study, we proposed a novel whole-eye 3D ophthalmic ultrasound B-scan (3D-UBS) system for automating image acquisition and improved 3D visualization, thereby improving sensitivity for detecting IOFBs. 3D-UBS utilizes 14 MHz Clarius L20 probe, a motorized translation stage, and a surgical microscope for precise placement and movement. The system’s 3D point spread function (PSF) is 0.377 × 0.550 × 0.894 mm3 characterized by the full-width at half-maximum intensity values in the axial, lateral and elevation directions. Digital phantom and ex vivo ocular models were prepared using four types of IOFBs (i.e., plastic, wood, metal, and glass). Ex vivo models were imaged with both 3D-UBS and clinical computed tomography (CT). Image preprocessing was performed on 3D-UBS images to remove uneven illumination and speckle noise. Multiplanar reformatting in 3D-UBS provides optimal plane selection after acquisition, reducing the need for a trained ultrasonographer. 3D-UBS outperforms CT in contrast for wood and plastic, with mean contrast improvement of 2.43 and 1.84 times, respectively. 3D-UBS was able to identify wood and plastic IOFBs larger than 250 µm and 300 in diameter, respectively. CT, with its wider PSF, was only able to detect wood and plastic IOFBs larger than 600 and 550 µm, respectively. Although contrast was higher in CT for metal and glass IOFBs, 3D-UBS provided sufficient contrast to identify those. 3D-UBS provides an easy-to-use, non-expert imaging approach for identifying small IOFBs of different materials and related ocular injuries at the point of care.

Large-FOV 3D localization microscopy by spatially variant point spread function generation.

Accurate characterization of the microscopic point spread function (PSF) is crucial for achieving high-performance localization microscopy (LM). Traditionally, LM assumes a spatially invariant PSF to simplify the modeling of the imaging system. However, for large fields of view (FOV) imaging, it becomes important to account for the spatially variant nature of the PSF. Here, we propose an accurate and fast principal components analysis-based field-dependent 3D PSF generator (PPG3D) and localizer for LM. Through simulations and experimental three-dimensional (3D) single-molecule localization microscopy (SMLM), we demonstrate the effectiveness of PPG3D, enabling super-resolution imaging of mitochondria and microtubules with high fidelity over a large FOV. A comparison of PPG3D with a shift-variant PSF generator for 3D LM reveals a threefold improvement in accuracy. Moreover, PPG3D is approximately 100 times faster than existing PSF generators, when used in image plane-based interpolation mode. Given its user-friendliness, we believe that PPG3D holds great potential for widespread application in SMLM and other imaging modalities.

High-density volumetric super-resolution microscopy

Volumetric super-resolution microscopy typically encodes the 3D position of single-molecule fluorescence into a 2D image by changing the shape of the point spread function (PSF) as a function of depth. However, the resulting large and complex PSF spatial footprints reduce biological throughput and applicability by requiring lower labeling densities to avoid overlapping fluorescent signals. We quantitatively compare the density dependence of single-molecule light field microscopy (SMLFM) to other 3D PSFs (astigmatism, double helix and tetrapod) showing that SMLFM enables an order-of-magnitude speed improvement compared to the double helix PSF by resolving overlapping emitters through parallax. We demonstrate this optical robustness experimentally with high accuracy ( > 99.2 ± 0.1%, 0.1 locs μm−2) and sensitivity ( > 86.6 ± 0.9%, 0.1 locs μm−2) through whole-cell (scan-free) imaging and tracking of single membrane proteins in live primary B cells. We also exemplify high-density volumetric imaging (0.15 locs μm−2) in dense cytosolic tubulin datasets.

Restoration of Radiographic Neutron Image Using Single-Channel Blind Deconvolution

This paper addresses the challenge of accurately estimating the 2D Point Spread Function (PSF) or blur kernel in neutron radiography, where traditional methods, such as the Edge Spread Function (ESF), prove time-consuming and reliant on manual edge selection. The proposed alternative introduces a robust sparse image prior known as the enhanced Patch-Wise Intensity (EPI) image prior in a singlechannel blind deconvolution algorithm, avoiding the need for intricate devices like pinhole or slit phantoms. Leveraging regularization and optimization techniques, the algorithm efficiently estimates PSF in a single image through a multilayer iterative alternating approach. The study aims to enhance PSF accuracy, leading to a more accurate solution to the neutron image restoration problem. Comparative results with real neutron images indicate the proposed method outperforms ESF, demonstrating improved overall image quality both visually and quantitatively in terms of blind/no reference evaluation (BRISQUE).

Visualization and characterization of agricultural sprays using machine learning based digital inline holography

Accurate characterization of agricultural sprays is crucial to predict in-field performance of liquid applied crop protection products. In this paper, we introduce a robust and efficient machine learning (ML) based Digital In-line Holography (DIH) algorithm to accurately characterize the droplet field for a wide range of commonly used agricultural spray nozzles. Compared to non-ML based DIH processing, the ML-based algorithm enhances accuracy, generalizability, and processing speed. The ML-based approach employs two neural networks: a modified U-Net to obtain the 3D droplet field from the numerically reconstructed optical field, followed by a VGG16 classifier to reduce false positives from the U-Net prediction. The modified U-Net is trained using holograms generated using a single spray nozzle (XR11003) at three different spray locations; center, half-span, and the spray edge to create training data with different number densities and droplet size ranges. VGG16 is trained using the minimum intensity projection of the droplet 3D point spread function (PSF). Data augmentation is used to increase the efficiency of classification and make the algorithm generalizable for different measurement settings. The model is validated using National Institute of Standards and Technology (NIST) traceable glass beads and six different agricultural spray nozzles representing various spray characteristics. The principal results demonstrate a high accuracy rate, with over 90% droplet extraction and less than 5% false positives, regardless of droplet number density and size. Compared to traditional spray measurement techniques, the new ML-based DIH methodology offers a significant leap forward in spatial resolution and generalizability. We show that the proposed ML-based algorithm can extract the real cumulative volume distribution of the NIST beads, where the LD system measurements are biased towards droplets moving at slower speeds. Additionally, the ML-based DIH approach enables the estimation of mass and momentum flux at different locations and the calculation of relative velocities of droplet pairs, which are difficult to obtain using conventional spray characterization techniques.

Improved On-Orbit MTF Measurement Method Based on Point Source Arrays

The modulation transfer function (MTF) is a key characteristic used to assess the performance of optical remote sensing satellite sensors. MTF detection can directly measure a sensor’s two-dimensional (2D) point spread function (PSF); therefore, it has been applied to various high-resolution remote sensing satellites (e.g., Pleiades) using point sources. However, current point source methods mainly use 2D Gaussian functions to fit the discrete digital number (DN) of the point source on the image to extract the center of the point source and fit the PSF after encrypting multiple point sources; thus, noise robustness is poor and measurement accuracy varies widely. In this study, we developed a noise-resistant on-orbit MTF detection method based on the object space constraint among point source arrays. Utilizing object space constraint relationships among points in a point source array, a homography transformation model was established, enabling accurate extraction of sub-pixel coordinates for each point source response. Subsequently, aligning the luminosity distribution of all point sources concerning a reference point source, the encrypted PSF was obtained and then fitted to obtain the MTF. To validate the method, Gaofen-2 (GF-2) satellite images were used to conduct an in-orbit imaging experiment on the point source array of the Chinese Zhongwei remote sensing satellite calibration site. Compared with the Gaussian model methods, the proposed method yielded more accurate peak positions for each point source. Standard deviations of peak position constant ratios in along- and cross-track directions improved by 2.8 and 4.8 times, respectively. The root-mean-square error (RMSE) of the collinearity test results increased by 92%, and the noise resistance of the MTF curve improved by two times. Dynamic MTF values at the Nyquist frequency for the GF-2 panchromatic band in along- and cross-track directions were 0.0476 and 0.0705, respectively, and MTF values in different directions were well distinguished.

Aperture-encoded snapshot hyperspectral imaging with a lensless camera

We present a lensless snapshot hyperspectral camera that is capable of hyperspectral imaging over a broad spectrum using a compact and low-cost hardware configuration. We leverage the multiplexing capability of a lensless camera, a novel type of computational imaging device that replaces the lens with a thin mask. Our device utilizes a linear variable filter and a phase mask to encode spectral information onto a monochromatic image sensor, enabling recovery of hyperspectral image stacks from a single measurement by utilizing spectral information encoded in different parts of the 2D point spread function. We perform spectral calibration using a reference color chart and verify the prototype device’s spectral and spatial resolution, as well as its imaging field of view. We report on the design and construction of the device, the image reconstruction algorithm, and spectral calibration methods and present hyperspectral images ranging from 410 to 800 nm obtained with our prototype device.

Improved design method for suppressing optical focal spots and experimental verification

Pupil filtering is an effective method to modulate the 3D point spread function distribution of an optical system. Based on vector Debye-Wolf diffraction integral and a new fusion optimization algorithm, an improved method is described for efficiently designing the most widely-used concentric multi-belt pupil filters. The main advantage of this method is that it actively selects the minimum belt width and the total belt number for facilitating low-cost batch processing. Pupil filters for visible and near-infrared bands have been separately designed for improving the spatial resolution of optical imaging and ultrafast laser processing. The method has been experimentally demonstrated for suppressing the optical focal spot. The proposed method can be readily modified for designing more sophisticated multi-level pupil filters for practical applications in high-resolution imaging and fine laser micromachining.

Model-based deconvolution for particle analysis applied to a through-focus series of HAADF-STEM images.

This paper presents an approach for determining the sizes and three-dimensional (3D) positions of nanoparticles from a through-focus series of high-angle annular dark-field scanning transmission electron microscopy images. By assuming spherical particles with uniform density, the sizes and 3D positions can be derived via Wiener deconvolution using a series of kernels prepared by the convolution of the 3D point spread function of the electron beam and the 3D density distribution of spheres with different radii. This process is referred to as a model-based deconvolution. Four 3D datasets with a volume size of 148 × 148 × 560 nm3 were obtained from the four sets of 256 high-angle annular dark-field scanning transmission electron microscopy images of 256 × 256 pixels taken from the same field of view under the through-focus condition. The 3D positions and radii of 14 particles in each 3D dataset were derived using the model-based deconvolution for ∼8 min. The observation errors of the 3D position were estimated as σx ≅ σy ≅ 0.3 nm and σz < 1.6 nm.

Pinhole shifting for reducing speckle contrast in reflectance confocal microscopy.

The high speckle contrast in reflectance confocal microscopy is perhaps the most limiting factor on this imaging modality, particularly in high scattering samples such as biological tissues. In this Letter, we propose and numerically analyze a method for speckle reduction that uses simple lateral shifting of the confocal pinhole in several directions, which results in reduced speckle contrast and only a moderate penalty in both lateral and axial resolutions. By simulating free-space electromagnetic wave propagation through a high-numerical-aperture (NA) confocal imaging system, and assuming only single-scattering events, we characterize the 3D point-spread function (PSF) that results from full-aperture pinhole shifting. Simple summation of four different pinhole-shifted images resulted in a 36% reduction in speckle contrast, with reductions of only 17% and 60% in the lateral and axial resolutions, respectively. This method could be particularly useful in noninvasive microscopy for clinical diagnosis, where fluorescence labeling is impractical and high image quality is imperative for achieving accurate diagnosis.

Miniaturized light-field endoscope via a GRIN lens array

Endoscopy provides an intuitive observation of the internal organ structures for surgeons. Most clinical 3D endoscopes generally rely on visual parallax to create a stereoscopic vision, which is incapable of obtaining quantitative depth information for precise diagnosis. Hence a compact probe-type 3D quantitative imaging remains a challenging task for the endoscopic system. In this work, we propose an integrated light-field endoscope that realizes 3D morphology observation with cellular-level resolution from a single-frame data record. In our prototype, a compact customized GRIN lens array (GLA) is used as an objective lens to capture the spatio-angular information directly from object space, as well as maintain consistent spatial measurements. In data processing, we employ a calibration method to estimate the 3D point spread function. After decoupling the spatio-angular information, the 3D morphology of sample can be reconstructed by a deconvolution operation. The system has a lateral resolution of about 15∼40 μm and an axial resolution of 100∼200 μm over an axial operating range of 10 mm. In experiment, we demonstrate the 3D imaging capacity of our platform via fixed structures and living worms.

Axial accuracy in localization microscopy with 3D point spread function engineering.

Single-molecule localization microscopy has developed into a widely used technique to overcome the diffraction limit and enables 3D localization of single-emitters with nanometer precision. A widely used method to enable 3D encoding is to use a cylindrical lens or a phase mask to engineer the point spread function (PSF). The performance of these PSFs is often assessed by comparing the precision they achieve, ignoring accuracy. Nonetheless, accurate localization is required in many applications, such as multi-plane imaging, measuring and modelling of physical processes based on volumetric data, and 3D particle averaging. However, there are PSF model mismatches in the localization schemes due to how reference PSFs are obtained, look-up-tables are created, or spots are fitted. Currently there is little insight in how these model mismatches give rise to systematic axial localization errors, how large these errors are, and how to mitigate them. In this theoretical and simulation work we use a vector PSF model, which incorporates super-critical angle fluorescence (SAF) and the appropriate aplanatic correction factor, to analyze the errors in z-localization. We introduce theory for defining the focal plane in SAF conditions and analyze the predicted axial errors for an astigmatic PSF, double-helix PSF, and saddle-point PSF. These simulations indicate that the absolute axial biases can be as large as 140 nm, 250 nm, and 120 nm for the astigmatic, saddle-point, and double-helix PSF respectively, with relative errors of more than 50%. Finally, we discuss potential experimental methods to verify these findings and propose a workflow to mitigate these effects.

Exploring seismic detection and resolution thresholds of fault zones and gas seeps in the shallow subsurface using seismic modelling

Seismic resolution and illumination issues are sources of challenges in the detailed imaging and detection of subsurface fault architecture and fluid migration. Improved constraints on resolution can provide input into monitoring requirements and detectability of CO2 leakage, and fault-sealing properties during subsurface visualization of migration pathways. This study explores detection and resolution thresholds via synthetic seismic imaging of a detailed shallow normal fault model with realistic fault architecture including sub-seismic structures of damage zones. The base fault model is built from interpretations of high-resolution P-Cable seismic data and is further developed and conditioned by outcrop-based observations and empirical laws for fracture and deformation band distribution. Damage zones of sandstone layers host deformation bands contrary to shale layers with fractures, while mixed lithologies (shaley sandstone and sandy shale) are subjected to a combination of the two deformation mechanisms. We utilize a 2D point-spread function based convolution seismic modelling to produce the synthetic seismic images. Test scenarios include one baseline fault model without a damage zone (M1), and five more advanced/detailed fault models incorporating features known from outcrops (M2-M6; damage zones, an isolated fracture corridor, and gas seeps (CO2) along damage zones of faults and in the fracture corridor). Furthermore, sensitivity analyses on two selected models test the effect of changing the illumination angle and wavelet. The results show that: (1) faults with damage zones have larger disturbances in seismic signals than faults without damage zones, (2) stronger amplitudes are distinguished for models with CO2-filled fractures, (3) a fracture corridor (5 m at it widest) is clearly visible where it crosses horizons bounding horizontal layers, (4) sensitivity tests show good imaging for illumination ≥45°, which is the average dip of the main fault segment, and (5) learnings from fault modelling offer guidance for seismic monitoring.

Deep learning multi-shot 3D localization microscopy using hybrid optical-electronic computing.

Current 3D localization microscopy approaches are fundamentally limited in their ability to image thick, densely labeled specimens. Here, we introduce a hybrid optical-electronic computing approach that jointly optimizes an optical encoder (a set of multiple, simultaneously imaged 3D point spread functions) and an electronic decoder (a neural-network-based localization algorithm) to optimize 3D localization performance under these conditions. With extensive simulations and biological experiments, we demonstrate that our deep-learning-based microscope achieves significantly higher 3D localization accuracy than existing approaches, especially in challenging scenarios with high molecular density over large depth ranges.

Three-dimensional transfer function of optical microscopes in reflection mode.

Three-dimensional (3D) transfer functions build the basis for a comprehensive characterization of optical imaging systems in the spatial frequency domain. Utilizing the projection-slice theorem, the 2D modulation transfer function of an incoherent imaging system can be derived from a 3D transfer function by integration with respect to the axial spatial frequency. For a diffraction limited microscope with homogeneous incoherent pupil illumination, the modulation transfer function equals the 2D autocorrelation function of a circular disc. However, until now to the best of our knowledge no 3D transfer function has been published, which exactly leads to the 2D modulation transfer function of a diffraction limited microscope in reflection mode. In this article, we derive a formula, which after integration with respect to the axial spatial frequency coordinate perfectly fits to the diffraction limited 2D modulation transfer function. The inverse three-dimensional Fourier transform of the 3D transfer function results in a complex-valued 3D point spread function, from which the depth of field, the lateral resolution and, in addition, the corresponding 3D point spread function of both, a conventional and an interference microscope, can beobtained.

Gaia Early Data Release 3

Context. The unprecedented astrometric precision of the Gaia mission relies on accurate estimates of the locations of sources in the Gaia data stream. This is ultimately performed by point spread function (PSF) fitting, which in turn requires an accurate reconstruction of the PSF, including calibrations of all the major dependences. These include a strong colour dependence due to Gaia’s broad G band and a strong time dependence due to the evolving contamination levels and instrument focus. Accurate PSF reconstruction is also important for photometry. Gaia Early Data Release 3 (EDR3) will, for the first time, use a PSF calibration that models several of the strongest dependences, leading to signficantly reduced systematic errors. Aims. We describe the PSF model and calibration pipeline implemented for Gaia EDR3, including an analysis of the calibration results over the 34 months of data. We include a discussion of the limitations of the current pipeline and directions for future releases. This will be of use both to users of Gaia data and as a reference for other precision astrometry missions. Methods. We develop models of the 1D line spread function (LSF) and 2D PSF profiles based on a linear combination of basis components. These are designed for flexibility and performance, as well as to meet several mathematical criteria such as normalisation. We fit the models to selected primary sources in independent time ranges, using simple parameterisations for the colour and other dependences. Variation in time is smoothed by merging the independent calibrations in a square root information filter, with resets at certain mission events that induce a discontinuous change in the PSF. Results. The PSF calibration shows strong time and colour dependences that accurately reproduce the varying state of the Gaia astrometric instrument. Analysis of the residuals reveals both the performance and the limitations of the current models and calibration pipeline, and indicates the directions for future development. Conclusions. The PSF modelling and calibration carried out for Gaia EDR3 represents a major step forwards in the data processing and will lead to reduced systematic errors in the core mission data products. Further significant improvements are expected in the future data releases.

Three-dimensional volumetric deconvolution in coherent optics and holography.

Methods of three-dimensional deconvolution (3DD) or volumetric deconvolution of optical complex-valued wavefronts diffracted by 3D samples with the 3D point spread function are presented. Particularly, the quantitative correctness of the recovered 3D sample distributions is addressed. Samples consisting of point-like objects can be retrieved from their 3D diffracted wavefronts with non-iterative (Wiener filter) 3DD. Continuous extended samples, including complex-valued (phase) samples, can be retrieved with iterative (Gold and Richardson-Lucy) 3DD algorithms. It is shown that quantitatively correct 3D sample distribution can be recovered only with iterative 3DD, and with the optimal protocols provided. It is demonstrated that 3DD can improve the lateral resolution to the resolution limit, and the axial resolution can be at least four times better than the resolution limit. The presented 3DD methods of complex-valued optical fields can be applied for 3D optical imaging and holography.

Three-dimensional deconvolution processing for STEM cryotomography

The complex environment of biological cells and tissues has motivated development of three-dimensional (3D) imaging in both light and electron microscopies. To this end, one of the primary tools in fluorescence microscopy is that of computational deconvolution. Wide-field fluorescence images are often corrupted by haze due to out-of-focus light, i.e., to cross-talk between different object planes as represented in the 3D image. Using prior understanding of the image formation mechanism, it is possible to suppress the cross-talk and reassign the unfocused light to its proper source post facto. Electron tomography based on tilted projections also exhibits a cross-talk between distant planes due to the discrete angular sampling and limited tilt range. By use of a suitably synthesized 3D point spread function, we show here that deconvolution leads to similar improvements in volume data reconstructed from cryoscanning transmission electron tomography (CSTET), namely a dramatic in-plane noise reduction and improved representation of features in the axial dimension. Contrast enhancement is demonstrated first with colloidal gold particles and then in representative cryotomograms of intact cells. Deconvolution of CSTET data collected from the periphery of an intact nucleus revealed partially condensed, extended structures in interphase chromatin.