Multifocal Plane Microscopy Research Articles

Fast and robust multiplane single-molecule localization microscopy using a deep neural network

Single-molecule localization microscopy is a widely used technique in biological research for measuring the nanostructures of samples smaller than the diffraction limit. This study uses multifocal plane microscopy and addresses the three-dimensional (3D) single-molecule localization problem, where lateral and axial locations of molecules are estimated. However, when multifocal plane microscopy is used, the estimation accuracy of 3D localization is easily deteriorated by the small lateral drifts of camera positions. A 3D molecule localization problem was presented along with the lateral drift estimation as a compressed sensing problem. A deep neural network (DNN) was applied to solve this problem accurately and efficiently. The results show that the proposed method is robust to lateral drift and achieves an accuracy of 20 nm laterally and 50 nm axially without an explicit drift correction.

Single-molecule localization by voxel-wise regression using convolutional neural network

Single-molecule localization microscopy is widely used in biological research for measuring the nanostructures of samples smaller than the diffraction limit. In this paper, a novel method for regression of the coordinates of molecules for multifocal plane microscopy is presented. A regression problem for the target space is decomposed into regression problems for small subsets of the target space. Then, a deep neural network is used to solve these problems. By decomposing the regression problem, a fully convolutional neural network can be used to solve the regression problems. The computation of the network is efficient, and a simple and parameter-free loss function can be used to train the network. The proposed algorithm is validated by both simulated and real data obtained by quad-plane microscopy.

Fast-tracking of single emitters in large volumes with nanometer precision.

Multifocal plane microscopy allows for capturing images at different focal planes simultaneously. Using a proprietary prism which splits the emitted light into paths of different lengths, images at 8 different focal depths were obtained, covering a volume of 50x50x4 µm3. The position of single emitters was retrieved using a phasor-based approach across the different imaging planes, with better than 10 nm precision in the axial direction. We validated the accuracy of this approach by tracking fluorescent beads in 3D to calculate water viscosity. The fast acquisition rate (>100 fps) also enabled us to follow the capturing of 0.2 µm fluorescent beads into an optical trap.

Remote focusing multifocal plane microscopy for the imaging of 3D single molecule dynamics with cellular context.

Three-dimensional (3D) single molecule fluorescence microscopy affords the ability to investigate subcellular traffcking at the level of individual molecules. An imaged single molecule trajectory, however, often reveals only limited information about the underlying biological process when insuffcient information is available about the organelles and other cellular structures with which the molecule interacts. A new 3D fluorescence microscopy imaging modality is described here that enables the simultaneous imaging of the trajectories of fast-moving molecules and the associated cellular context. The new modality is called remote focusing multifocal plane microscopy (rMUM), as it extends multifocal plane microscopy (MUM) with a remote focusing module. MUM is a modality that uses multiple detectors to image distinct focal planes within the specimen at the same time, and it has been demonstrated to allow the determination of 3D single molecule trajectories with high accuracy. Remote focusing is a method that makes use of two additional objective lenses to enable the acquisition of a z-stack of the specimen without having to move the microscope's objective lens or sample stage, components which are required by MUM to be fixed in place. rMUM's remote focusing module thus allows the cellular context to be imaged in the form of z-stacks as the trajectories of molecules or other objects of interest are imaged by MUM. In addition to a description of the modality, a discussion of rMUM data analysis and an example of data acquired using an rMUM setup are provided in this paper.

Intensity-based axial localization approaches for multifocal plane microscopy.

Multifocal plane microscopy (MUM) can be used to visualize biological samples in three dimensions over large axial depths and provides for the high axial localization accuracy that is needed in applications such as the three-dimensional tracking of single particles and super-resolution microscopy. This report analyzes the performance of intensity-based axial localization approaches as applied to MUM data using Fisher information calculations. In addition, a new non-parametric intensity-based axial location estimation method, Multi-Intensity Lookup Algorithm (MILA), is introduced that, unlike typical intensity-based methods that make use of a single intensity value per data image, utilizes multiple intensity values per data image in determining the axial location of a point source. MILA is shown to be robust against potential bias induced by differences in the sub-pixel location of the imaged point source. The method's effectiveness on experimental data is also evaluated.

Imaging of Three-Dimensional Single Molecule Dynamics in their Cellular Context

Single molecule microscopy has provided a wealth of information regarding the dynamics of receptors on the plasma membrane. In contrast, due to lack of appropriate imaging approaches, very little is known about three-dimensional (3D) single molecule trafficking in subcellular compartments. There are two major technical problems in imaging 3D subcellular single molecule dynamics. First, the dynamics of the single molecules are not confined to one focal plane of the microscope and second, having only the trajectories available of single molecules is typically not meaningful unless the cellular context of the single molecule dynamics is also imaged. For example, only by the simultaneous imaging of the single molecule dynamics and the cellular context can, for example, the diffusion of a receptor be analyzed on a non-stationary endosome. Here we introduce a novel microscopy imaging modality that is designed to simultaneously image both the dynamics of single molecules in three dimensions and the cellular context which may labeled with several fluorophores. Imaging of the single molecule dynamics is carried out using multi-focal plane microscopy (MUM), a previously demonstrated and powerful tool for this task. The cellular context is imaged using a remote focusing approach that is highly compatible with MUM imaging. The efficacy of this new imaging modality, called rMUM for remote focusing multifocal plane microscopy, is demonstrated by an investigation of the trafficking pathways of antibodies against the prostate-specific membrane antigen (PSMA) in prostate cancer cells. We demonstrate that novel experiments such as a single molecule diffusion study on sorting endosomes are possible with the new microscope, and we completely characterize the pathway of PSMA-specific antibodies, starting with diffusion towards the plasma membrane, followed by endocytosis, transport to sorting endosomes, and subsequent diffusion on sorting endosomes, and ending with either recycling or maturation into a multivesicular body.

Tracking Subcellular Dynamics with Multifocal Plane Microscopy

The study of cellular dynamics has often been hampered by the limited capabilities of standard microscopes. Classical microscopes are very well suited to the imaging of events that are constrained to one focal plane, but much less so for the investigation of fast dynamics that are not constrained to one focal plane. As cells are inherently three dimensional, this might account for the relatively limited amount of knowledge that is currently available regarding the specifics of subcellular trafficking.To address this problem we have introduced a microscopy modality into the study of cellular dynamics, multi-focal plane microscopy (MUM), which allows for the simultaneous imaging of several focal planes. We will review multifocal plane microscopy and discuss subsequent developments. Multifocal plane microscopy is an ideal tool for single molecule and single particle tracking in three dimensions. Single molecule trajectories by themselves, are, however, only of limited biological significance. It is the cellular environment that also needs to be imaged to provide the biological context for the observed dynamics. We will discuss our approaches to address this question.Antibodies form a central component of the human immune system. Over recent years, antibodies, especially IgG molecules, have also been developed as major new drugs. We will present data related to the trafficking of IgG molecules during endocytosis, exocytosis and sorting as imaged by multifocal plane microscopy.

Abstract PR08: Macrophage-mediated trogocytosis leads to death of antibody-opsonized tumor cells

Abstract The interplay between innate immune cells and antibodies to control cancerous growth is of central importance for mediating direct anti-tumor effects and the induction of long-lived anti-tumor immunity. Phagocytosis of antibody-opsonized cells by macrophages through Fcgamma receptor interactions is well characterized and leads to cell death through target cell engulfment into phagosomes. By contrast, the contribution of the related process called trogocytosis, involving the ingestion of limited amounts of the target cell by phagocytic cells, to cell attrition is less certain. In the current study we have developed a high throughput, quantitative assay to distinguish whole cell phagocytosis (WCP) and trogocytosis for different macrophage:breast cancer cell combinations. These analyses, combined with long term microscopic imaging, demonstrate that trogocytosis leads to target cell death. The implementation of multifocal plane microscopy to investigate trogocytic events at high spatiotemporal resolution reveals that these processes involve tubular extensions of opsonized tumor cells that are subsequently pinched off by the recipient macrophage. Of relevance to the design of second generation antibodies with improved efficacy, antibody engineering to enhance Fcgamma receptor interactions results in increased trogocytic activity. Collectively, these studies have significance to the rapidly expanding use of antibodies to treat cancer. Citation Format: Ramraj Velmurugan, Dilip K. Challa, Sripad Ram, Raimund J. Ober, E. Sally Ward. Macrophage-mediated trogocytosis leads to death of antibody-opsonized tumor cells. [abstract]. In: Proceedings of the Fourth AACR International Conference on Frontiers in Basic Cancer Research; 2015 Oct 23-26; Philadelphia, PA. Philadelphia (PA): AACR; Cancer Res 2016;76(3 Suppl):Abstract nr PR08.

Abstract B089: Macrophage-mediated trogocytosis leads to death of antibody-opsonized tumor cells

Abstract The interplay between innate immune cells and antibodies to control cancerous growth is of central importance for mediating direct anti-tumor effects and the induction of long-lived anti-tumor immunity. Phagocytosis of antibody-opsonized cells by macrophages through Fc gamma receptor interactions is well characterized and leads to cell death through target cell engulfment into phagosomes. By contrast, the contribution of the related process called trogocytosis, involving the ingestion of limited amounts of the target cell by phagocytic cells, to cell attrition is less certain. In the current study we have developed a high throughput, quantitative assay to distinguish whole cell phagocytosis and trogocytosis for different macrophage:breast cancer cell combinations. These analyses, combined with long term microscopic imaging, demonstrate that trogocytosis leads to death of HER2-overexpressing breast tumor cells. The implementation of multifocal plane microscopy to investigate trogocytic events at high spatiotemporal resolution reveals that these processes involve tubular extensions of opsonized tumor cells that are subsequently pinched off by the recipient macrophage. Of relevance to the design of second generation antibodies with improved efficacy, antibody engineering to enhance Fc gamma receptor interactions results in increased trogocytic activity. Collectively, these studies have significance to the rapidly expanding use of antibodies to treat cancer. This research was supported in part by grants from the Cancer Prevention and Research Institute of Texas (CPRIT, RP 110069) and the National Institutes of Health (R01 GM85575). Citation Format: Ramraj Velmurugan, Dilip Challa, Sripad Ram, Raimund Ober, E. Sally Ward. Macrophage-mediated trogocytosis leads to death of antibody-opsonized tumor cells. [abstract]. In: Proceedings of the CRI-CIMT-EATI-AACR Inaugural International Cancer Immunotherapy Conference: Translating Science into Survival; September 16-19, 2015; New York, NY. Philadelphia (PA): AACR; Cancer Immunol Res 2016;4(1 Suppl):Abstract nr B089.

An information-theoretic approach to designing the plane spacing for multifocal plane microscopy.

Multifocal plane microscopy (MUM) is a 3D imaging modality which enables the localization and tracking of single molecules at high spatial and temporal resolution by simultaneously imaging distinct focal planes within the sample. MUM overcomes the depth discrimination problem of conventional microscopy and allows high accuracy localization of a single molecule in 3D along the z-axis. An important question in the design of MUM experiments concerns the appropriate number of focal planes and their spacings to achieve the best possible 3D localization accuracy along the z-axis. Ideally, it is desired to obtain a 3D localization accuracy that is uniform over a large depth and has small numerical values, which guarantee that the single molecule is continuously detectable. Here, we address this concern by developing a plane spacing design strategy based on the Fisher information. In particular, we analyze the Fisher information matrix for the 3D localization problem along the z-axis and propose spacing scenarios termed the strong coupling and the weak coupling spacings, which provide appropriate 3D localization accuracies. Using these spacing scenarios, we investigate the detectability of the single molecule along the z-axis and study the effect of changing the number of focal planes on the 3D localization accuracy. We further review a software module we recently introduced, the MUMDesignTool, that helps to design the plane spacings for a MUM setup.

Temporal resolution in fluorescence imaging.

Temporal resolution is a key factor for imaging rapidly occurring events in biology. In this feature article, I investigate an approximate estimate for determining the temporal resolution limit. The condition that led to this limit is, the time taken by the ensemble (99.9%) of excited molecules to relax to ground state, assuming all the emitted photons are detected. In a simplistic three-level system, the temporal resolution is, ≈3τp, where τp = (loge10)/(kf + knr) and, kf and knr are respectively the radiative and non-radiative emission rates. This further assumes the ideal condition that, the quantum efficiency of the detector is unity and there are no other loses. We discuss few state-of-art microscopy techniques that are capable of high temporal resolution. This includes techniques such as multifocal multiphoton microscopy (MMM), multifocal plane microscopy, multiple excitation spot optical microscopy (MESO), multiplane microscopy and multiple light-sheet microscopy (MLSM).

Designing the focal plane spacing for multifocal plane microscopy.

Multifocal plane microscopy (MUM) has made it possible to study subcellular dynamics in 3D at high temporal and spatial resolution by simultaneously imaging distinct planes within the specimen. MUM allows high accuracy localization of a point source along the z-axis since it overcomes the depth discrimination problem of conventional single plane microscopy. An important question in MUM experiments is how the number of focal planes and their spacings should be chosen to achieve the best possible localization accuracy along the z-axis. Here, we propose approaches based on the Fisher information matrix and report spacing scenarios called strong coupling and weak coupling which yield an appropriate 3D localization accuracy. We examine the effect of numerical aperture, magnification, photon count, emission wavelength and extraneous noise on the spacing scenarios. In addition, we investigate the effect of changing the number of focal planes on the 3D localization accuracy. We also introduce a new software package that provides a user-friendly framework to find appropriate plane spacings for a MUM setup. These developments should assist in optimizing MUM experiments.

Fast 3D Single Molecule Tracking with Multifocal Plane Microscopy in Polarized Epithelia Reveals a Novel Cellular Process of Intercellular Transfer

The study of intracellular trafficking processes represents a fundamental problem in many areas of biomedical research. Single molecule imaging approaches are well suited to study heterogeneous processes in live cells. However, 3D single molecule imaging of intracellular trafficking events in a thick sample such as an epithelial-cell monolayer poses several technical challenges. Specifically, we require a methodology that not only enables fast 3D tracking of single molecules across a cell monolayer, but also enables the imaging of the cellular environment with which the single molecule interacts. Current approaches that are widely used for 3D single molecule imaging/tracking are not well suited for studying the intracellular trafficking pathways due to restricted imaging depth, poor temporal resolution, and the ability to track only a few molecules at one time. Here we show that multifocal plane microscopy (MUM), a 3D imaging modality developed by our group, provides the much needed solution to this longstanding problem by demonstrating fast 3D tracking of quantum dot labeled transferrin molecules in a ∼10 micron thick epithelial cell monolayer.The use of MUM led to the unexpected discovery of a novel cellular process, intercellular transfer, that involves the rapid exchange of Tf molecules between two adjacent cells in the monolayer. We also report 3D single molecule tracking of endocytosis and exocytosis at the lateral plasma membrane of cells in the monolayer. This lateral membrane has been notoriously difficult to image with other cellular imaging modalities. A detailed characterization of these events based on the temporal and 3D intracellular spatial behavior of Tf molecules has been made. The methods and approaches used in this study have broad applicability to investigate 3D trafficking pathways in other cell systems and models.

Using multifocal plane microscopy to reveal novel trafficking processes in the recycling pathway

A major outstanding issue in cell biology is the lack of understanding of the contribution of tubulovesicular transport carriers (TCs) to intracellular trafficking pathways within 3D cellular environments. This is primarily due to the challenges associated with the use of microscopy techniques to track these highly motile, small compartments. In the present study we have used multifocal plane microscopy with localized photoactivation to overcome these limitations. Using this approach, we have characterized individual components constituting the recycling pathway of the receptor FcRn. Specifically, several different pathways followed by TCs that intersect with larger, relatively static sorting endosomes have been defined. These pathways include a novel 'looping' process in which TCs leave and return to the same sorting endosome. Significantly, TCs with different itineraries can be identified by associations with distinct complements of Rab GTPases, APPL1 and SNX4. These studies provide a framework for further analyses of the recycling pathway.

3D Single Molecule Tracking with Multifocal Plane Microscopy Reveals Rapid Intercellular Transferrin Transport at Epithelial Cell Barriers

The study of intracellular transport pathways at epithelial cell barriers that line diverse tissue sites is fundamental to understanding tissue homeostasis. A major impediment to investigating such processes at the subcellular level has been the lack of imaging approaches that support fast three-dimensional (3D) tracking of cellular dynamics in thick cellular specimens. Here, we report significant advances in multifocal plane microscopy and demonstrate 3D single molecule tracking of rapid protein dynamics in a 10 micron thick live epithelial cell monolayer. We have investigated the transferrin receptor (TfR) pathway, which is not only essential for iron delivery but is also of importance for targeted drug delivery across cellular barriers at specific body sites, such as the brain that is impermeable to blood-borne substances. Using multifocal plane microscopy, we have discovered a cellular process of intercellular transfer involving rapid exchange of Tf molecules between two adjacent cells in the monolayer. Furthermore, 3D tracking of Tf molecules at the lateral plasma membrane has led to the identification of different modes of endocytosis and exocytosis, which exhibit distinct temporal and intracellular spatial trajectories. These results reveal the complexity of the 3D trafficking pathways in epithelial cell barriers. The methods and approaches reported here can enable the study of fast 3D cellular dynamics in other cell systems and models, and underscore the importance of developing advanced imaging technologies to study such processes.

3D single molecule tracking of rapid intracellular trafficking imaged by multifocal plane microscopy

Extended abstract of a paper presented at Microscopy and Microanalysis 2012 in Phoenix, Arizona, USA, July 29 – August 2, 2012.

3D Single Molecule Tracking in Thick Cellular Samples using Multifocal Plane Microscopy

Protein transport across a cell monolayer represents a fundamental cellular process in cell biology. However, the imaging of this process poses several challenges. Protein transport is highly dynamic occurring across the entire cell-monolayer, which is several tens of microns thick, and over widely varying time scales. Additional challenges arise if 3D single molecule tracking is also desired that are particularly related to imaging single molecules for extended periods of time and across varying depths in a thick sample. Approaches that use a focusing device are not well suited for such applications as they are typically slow and suffer from poor z-localization/3D-tracking capability. To date the 3D spatial-temporal dynamics of many biological processes are poorly understood, as there is a lack of an appropriate methodology that supports 3D single-molecule tracking in thick samples.Here we show that multifocal plane microscopy (MUM) [1-2] provides the much needed solution to this longstanding problem. While MUM has been previously used for 3D single molecule tracking across shallow depths (1 micron) in live cells [2], the question arises if MUM can also live up to the significant challenge of tracking single molecules in a thick sample. By substantially expanding the capabilities of MUM, here we demonstrate 3D tracking of quantum-dot labeled single molecules in a ∼10 micron thick live-cell monolayer, for extended periods of time (minutes). We have implemented a 4-plane MUM setup with increased focal plane spacing and a novel multicolor imaging approach, which enables the imaging of the molecule of interest as well as the cellular environment with which it interacts. Using this new methodology we have reconstructed the complete 3D trafficking itinerary of single molecules at high spatial and temporal resolution.1. PNAS, 2007, 104:5889-5894.2. Biophys J., 2008, 95:6025-6043.

1N1412 生細胞における立体的分子イメージングと定量のための複数観察焦点面顕微鏡(バイオイメージング1,第49回日本生物物理学会年会)

1N1412 生細胞における立体的分子イメージングと定量のための複数観察焦点面顕微鏡(バイオイメージング1,第49回日本生物物理学会年会)

Comparison of Three-Dimensional Imaging Configurations for High Resolution Microscopy Measurements

In recent years, a class of fluorescence microscopy imaging techniques has emerged which enables the imaging of single fluorophores at high resolution by reducing the problem of resolution to one of localization. The photoactivated localization microscopy (PALM) technique, for example, constructs finely resolved images by way of accurately localizing closely spaced fluorophores that are detected separately in time by successively photoactivating small and stochastically different subsets of fluorophores.Due to the optical microscope's poor depth discrimination capability, the resolution of three-dimensional (3D) versions of techniques like PALM is limited by the z-localization accuracy of a single fluorophore, which can be especially poor when a fluorophore is near-focus. An imaging technique that overcomes the near-focus problem is multifocal plane microscopy (MUM) (Prabhat, P. et. al., IEEE Trans. Nanobiosci., 2004), which allows the simultaneous imaging of a fluorophore from distinct focal planes. Images from multiple focal planes enable MUM to accurately localize a near-focus fluorophore (Ram, S. et. al., Proc. SPIE, 64430D1, 2007) and to support high accuracy 3D localization over a wide depth range.Here we compare 3D fluorescence imaging configurations which employ different combinations of conventional excitation, PALM excitation, conventional emission, and MUM emission. Using a Cramer-Rao lower bound-based 3D resolution measure (Chao, J. et. al., Opt. Commun., 2009), comparisons are made in terms of the accuracy with which the distance separating two closely spaced fluorophores can be estimated. Such distance information can be important as it can help to characterize the interaction between two biomolecules. Our results show that configurations incorporating PALM excitation provide superior distance estimation accuracies for fluorophore pairs characterized by small distances of separation and orientations near parallel to the optical axis. Meanwhile, configurations incorporating MUM emission provide the best accuracies for near-focus fluorophore pairs.

Improved single particle localization accuracy with dual objective multifocal plane microscopy

In single particle imaging applications, the number of photons detected from the fluorescent label plays a crucial role in the quantitative analysis of the acquired data. For example, in tracking experiments the localization accuracy of the labeled entity can be improved by collecting more photons from the labeled entity. Here, we report the development of dual objective multifocal plane microscopy (dMUM) for single particle studies. The new microscope configuration uses two opposing objective lenses, where one of the objectives is in an inverted position and the other objective is in an upright position. We show that dMUM has a higher photon collection efficiency when compared to standard microscopes. We demonstrate that fluorescent labels can be localized with better accuracy in 2D and 3D when imaged through dMUM than when imaged through a standard microscope. Analytical tools are introduced to estimate the nanoprobe location from dMUM images and to characterize the accuracy with which they can be determined.