Light Field Microscopy Research Articles

Axial refocusing precision model with light fields

In light field imaging, axial refocusing precision corresponds to the minimum distance in the axial direction between two distinguishable refocusing planes. The refocusing precision can be essential for applications like light field microscopy. In this paper, we introduce a refocusing precision model based on a geometrical analysis of the flow of rays within the virtual camera. The model establishes the relationship between the feature separability of refocusing and different camera settings. As extending numerical aperture (NA) in classical imaging, the baseline extension of light field also gives more accurate refocusing results. To test the axial refocus precision, we conduct experiments with 1st generation Lytro camera as well as a Blender light field simulation. The results is basically consistent with our prediction. Then, we show that computationally extending the light field baseline increases the axial refocusing precision on real plenoptic camera and light field microscopy datasets.

Machine Learning-Based View Synthesis in Fourier Lightfield Microscopy.

Current interest in Fourier lightfield microscopy is increasing, due to its ability to acquire 3D images of thick dynamic samples. This technique is based on simultaneously capturing, in a single shot, and with a monocular setup, a number of orthographic perspective views of 3D microscopic samples. An essential feature of Fourier lightfield microscopy is that the number of acquired views is low, due to the trade-off relationship existing between the number of views and their corresponding lateral resolution. Therefore, it is important to have a tool for the generation of a high number of synthesized view images, without compromising their lateral resolution. In this context we investigate here the use of a neural radiance field view synthesis method, originally developed for its use with macroscopic scenes acquired with a moving (or an array of static) digital camera(s), for its application to the images acquired with a Fourier lightfield microscope. The results obtained and presented in this paper are analyzed in terms of lateral resolution and of continuous and realistic parallax. We show that, in terms of these requirements, the proposed technique works efficiently in the case of the epi-illumination microscopy mode.

Improving signal-to-background ratio by orders of magnitude in high-speed volumetric imaging in vivo by robust Fourier light field microscopy

Fourier light field microscopy (FLFM) shows great potential in high-speed volumetric imaging of biodynamics. However, due to the inherent disadvantage of wide-field illumination, it suffers from intense background, arising from out of the depth-of-field signal and tissue scattered noise. The background will not only deteriorate the image contrast, making quantitative measurement difficult, but also introduce artifacts, especially in functional imaging of the neuronal network activity in vivo. Here, we propose the robust Fourier light field microscopy (RFLFM), which suppresses the background in FLFM by introducing structured illumination and computational reconstruction based on HiLo. The superior performance of RFLFM is verified by volumetric imaging of biological dynamics in larval zebrafish and mouse in vivo, at a volumetric imaging rate up to 33.3 Hz. The statistical results show that the fluorescence background can be significantly depressed, with the signal-to-background ratio improved by orders of magnitude and the whole image contrast improved by as much as ∼ 10.4 times. Moreover, we stress that, in functional imaging of neuronal network activity in turbid brain tissues, our system can avoid artifacts resulting from background fluctuations, while conventional light field microscopy fails. As a simple but powerful tool, we anticipate our technique to be widely adopted in robust, high-contrast, high-speed volumetric imaging.

Fourier light-field imaging of human organoids with a hybrid point-spread function

Volumetric interrogation of the cellular morphology and dynamic processes of organoid systems with a high spatiotemporal resolution provides critical insights for understanding organogenesis, tissue homeostasis, and organ function. Fluorescence microscopy has emerged as one of the most vital and informative driving forces for probing the cellular complexity in organoid research. However, the underlying scanning mechanism of conventional imaging methods inevitably compromises the time resolution of volumetric acquisition, leading to increased photodamage and inability to capture fast cellular and tissue dynamic processes. Here, we report Fourier light-field microscopy using a hybrid point-spread function (hPSF-FLFM) for fast, volumetric, and high-resolution imaging of entire organoids. hPSF-FLFM transforms conventional 3D microscopy and enables exploration of less accessible spatiotemporally-challenging regimes for organoid research. To validate hPSF-FLFM, we demonstrate 3D imaging of rapid responses to extracellular physical cues such as osmotic and mechanical stresses on human induced pluripotent stem cells-derived colon organoids (hCOs). The system offers cellular (2–3 μm and 5–6 μm in x-y and z, respectively) and millisecond-scale spatiotemporal characterization of whole-organoid dynamic changes that span large imaging volumes (>900 μm × 900 μm × 200 μm in x, y, z, respectively). The hPSF-FLFM method provides a promising avenue to explore spatiotemporal-challenging cellular responses in a wide variety of organoid research.

Fourier lightfield microscopy: a practical design guide.

In this work, a practical guide for the design of a Fourier lightfield microscope is reported. The fundamentals of the Fourier lightfield are presented and condensed on a set of contour plots from which the user can select the design values of the spatial resolution, the field of view, and the depth of field, as function of the specifications of the hardware of the host microscope. This work guides the reader to select the parameters of the infinity-corrected microscope objective, the optical relay lenses, the aperture stop, the microlens array, and the digital camera. A user-friendly graphic calculator is included to ease the design, even to those who are not familiar with the lightfield technology. The guide is aimed to simplify the design process of a Fourier lightfield microscope, which sometimes could be a daunting task, and in this way, to invite the widespread use of this technology. An example of a design and experimental results on imaging different types of samples is also presented.

Light-Field Microscopy for Optical Imaging of Neuronal Activity: When Model-Based Methods Meet Data-Driven Approaches.

Understanding how networks of neurons process information is one of the key challenges in modern neuroscience. A necessary step to achieve this goal is to be able to observe the dynamics of large populations of neurons over a large area of the brain. Light-field microscopy (LFM), a type of scanless microscope, is a particularly attractive candidate for high-speed three-dimensional (3D) imaging. It captures volumetric information in a single snapshot, allowing volumetric imaging at video frame-rates. Specific features of imaging neuronal activity using LFM call for the development of novel machine learning approaches that fully exploit priors embedded in physics and optics models. Signal processing theory and wave-optics theory could play a key role in filling this gap, and contribute to novel computational methods with enhanced interpretability and generalization by integrating model-driven and data-driven approaches. This paper is devoted to a comprehensive survey to state-of-the-art of computational methods for LFM, with a focus on model-based and data-driven approaches.

Photothermal ablation of murine melanomas by Fe3O4 nanoparticle clusters.

Melanoma is one of the deadliest forms of cancer, for which therapeutic regimens are usually limited by the development of resistance. Here, we fabricated Fe3O4 nanoparticle clusters (NPCs), which have drawn widespread attention, and investigated their role in the treatment of melanoma by photothermal therapy (PTT). Scanning electron microscopy imaging shows that our synthesized NPCs are spherical with an average diameter of 329.2 nm. They are highly absorptive at the near-infrared wavelength of 808 nm and efficient at locally converting light into heat. In vitro experiments using light-field microscopy and cell viability assay showed that Fe3O4 NPCs, in conjunction with near-infrared irradiation, effectively ablated A375 melanoma cells by inducing overt apoptosis. Consistently, in vivo studies using BALB/c mice found that intratumoral administration of Fe3O4 NPCs and concomitant in situ exposure to near-infrared light significantly inhibited the growth of implanted tumor xenografts. Finally, we revealed, by experimental approaches including semi-quantitative PCR, western blot and immunohistochemistry, the heat shock protein HSP70 to be upregulated in response to PTT, suggesting this chaperone protein could be a plausible underlying mechanism for the observed therapeutic outcome. Altogether, our results highlight the promise of Fe3O4 NPCs as a new PTT option to treat melanoma.

Handheld and Cost-Effective Fourier Lightfield Microscope.

In this work, the design, building, and testing of the most portable, easy-to-build, robust, handheld, and cost-effective Fourier Lightfield Microscope (FLMic) to date is reported. The FLMic is built by means of a surveillance camera lens and additional off-the-shelf optical elements, resulting in a cost-effective FLMic exhibiting all the regular sought features in lightfield microscopy, such as refocusing and gathering 3D information of samples by means of a single-shot approach. The proposed FLMic features reduced dimensions and light weight, which, combined with its low cost, turn the presented FLMic into a strong candidate for in-field application where 3D imaging capabilities are pursued. The use of cost-effective optical elements has a relatively low impact on the optical performance, regarding the figures dictated by the theory, while its price can be at least 100 times lower than that of a regular FLMic. The system operability is tested in both bright-field and fluorescent modes by imaging a resolution target, a honeybee wing, and a knot of dyed cotton fibers.

Toward implantable devices for angle-sensitive, lens-less, multifluorescent, single-photon lifetime imaging in the brain using Fabry\u2013Perot and absorptive color filters

Implantable image sensors have the potential to revolutionize neuroscience. Due to their small form factor requirements; however, conventional filters and optics cannot be implemented. These limitations obstruct high-resolution imaging of large neural densities. Recent advances in angle-sensitive image sensors and single-photon avalanche diodes have provided a path toward ultrathin lens-less fluorescence imaging, enabling plenoptic sensing by extending sensing capabilities to include photon arrival time and incident angle, thereby providing the opportunity for separability of fluorescence point sources within the context of light-field microscopy (LFM). However, the addition of spectral sensitivity to angle-sensitive LFM reduces imager resolution because each wavelength requires a separate pixel subset. Here, we present a 1024-pixel, 50 µm thick implantable shank-based neural imager with color-filter-grating-based angle-sensitive pixels. This angular-spectral sensitive front end combines a metal–insulator–metal (MIM) Fabry–Perot color filter and diffractive optics to produce the measurement of orthogonal light-field information from two distinct colors within a single photodetector. The result is the ability to add independent color sensing to LFM while doubling the effective pixel density. The implantable imager combines angular-spectral and temporal information to demix and localize multispectral fluorescent targets. In this initial prototype, this is demonstrated with 45 μm diameter fluorescently labeled beads in scattering medium. Fluorescent lifetime imaging is exploited to further aid source separation, in addition to detecting pH through lifetime changes in fluorescent dyes. While these initial fluorescent targets are considerably brighter than fluorescently labeled neurons, further improvements will allow the application of these techniques to in-vivo multifluorescent structural and functional neural imaging.

Volumetric segmentation for integral microscopy with Fourier plane recording

Light Field (LF) microscopy has emerged as a fast-growing field of interest in recent years due to its undoubted capacity of capturing in-vivo samples from multiple perspectives. In this work, we present a framework for Volume Reconstruction from LF images created following the setup of a Fourier Integral Microscope (FIMic). In our approach we do not use real images, instead, we use a dataset generated in Blender which mimics the capturing process of a FIMic. The resulted images have been used to create a Focal Stack (FS) of the LF, from which Epipolar Plane Images (EPIs) have been extracted. The FS and the EPIs have been used to train three different deep neural networks based on the classic U-Net architecture. The Volumetric Reconstruction is the result of the average of the probabilities produced by such networks.

Neuroimaging with light field microscopy: a mini review of imaging systems

Neuroimaging with light field microscopy: a mini review of imaging systems

Shift-Invariant-Subspace Discretization and Volume Reconstruction for Light Field Microscopy

Light Field Microscopy (LFM) is an imaging technique that captures 3D spatial information with a single 2D image. LFM is attractive because of its relatively simple implementation and fast volume acquisition rate. Capturing volume time series at a camera frame rate can enable the study of the behaviour of many biological systems. For instance, it could provide insights into the communication dynamics of living 3D neural networks. However, conventional 3D reconstruction algorithms for LFM typically suffer from high computational cost, low lateral resolution, and reconstruction artifacts. In this work, we study the origin of these issues and propose novel techniques to improve the performance of the reconstruction process. First, we propose a discretization approach that uses shift-invariant subspaces to generalize the typical discretization framework used in LFM. Then, we study the shift-invariant-subspace assumption as a prior for volume reconstruction under ideal conditions. Furthermore, we present a method to reduce the computational time of the forward model by using singular value decomposition (SVD). Finally, we propose to use iterative approaches that incorporate additional priors to perform artifact-free 3D reconstruction from real light field images. We experimentally show that our approach performs better than Richardson-Lucy-based strategies in computational time, image quality, and artifact reduction.

Image Quality Enhancement of 4d Light Field Microscopy Via Reference Image Propagation-Based One-Shot Learning

Image Quality Enhancement of 4d Light Field Microscopy Via Reference Image Propagation-Based One-Shot Learning

Spatial-temporal low-rank prior for low-light volumetric fluorescence imaging

In biological fluorescence imaging, obtaining high spatial-temporal resolution volumetric images under low light conditions is one of the critical requirements. As a widely-used snapshot volumetric imaging modality, light field microscopy has the problem of impeded imaging performance caused by reconstruction artifacts, especially under low light conditions. Fortunately, low-rank prior-based approaches have recently shown great success in image, video and volume denoising. In this paper, we propose an approach based on the spatial-temporal low-rank prior combining weighted nuclear norm minimization (WNNM) denoising and phase-space 3D deconvolution to enhance the performance of light field microscopy (LFM) under low light conditions. We evaluated the method quantitatively through various numerical simulations. Experiments on fluorescence beads and Drosophila larvae were also conducted to show the effectiveness of our approach in biological applications.

Mirror-enhanced scanning light-field microscopy for long-term high-speed 3D imaging with isotropic resolution

Various biological behaviors can only be observed in 3D at high speed over the long term with low phototoxicity. Light-field microscopy (LFM) provides an elegant compact solution to record 3D information in a tomographic manner simultaneously, which can facilitate high photon efficiency. However, LFM still suffers from the missing-cone problem, leading to degraded axial resolution and ringing effects after deconvolution. Here, we propose a mirror-enhanced scanning LFM (MiSLFM) to achieve long-term high-speed 3D imaging at super-resolved axial resolution with a single objective, by fully exploiting the extended depth of field of LFM with a tilted mirror placed below samples. To establish the unique capabilities of MiSLFM, we performed extensive experiments, we observed various organelle interactions and intercellular interactions in different types of photosensitive cells under extremely low light conditions. Moreover, we demonstrated that superior axial resolution facilitates more robust blood cell tracking in zebrafish larvae at high speed.

Computational optical sectioning with an incoherent multiscale scattering model for light-field microscopy

Quantitative volumetric fluorescence imaging at high speed across a long term is vital to understand various cellular and subcellular behaviors in living organisms. Light-field microscopy provides a compact computational solution by imaging the entire volume in a tomographic way, while facing severe degradation in scattering tissue or densely-labelled samples. To address this problem, we propose an incoherent multiscale scattering model in a complete space for quantitative 3D reconstruction in complicated environments, which is called computational optical sectioning. Without the requirement of any hardware modifications, our method can be generally applied to different light-field schemes with reduction in background fluorescence, reconstruction artifacts, and computational costs, facilitating more practical applications of LFM in a broad community. We validate the superior performance by imaging various biological dynamics in Drosophila embryos, zebrafish larvae, and mice.

The Lightfield Microscope Eyepiece.

Lightfield microscopy has raised growing interest in the last few years. Its ability to get three-dimensional information about the sample in a single shot makes it suitable for many applications in which time resolution is fundamental. In this paper we present a novel device, which is capable of converting any conventional microscope into a lightfield microscope. Based on the Fourier integral microscope concept, we designed the lightfield microscope eyepiece. This is coupled to the eyepiece port, to let the user exploit all the host microscope’s components (objective turret, illumination systems, translation stage, etc.) and get a 3D reconstruction of the sample. After the optical design, a proof-of-concept device was built with off-the-shelf optomechanical components. Here, its optical performances are demonstrated, which show good matching with the theoretical ones. Then, the pictures of different samples taken with the lightfield eyepiece are shown, along with the corresponding reconstructions. We demonstrated the functioning of the lightfield eyepiece and lay the foundation for the development of a commercial device that works with any microscope.

Identification of pollen taxa by different microscopy techniques.

Melissopalynology is an important analytical method to identify botanical origin of honey. Pollen grain recognition is commonly performed by visual inspection by a trained person. An alternative method for visual inspection is automated pollen analysis based on the image analysis technique. Image analysis transfers visual information to mathematical descriptions. In this work, the suitability of three microscopic techniques for automatic analysis of pollen grains was studied. 2D and 3D morphological characteristics, textural and colour features, and extended depth of focus characteristics were used for the pollen discrimination. In this study, 7 botanical taxa and a total of 2482 pollen grains were evaluated. The highest correct classification rate of 93.05% was achieved using the phase contrast microscopy, followed by the dark field microscopy reaching 91.02%, and finally by the light field microscopy reaching 88.88%. The most significant discriminant characteristics were morphological (2D and 3D) and colour characteristics. Our results confirm the potential of using automatic pollen analysis to discriminate pollen taxa in honey. This work provides the basis for further research where the taxa dataset will be increased, and new descriptors will be studied.

High-speed large-scale 4D activities mapping of moving C. elegans by deep-learning-enabled light-field microscopy on a chip

We report a novel fusion of microfluidics and light-field microscopy, to achieve high-speed 4D (space + time) imaging of moving C. elegans on a chip. Our approach combines automatic chip-based worm loading/compartmentalization/flushing/reloading with instantaneous deep-learning light-field imaging of behaving worm. Taken together, we realized intoto image-based screening of wild-type and uncoordinated-type worms at a volumetric imaging rate of 33 Hz, with sustained observation of 1 worm per minute and totally 42 worms per hour. Through four-dimensionally visualizing the dynamics of all the worms at an ultra-high throughput of 84000 image volumes in an hour, we can quantitatively analyze their behaviors as well as the neural activities, and correlate the phenotypes with the neuron functions. The different types of worms can be readily identified as a result of the high-throughput activity mapping. Our approach shows great potential for various lab-on-a-chip biological studies, such as high-throughput embryo sorting and cell growth assays.

DiLFM: an artifact-suppressed and noise-robust light-field microscopy through dictionary learning

Light field microscopy (LFM) has been widely used for recording 3D biological dynamics at camera frame rate. However, LFM suffers from artifact contaminations due to the illness of the reconstruction problem via naïve Richardson–Lucy (RL) deconvolution. Moreover, the performance of LFM significantly dropped in low-light conditions due to the absence of sample priors. In this paper, we thoroughly analyze different kinds of artifacts and present a new LFM technique termed dictionary LFM (DiLFM) that substantially suppresses various kinds of reconstruction artifacts and improves the noise robustness with an over-complete dictionary. We demonstrate artifact-suppressed reconstructions in scattering samples such as Drosophila embryos and brains. Furthermore, we show our DiLFM can achieve robust blood cell counting in noisy conditions by imaging blood cell dynamic at 100 Hz and unveil more neurons in whole-brain calcium recording of zebrafish with low illumination power in vivo.