Real-time Fluorescence Lifetime Imaging Research Articles

Coupling a recurrent neural network to SPAD TCSPC systems for real-time fluorescence lifetime imaging

Fluorescence lifetime imaging (FLI) has been receiving increased attention in recent years as a powerful diagnostic technique in biological and medical research. However, existing FLI systems often suffer from a tradeoff between processing speed, accuracy, and robustness. Inspired by the concept of Edge Artificial Intelligence (Edge AI), we propose a robust approach that enables fast FLI with no degradation of accuracy. This approach couples a recurrent neural network (RNN), which is trained to estimate the fluorescence lifetime directly from raw timestamps without building histograms, to SPAD TCSPC systems, thereby drastically reducing transfer data volumes and hardware resource utilization, and enabling real-time FLI acquisition. We train two variants of the RNN on a synthetic dataset and compare the results to those obtained using center-of-mass method (CMM) and least squares fitting (LS fitting). Results demonstrate that two RNN variants, gated recurrent unit (GRU) and long short-term memory (LSTM), are comparable to CMM and LS fitting in terms of accuracy, while outperforming them in the presence of background noise by a large margin. To explore the ultimate limits of the approach, we derive the Cramer-Rao lower bound of the measurement, showing that RNN yields lifetime estimations with near-optimal precision. To demonstrate real-time operation, we build a FLI microscope based on an existing SPAD TCSPC system comprising a 32×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes $$\\end{document}32 SPAD sensor named Piccolo. Four quantized GRU cores, capable of processing up to 4 million photons per second, are deployed on the Xilinx Kintex-7 FPGA that controls the Piccolo. Powered by the GRU, the FLI setup can retrieve real-time fluorescence lifetime images at up to 10 frames per second. The proposed FLI system is promising and ideally suited for biomedical applications, including biological imaging, biomedical diagnostics, and fluorescence-assisted surgery, etc.

Simple and Robust Deep Learning Approach for Fast Fluorescence Lifetime Imaging.

Fluorescence lifetime imaging (FLIM) is a powerful tool that provides unique quantitative information for biomedical research. In this study, we propose a multi-layer-perceptron-based mixer (MLP-Mixer) deep learning (DL) algorithm named FLIM-MLP-Mixer for fast and robust FLIM analysis. The FLIM-MLP-Mixer has a simple network architecture yet a powerful learning ability from data. Compared with the traditional fitting and previously reported DL methods, the FLIM-MLP-Mixer shows superior performance in terms of accuracy and calculation speed, which has been validated using both synthetic and experimental data. All results indicate that our proposed method is well suited for accurately estimating lifetime parameters from measured fluorescence histograms, and it has great potential in various real-time FLIM applications.

Smart Wide-field Fluorescence Lifetime Imaging System with CMOS Single-photon Avalanche Diode Arrays.

Wide-field fluorescence lifetime imaging (FLIM) is a promising technique for biomedical and clinic applications. Integrating with CMOS single-photon avalanche diode (SPAD) sensor arrays can lead to cheaper and portable real-time FLIM systems. However, the FLIM data obtained by such sensor systems often have sophisticated noise features. There is still a lack of fast tools to recover lifetime parameters from highly noise-corrupted fluorescence signals efficiently. This paper proposes a smart wide-field FLIM system containing a 192×128 COMS SPAD sensor and a field-programmable gate array (FPGA) embedded deep learning (DL) FLIM processor. The processor adopts a hardware-friendly and light-weighted neural network for fluorescence lifetime analysis, showing the advantages of high accuracy against noise, fast speed, and low power consumption. Experimental results demonstrate the proposed system's superior and robust performances, promising for many FLIM applications such as FLIM-guided clinical surgeries, cancer diagnosis, and biomedical imaging.

One-Dimensional Deep Learning Architecture for Fast Fluorescence Lifetime Imaging

We present a hardware-friendly deep learning architecture with one-dimensional convolutional neural networks (1D CNN) for fast analyzing fluorescence lifetime imaging (FLIM) data. A 1D CNN shows unparalleled advantages; they are more straightforward, quicker to train, and faster than high dimensional CNNs. 1D CNNs can be easily applied to multi-exponential fluorescence decay models. Compared with traditional least-square methods, superior performances of 1D CNNs on fluorescence lifetime image reconstruction have been validated using simulated data. We also employ the proposed 1D CNN to analyze two-photon FLIM images of functionalized gold nanoprobes in Hek293 and human prostate cancer cells. The results further demonstrate that 1D CNNs are fast and can accurately extract lifetime parameters from fluorescence signals. Our study shows that 1D CNNs have great potential in various real-time FLIM applications.

A coumarin-based TICT fluorescent probe for real-time fluorescence lifetime imaging of mitochondrial viscosity and systemic inflammation in vivo.

Systemic inflammation, linked with abnormal mitochondrial viscosity, is reported to be associated with cerebro-cardiovascular disease and Alzheimer's disease. Therefore, it is of great significance to detect the mitochondrial viscosity to indicate the inflammatory signal in vivo. Considering the strategies of fluorescent molecular rotors (FMRs) and fluorescence lifetime imaging microscopy (FLIM), we have rationally designed a novel mitochondrial viscosity-specific fluorescent probe Mito-VCI, based on coumarin fluorophores with benzo[e]indolium as the rotor group. In a high viscosity solution system, the fluorescence lifetime of the probe Mito-VCI was prolonged due to the planarization and rigidity enhancement of the molecular rotor. Satisfactorily, the probe was only sensitive to viscosity, instead of non-viscosity factors such as pH and polarity. Furthermore, the probe sensitively targeted mitochondria in HeLa cells with a Pearson's correlation of 0.93, and specifically detected dynamics variation of mitochondrial viscosity with FLIM imaging in HeLa cells induced by LPS. Notably, significant fluorescence lifetime changes of Mito-VCI between normal and inflammatory tissues also occurred (for example, the fluorescence lifetime in the spleen changed from 1.128 to 1.432 ns). It can be inferred from the above observations that Mito-VCI could work as an effective and sensitive fluorescent molecular rotor for mitochondrial viscosity monitoring through FLIM imaging with a systemic inflammatory response, and provide potential applications for the diagnosis of systemic inflammation in pharmacology and toxicology studies.

Towards real-time wide-field fluorescence lifetime imaging of 5-ALA labeled brain tumors with multi-tap CMOS cameras.

Fluorescence guided neurosurgery based on 5-aminolevulinic acid (5-ALA) has significantly increased maximal safe resections. Fluorescence lifetime imaging (FLIM) of 5-ALA could further boost this development by its increased sensitivity. However, neurosurgeons require real-time visual feedback which was so far limited in dual-tap CMOS camera based FLIM. By optimizing the number of phase frames required for reconstruction, we here demonstrate real-time 5-ALA FLIM of human high- and low-grade glioma with up to 12 Hz imaging rate over a wide field of view (11.0 x 11.0 mm). Compared to conventional fluorescence imaging, real-time FLIM offers enhanced contrast of weakly fluorescent tissue.

Real-time Fluorescence Lifetime Imaging Microscopy Implementation by Analog Mean-Delay Method through Parallel Data Processing

Fluorescence lifetime imaging microscopy (FLIM), especially based on the most accurate and frequently used technique called time-correlated single photon counting (TCSPC), has been widely employed for it gives not only morphological information but also chemical information. Chemical properties such as pH value (Hanson et al., 2002), ion or oxygen concentration (Gilbert et al., 2007), molecular dynamics and even the disease progression (Park et al., 2012) can be investigated and studied by TCSPC-FLIM. This powerful traits of FLIM has led many researchers and developers to include FLIM into their imaging system. Even though TCSPC-FLIM technique gives precise and accurate measurement of fl uorescence decay curves, its core technology, called single photon counting (SPC), inherently limits the system from high speed imaging. In the case of the state-of-art single channel TCSPC-FLIM, more than one minute is required to produce an image of 512 by 512 pixels with 10% accuracy. The acquisition time could be lengthened up to five minutes for 3% accuracy (Kollner & Wolfrum, 1992). To overcome such drawback of the long acquisition time of TCSPC-FLIM, analog mean-delay FLIM (AMD-FLIM) was recently developed (Moon et al., 2009; Won et al., 2009). Rather than using the stochastic reconstruction used in SPC techniques, AMD method extracts a decay constant directly by subtracting the mean-delay time (or mean-arrival time) of reflected photon flux from that of fluorescence photon fl ux. The total amount of alterations in the mean-delay time

Visualization of Cu2+ uptake and release in plant cells by fluorescence lifetime imaging microscopy

A principal objective in life sciences is the visualization of biochemical processes. Fluorescence-based techniques are widely used to demonstrate transport of relevant substances across cellular membranes. In this paper we report a novel noninvasive, real-time fluorescence lifetime imaging microscopy method for visualizing uptake and release of divalent copper ions (Cu(2+) ) in vivo. For this purpose, we employed a green fluorescent protein (GFP) form able to change its fluorescence lifetime upon Cu(2+) binding. We demonstrate that this technique is selective for Cu(2+) . We show the reversible decrease of the fluorescence lifetime of GFP from 2.2 to 1.6 ns in Escherichia coli and from 1.8 to 1.3 ns in root cells of Arabidopsis after the addition of Cu(2+) . Cu(2+) uptake of epidermal tobacco cells leads to a drop of the GFP lifetime from 2.5 to 2.2 ns. In summary, the spatially resolved visualization of Cu(2+) distribution in vivo is demonstrated in prokaryote and eukaryote cells.

Wide-field fluorescence lifetime imaging of cancer

Optical imaging of tissue autofluorescence has the potential to provide rapid label-free screening and detection of surface tumors for clinical applications, including when combined with endoscopy. Quantitative imaging of intensity-based contrast is notoriously difficult and spectrally resolved imaging does not always provide sufficient contrast. We demonstrate that fluorescence lifetime imaging (FLIM) applied to intrinsic tissue autofluorescence can directly contrast a range of surface tissue tumors, including in gastrointestinal tissues, using compact, clinically deployable instrumentation achieving wide-field fluorescence lifetime images of unprecedented clarity. Statistically significant contrast is observed between cancerous and healthy colon tissue for FLIM with excitation at 355 nm. To illustrate the clinical potential, wide-field fluorescence lifetime images of unstained ex vivo tissue have been acquired at near video rate, which is an important step towards real-time FLIM for diagnostic and interoperative imaging, including for screening and image-guided biopsy applications.

Real-time fluorescence lifetime imaging system with a 32 × 32 013μm CMOS low dark-count single-photon avalanche diode array

A compact real-time fluorescence lifetime imaging microscopy (FLIM) system based on an array of low dark count 0.13microm CMOS single-photon avalanche diodes (SPADs) is demonstrated. Fast background-insensitive fluorescence lifetime determination is achieved by use of a recently proposed algorithm called 'Integration for Extraction Method' (IEM) [J. Opt. Soc. Am. A 25, 1190 (2008)]. Here, IEM is modified for a wider resolvability range and implemented on the FPGA of the new SPAD array imager. We experimentally demonstrate that the dynamic range and accuracy of calculated lifetimes of this new camera is suitable for widefield FLIM applications by imaging a variety of test samples, including various standard fluorophores covering a lifetime range from 1.6ns to 16ns, microfluidic mixing of fluorophore solutions, and living fungal spores of Neurospora Crassa. The calculated lifetimes are in a good agreement with literature values. Real-time fluorescence lifetime imaging is also achieved, by performing parallel 32 x 16 lifetime calculations, realizing a compact and low-cost FLIM camera and promising for bigger detector arrays.

Real-time time-domain fluorescence lifetime imaging including single-shot acquisition with a segmented optical image intensifier

High-speed (video-rate) fluorescence lifetime imaging (FLIM) is reported using two different time-domain approaches based on gated optical image intensifier technology. The first approach utilizes a rapidly switchable variable delay generator with sequential image acquisition, while the second employs a novel segmented gated optical imager to acquire lifetime maps in a single shot. Lifetimes are fitted using both a non-linear least-squares fit analysis and the rapid lifetime determination method. Monte Carlo simulations were used to optimize the acquisition parameters and a comparison between theory and experiment is presented. The importance of single-shot imaging to minimize the deleterious impact of sample movements is highlighted. Real-time FLIM movies of multi-well plate samples and tissue autofluorescence are presented.

Instrumentation for real-time fluorescence lifetime imaging in endoscopy

The fluorescence lifetime of living tissues is, in certain cases, related to their pathologic state and is therefore of interest for cancer detection. Measuring fluorescence lifetime in vivo during an endoscopic examination has thus been a challenging objective for several years. The present article deals with the development and first clinical trails of an instrumentation producing fluorescence lifetime images in real time. The acquisition of such fast phenomenon (nanosecond time scale) on an image has been made possible by using the homodyne detection approach, in which the excitation light and the detection gain are modulated in a phase-coherent way. Based on images acquired at different phase between the excitation and detection modulation, the fluorescence lifetime is calculated for each pixel of the image. Different configurations of excitation modulation characteristics (pulse train versus sine-wave amplitude modulation) have been investigated and compared using Fourier transforms. Interestingly, a pulsed excitation combined with a sine-wave modulation detection gives valuable results. The expected auto-fluorescence signal emitted by human tissues under subthermal light excitation irradiance has been estimated at the wavelengths of interest. The limited number of auto-fluorescence photons results in relatively high noise on the lifetime calculated. The typical standard deviation is about 125 ps for lifetimes of 2.5 ns with a 322-pixel image (spatial integration). An in vivo image in the bronchi illustrates the potentiality of the new instrumentation. The results of this preliminary study indicate that the healthy bronchial mucosa, excited in the blue or in the green, fluoresces with a lifetime of 2.5 ns.