Video-rate Imaging Research Articles

Freehand and video-rate all-optical ultrasound imaging

All-optical ultrasound (AOUS) imaging, which uses light to both generate and detect ultrasound, is an emerging alternative to conventional electronic ultrasound imaging. To date, AOUS imaging has been performed using paradigms that either resulted in long acquisition times or employed bench-top imaging systems that were impractical for clinical use. In this work, we present a novel AOUS imaging paradigm where scanning optics are used to rapidly synthesise an imaging aperture. This paradigm enabled the first AOUS system with a flexible, handheld imaging probe, which represents a critical step towards clinical translation. This probe, which provides video-rate imaging and a real-time display, is demonstrated with phantoms and in vivo human tissue.

Real-time pixelwise phasor analysis for video-rate two-photon fluorescence lifetime imaging microscopy.

Two-photon fluorescence lifetime imaging microscopy (FLIM) is a widely used technique in biomedical optical imaging. Presently, many two-photon time-domain FLIM setups are limited by long acquisition and postprocessing times that decrease data throughput and inhibit the ability to image fast sub-second processes. Here, we present a versatile two-photon FLIM setup capable of video-rate (up to 25 fps) imaging with graphics processing unit (GPU)-accelerated pixelwise phasor analysis displayed and saved simultaneously with acquisition. The system uses an analog output photomultiplier tube in conjunction with 12-bit digitization at 3.2 GHz to overcome the limited maximum acceptable photon rate associated with the photon counting electronics in many FLIM systems. This allows for higher throughput FLIM acquisition and analysis, and additionally enables the user to assess sample fluorescence lifetime in real-time. We further explore the capabilities of the system to examine the kinetics of Rhodamine B uptake by human breast cancer cells and characterize the effect of pixel dwell time on the reduced nicotinamide adenine dinucleotide and reduced nicotinamide adenine dinucleotide phosphate (NAD(P)H) autofluorescence lifetime estimation accuracy.

Second Harmonic Imaging Enhanced by Deep Learning Decipher

Wavefront sensing and reconstruction are widely used for adaptive optics, aberration correction, and high-resolution optical phase imaging. Traditionally, interference and/or microlens arrays are used to convert the optical phase into intensity variation. Direct imaging of distorted wavefront usually results in complicated phase retrieval with low contrast and low sensitivity. Here, a novel approach has been developed and experimentally demonstrated based on the phase-sensitive information encoded into second harmonic signals, which are intrinsically sensitive to wavefront modulations. By designing and implementing a deep neural network, we demonstrate the second harmonic imaging enhanced by deep learning decipher (SHIELD) for efficient and resilient phase retrieval. Inheriting the advantages of two-photon microscopy, SHIELD demonstrates single-shot, reference-free, and video-rate phase imaging with sensitivity better than {\lambda}/100 and high robustness against noises, facilitating numerous applications from biological imaging to wavefront sensing.

Non-Rigid Shape From Water

We introduce a novel 3D sensing method for recovering a consistent, dense 3D shape of a dynamic, non-rigid object in water. The method reconstructs a complete (or fuller) 3D surface of the target object in a canonical frame (e.g., rest shape) as it freely deforms and moves between frames by estimating underwater 3D scene flow and using it to integrate per-frame depth estimates recovered from two near-infrared observations. The reconstructed shape is refined in the course of this global non-rigid shape recovery by leveraging both geometric and radiometric constraints. We implement our method with a single camera and a light source without the orthographic assumption on either by deriving a practical calibration method that estimates the point source position with respect to the camera. Our reconstruction method also accounts for scattering by water. We prototype a video-rate imaging system and show 3D shape reconstruction results on a number of real-world static, deformable, and dynamic objects and creatures in real-world water. The results demonstrate the effectiveness of the method in recovering complete shapes of complex, non-rigid objects in water, which opens new avenues of application for underwater 3D sensing in the sub-meter range.

Real-time tool to layer distance estimation for robotic subretinal injection using intraoperative 4D OCT.

The emergence of robotics could enable ophthalmic microsurgical procedures that were previously not feasible due to the precision limits of manual delivery, for example, targeted subretinal injection. Determining the distance between the needle tip, the internal limiting membrane (ILM), and the retinal pigment epithelium (RPE) both precisely and reproducibly is required for safe and successful robotic retinal interventions. Recent advances in intraoperative optical coherence tomography (iOCT) have opened the path for 4D image-guided surgery by providing near video-rate imaging with micron-level resolution to visualize retinal structures, surgical instruments, and tool-tissue interactions. In this work, we present a novel pipeline to precisely estimate the distance between the injection needle and the surface boundaries of two retinal layers, the ILM and the RPE, from iOCT volumes. To achieve high computational efficiency, we reduce the analysis to the relevant area around the needle tip. We employ a convolutional neural network (CNN) to segment the tool surface, as well as the retinal layer boundaries from selected iOCT B-scans within this tip area. This results in the generation and processing of 3D surface point clouds for the tool, ILM and RPE from the B-scan segmentation maps, which in turn allows the estimation of the minimum distance between the resulting tool and layer point clouds. The proposed method is evaluated on iOCT volumes from ex-vivo porcine eyes and achieves an average error of 9.24 µm and 8.61 µm measuring the distance from the needle tip to the ILM and the RPE, respectively. The results demonstrate that this approach is robust to the high levels of noise present in iOCT B-scans and is suitable for the interventional use case by providing distance feedback at an average update rate of 15.66 Hz.

Optimizing the performance of multiline-scanning confocal microscopy

Line-scanning (LS) confocal microscopy provides high imaging speed and moderate optical sectioning strength, which makes it a useful tool for imaging various biospecimens ranging from living cells to fixed tissues. Conventional LS systems have only used a single excitation line and slit, and thus have not fully exploited the benefits of parallelization. Here we investigate the optical performance of multi-LS confocal microscopy (mLS) by employing a digital micro-mirror device that provides programmable patterns of the illumination beam and the detection slit. Through experimental results and optical simulations, we assess the depth discrimination of mLS under different optical parameters and compare it with multi-point systems such as spinning disk confocal microscopy (SDCM). Under the same illumination duty cycle, we find that mLS has better optical sectioning than SDCM at a high degree of parallelization. The optimized mLS provides a low photobleaching rate and video-rate imaging while its optical sectioning is similar to single LS confocal microscopy.

Real-time video-rate perfusion imaging using multi-exposure laser speckle contrast imaging and machine learning

.Significance: Multi-exposure laser speckle contrast imaging (MELSCI) estimates microcirculatory blood perfusion more accurately than single-exposure LSCI. However, the technique has been hampered by technical limitations due to massive data throughput requirements and nonlinear inverse search algorithms, limiting it to an offline technique where data must be postprocessed.Aim: To present an MELSCI system capable of continuous acquisition and processing of MELSCI data, enabling real-time video-rate perfusion imaging with high accuracy.Approach: The MELSCI algorithm was implemented in programmable hardware (field programmable gate array) closely interfaced to a high-speed CMOS sensor for real-time calculation. Perfusion images were estimated in real-time from the MELSCI data using an artificial neural network trained on simulated data. The MELSCI perfusion was compared to two existing single-exposure metrics both quantitatively in a controlled phantom experiment and qualitatively in vivo.Results: The MELSCI perfusion shows higher signal dynamics compared to both single-exposure metrics, both spatially and temporally where heartbeat-related variations are resolved in much greater detail. The MELSCI perfusion is less susceptible to measurement noise and is more linear with respect to laser Doppler perfusion in the phantom experiment ().Conclusions: The presented MELSCI system allows for real-time acquisition and calculation of high-quality perfusion at 15.6 frames per second.

Iterative Learning Control for Video-Rate Atomic Force Microscopy

We present a control scheme for video-rate atomic force microscopy with rosette pattern. The controller structure involves a feedback internal-model-based controller and a feedforward iterative learning controller. The iterative learning controller is designed to improve tracking performance of the feedback-controlled scanner by rejecting the repetitive disturbances arising from the system nonlinearities. We investigate the performance of two inversion techniques for constructing the learning filter. We conduct tracking experiments using a two-degree-of-freedom microelectromechanical system (MEMS) nanopositioner at frame rates ranging from 5 to 20 frames per second. The results reveal that the algorithm converges rapidly and the iterative learning controller significantly reduces both the transient and steady-state tracking errors. We acquire and report a series of high-resolution time-lapsed video-rate AFM images with the rosette pattern.

Shortwave infrared polymethine fluorophores matched to excitation lasers enable non-invasive, multicolour in vivo imaging in real time.

High resolution, multiplexed experiments are a staple in cellular imaging. Analogous experiments in animals are challenging, however, due to significant scattering and autofluorescence in tissue at visible (VIS, 350–700 nm) and near-infrared (NIR, 700–1000 nm) wavelengths. Here, we enable real-time, non-invasive multicolor imaging experiments in animals through the design of optical contrast agents for the shortwave infrared (SWIR, 1000–2000 nm) region and complementary advances in imaging technologies. We developed tunable, SWIR-emissive flavylium polymethine dyes and established structure-photophysical property relationships for this class of bright SWIR contrast agents. In parallel, we designed an imaging system with variable NIR/SWIR excitation and single-channel detection, facilitating video-rate multicolor SWIR imaging for optically guided surgery and imaging of awake and moving mice with multiplexed detection. Optimized dyes matched to 980 nm and 1064 nm lasers, combined with the clinically approved indocyanine green, enabled real-time, three-color imaging with high temporal and spatial resolutions.

A passive video-rate terahertz human body imager with real-time calibration for security applications

Real-time video-rate passive terahertz imaging systems are highly demanded for practical applications, especially in security checking. Here, we demonstrate a passive video-rate terahertz human body imaging system, which was mainly consisted of a scanning module, a quasi-optical lens, a calibration module and a one-dimensional terahertz detector array. The terahertz waves radiated from human bodies in front of the imager can transmit through a terahertz window into the imager, and were reflected by the scanning module, and then focused on the detector array by the quasi-optical lens. A calibration module was also designed to calibrate the terahertz detectors in real-time without disturbing the imaging process. In combination of the scanning module with the detector array, the imager can obtain a full image of a human body standing at a distance of 1.5 m in front of the imager with a resolution of 1.5 cm and a frame rate of 10 fps. The imaging system can discover suspected dangerous items carried on the human body such as metals, ceramics, powders and liquids. Furthermore, an intelligent terahertz imaging algorithm employing convolutional neural network was also successfully realized based on the terahertz images produced by this system to improve the image quality and mark the detected items automatically. We believe our real-time video-rate terahertz imaging techniques and systems not only have great values for further inspiring developing terahertz imaging systems but also can accelerate the terahertz technology towards more practical applications.

Novel Technology for Studying Insulin Secretion: Imaging and Quantitative Analysis by a Bioluminescence Method

We developed a method of video-rate bioluminescence imaging to visualize proteins secreted from living cells. A protein of interest was fused to Gaussia luciferase (GLase), and the luminescence signals of secreted GLase with coelenterazine (luciferin) were visualized at a video-rate of 30-500 ms/frame by using a water-cooled EM-CCD camera. We established a subclonal rat INS-1E cell line, named iGL cells, stably expressing the fusion protein of insulin and GLase (Insulin-GLase). By stimulation with high glucose, 3D-cultured iGL cells showed synchronized oscillatory secretion of insulin for over 1 h, as similarly observed in an isolated rat pancreatic islet. In 2D-cultured iGL cells, the luminescence images indicated that synchronized insulin secretion was localized in intercellular spaces between cells. Further, the relative amount of insulin secretion from iGL cells was easily determined with a luminometer, and we demonstrated that cell-cell interaction of beta cells is fundamental to increase glucose-stimulated insulin secretion by synchronization. Thus, iGL cells would be valuable for studying oscillatory insulin secretion and evaluating anti-diabetic drugs. Our bioluminescence imaging method with GLase could be generally used for investigating protein secretion in 2D and 3D cell culture systems.

Video-rate lensless endoscope with self-calibration using wavefront shaping.

Lensless fiber endoscopes are of great importance for keyhole imaging. Coherent fiber bundles (CFB) can be used in endoscopes as remote phased arrays to capture images. One challenge is to image at high speed while correcting aberrations induced by the CFB. We propose the combination of digital optical phase conjugation, using a spatial light modulator, with fast scanning, for which a 2D galvo scanner and an adaptive lens are employed. We achieve the transmission of laser and image scanning through the CFB. Video-rate imaging at 20 Hz in 2D with subcellular resolution is demonstrated in 3D with 1 Hz. The sub-millimeter-diameter scanning endoscope has a great potential in biomedicine, for manipulation, e.g., in optogenetics, as well as in imaging.

9.4 MHz A-line rate optical coherence tomography at 1300\u2009nm using a wavelength-swept laser based on stretched-pulse active mode-locking

In optical coherence tomography (OCT), high-speed systems based at 1300 nm are among the most broadly used. Here, we present 9.4 MHz A-line rate OCT system at 1300 nm. A wavelength-swept laser based on stretched-pulse active mode locking (SPML) provides a continuous and linear-in-wavenumber sweep from 1240 nm to 1340 nm, and the OCT system using this light source provides a sensitivity of 98 dB and a single-sided 6-dB roll-off depth of 2.5 mm. We present new capabilities of the 9.4 MHz SPML-OCT system in three microscopy applications. First, we demonstrate high quality OCTA imaging at a rate of 1.3 volumes/s. Second, by utilizing its inherent phase stable characteristics, we present wide dynamic range en face Doppler OCT imaging with multiple time intervals ranging from 0.25 ms to 2.0 ms at a rate of 0.53 volumes/s. Third, we demonstrate video-rate 4D microscopic imaging of a beating Xenopus embryo heart at a rate of 30 volumes/s. This high-speed and high-performance OCT system centered at 1300 nm suggests that it can be one of the most promising high-speed OCT platforms enabling a wide range of new scientific research, industrial, and clinical applications at speeds of 10 MHz.

A beetle-type, variable-temperature scanning tunneling microscope for video-rate imaging

A setup is described that combines video-rate imaging with temperature variability for a beetle-type scanning tunneling microscope (STM). The STM is operated in the constant height mode. Sinusoidal driving voltages are used for the scan lines; the resulting image distortions are corrected by a straightforward mathematical operation. A phase shifter is used to compensate the time lag between the driving voltage and the tip motion. A preamplifier with a bandwidth of 300 kHz and an rms noise of ∼0.1 nA has been developed. Software has been written for data acquisition rates of up to 2 MHz and for saving movies of more than 3700 STM images of 200 × 200 pixels. Frame rates between 10 and 50 frames s−1 have been achieved, where the highest line rates are above the lowest resonance frequency of the STM. The accessible temperature range of 50–500 K is not impaired by the video mode.

Bringing real-time traceability to high-speed atomic force microscopy

In recent years, there has been growth in the development of high-speed AFMs, which offer the possibility of video rate scanning and long-range scanning over several hundred micrometres. However, until recently these instruments have been lacking full traceable metrology. In this paper traceable metrology, using optical interferometry, has been added to an open-loop contact-mode high-speed AFM to provide traceability both for short-range video rate images and large-area scans made using a combination of a high-speed dual-axis scanner and long-range positioning system. Using optical interferometry to determine stages’ positions and cantilever displacement enables the direct formation of images, obviating the need for complex post-processing corrections to compensate for lateral stage error. The application of metrology increases the spatial accuracy and linearisation of the high-speed AFM measurements, enabling the generation of very large traceable composite images.

Upconversion Nonlinear Structured Illumination Microscopy.

Video-rate super-resolution imaging through biological tissue can visualize and track biomolecule interplays and transportations inside cellular organisms. Structured illumination microscopy allows for wide-field super resolution observation of biological samples but is limited by the strong extinction of light by biological tissues, which restricts the imaging depth and degrades its imaging resolution. Here we report a photon upconversion scheme using lanthanide-doped nanoparticles for wide-field super-resolution imaging through the biological transparent window, featured by near-infrared and low-irradiance nonlinear structured illumination. We demonstrate that the 976 nm excitation and 800 nm upconverted emission can mitigate the aberration. We found that the nonlinear response of upconversion emissions from single nanoparticles can effectively generate the required high spatial frequency components in the Fourier domain. These strategies lead to a new modality in microscopy with a resolution below 131 nm, 1/7th of the excitation wavelength, and an imaging rate of 1 Hz.

LED array reflectance microscopy for scattering-based multi-contrast imaging.

LED array microscopy is an emerging platform for computational imaging with significant utility for biological imaging. Existing LED array systems often exploit transmission imaging geometries of standard brightfield microscopes that leave the rich backscattered field undetected. This backscattered signal contains high-resolution sample information with superb sensitivity to subtle structural features that make it ideal for biological sensing and detection. Here, we develop an LED array reflectance microscope capturing the sample's backscattered signal. In particular, we demonstrate multimodal brightfield, darkfield, and differential phase contrast imaging on fixed and living biological specimens including Caenorhabditis elegans (C. elegans), zebrafish embryos, and live cell cultures. Video-rate multimodal imaging at 20 Hz records real time features of freely moving C. elegans and the fast beating heart of zebrafish embryos. Our new reflectance mode is a valuable addition to the LED array microscopic toolbox.

Development a flexible light-sheet fluorescence microscope for high-speed 3D imaging of calcium dynamics and 3D imaging of cellular microstructure.

We report a flexible light-sheet fluorescence microscope (LSFM) designed for studying dynamic events in cardiac tissue at high speed in 3D and the correlation of these events to cell microstructure. The system employs two illumination-detection modes: the first uses angle-dithering of a Gaussian light sheet combined with remote refocusing of the detection plane for video-rate volumetric imaging; the second combines digitally-scanned light-sheet illumination with an axially-swept light-sheet waist and stage-scanned acquisition for improved axial resolution compared to the first mode. We present a characterisation of the spatial resolution of the system in both modes. The first illumination-detection mode achieves dual spectral-channel imaging at 25 volumes per second with 1024 × 200 × 50 voxel volumes and is demonstrated by time-lapse imaging of calcium dynamics in a live cardiomyocyte. The second illumination-detection mode is demonstrated through the acquisition of a higher spatial resolution structural map of the t-tubule network in a fixed cardiomyocyte cell.

Live feedback and 3D photoacoustic remote sensing.

As photoacoustic (PA) techniques progress towards clinical adoption, providing a high-speed live feedback becomes a high priority. To keep up with the instantaneous optical feedback of conventional light microscopes, PA imaging would need to provide a high-resolution video-rate live feed to the user. However, conventional PA microscopy typically trades resolution, sensitivity and imaging speed when optically scanning due to the difficult opto-acoustic confocal geometry. Here, we employ photoacoustic remote sensing (PARS), an all-optical technique that relies on optical confocal geometry, to provide a high-resolution live display in a reflection-mode PA architecture. Employing a conventional x-y galvanometer scanner and a 600 KHz pulse repetition rate laser we implement a system capable of acquiring 2.5 frames per second in 2D. To complement this fast scanning optical system, we implement a computationally inexpensive image reconstruction method that is able to render the frames with minimal overhead, providing a live display. Employing the proposed method, we demonstrate a live feedback with frame rates as high as 2.5 Hz in 2D and also report the first results of 3D imaging with a non-contact label-free reflection-mode technique. The method is validated with phantom studies and in-vivo imaging. Employing a repetition rate of 600 KHz, a live feed of carbon fibers is realized with a C-scan rate of 2.5 Hz. The imaging resolution was measured to be 1.2 µm, the highest reported for a real-time reflection-mode architecture. The mean and peak SNR were measured to be 44 and 62 dB respectively in-vivo. 3D visualizations of carbon fiber phantoms and mouse ear microvasculature structure are also demonstrated. In summary, we present a method that has a small computational overhead for image rendering, resulting in a live display capable of real-time frame rates. We also report the first 3D imaging with a non-contact label-free reflection-mode PA technique. The all-optical confocal geometry required by PARS is significantly easier to implement and maintain than the opto-acoustic geometry of conventional PA microscopy techniques. This results in a system capable of high resolution and sensitivity, imaging at real-time rates. The authors believe this work represents a vital step towards a clinical high-resolution reflection-mode video-rate PA imaging system.

Multiplex Stimulated Raman Scattering Imaging Cytometry Reveals Lipid-Rich Protrusions in Cancer Cells under Stress Condition.

SummaryIn situ measurement of cellular metabolites is still a challenge in biology. Conventional methods, such as mass spectrometry or fluorescence microscopy, would either destroy the sample or introduce strong perturbations to target molecules. Here, we present multiplex stimulated Raman scattering (SRS) imaging cytometry as a label-free single-cell analysis platform with chemical specificity and high-throughput capabilities. Using SRS imaging cytometry, we studied the metabolic responses of human pancreatic cancer cells under stress by starvation and chemotherapeutic drug treatments. We unveiled protrusions containing lipid droplets as a metabolic marker for stress-resistant cancer cells. Furthermore, by spectroscopic SRS mapping, we unveiled that triglyceride in lipid droplets are used for local energy production through lipolysis, autophagy, and β-oxidation. Our findings demonstrate the potential of targeting lipid metabolism for selective treatment of stress-resistant cancers. Collectively, these results highlight SRS imaging cytometry as a powerful label-free tool for biological discoveries with a high-throughput, high-content capacity.