Slow Frame Rate Research Articles

High-resolution in vivo 4D-OCT fish-eye imaging using 3D-UNet with multi-level residue decoder.

Optical coherence tomography (OCT) allows high-resolution volumetric imaging of biological tissues in vivo. However, 3D-image acquisition often suffers from motion artifacts due to slow frame rates and involuntary and physiological movements of living tissue. To solve these issues, we implement a real-time 4D-OCT system capable of reconstructing near-distortion-free volumetric images based on a deep learning-based reconstruction algorithm. The system initially collects undersampled volumetric images at a high speed and then upsamples the images in real-time by a convolutional neural network (CNN) that generates high-frequency features using a deep learning algorithm. We compare and analyze both dual-2D- and 3D-UNet-based networks for the OCT 3D high-resolution image reconstruction. We refine the network architecture by incorporating multi-level information to accelerate convergence and improve accuracy. The network is optimized by utilizing the 16-bit floating-point precision for network parameters to conserve GPU memory and enhance efficiency. The result shows that the refined and optimized 3D-network is capable of retrieving the tissue structure more precisely and enable real-time 4D-OCT imaging at a rate greater than 10 Hz with a root mean square error (RMSE) of ∼0.03.

Broadband dual-comb hyperspectral imaging and adaptable spectroscopy with programmable frequency combs

We explore the advantages of a free-form dual-comb spectroscopy (DCS) platform based on time-programmable frequency combs for real-time, penalty-free apodized scanning. In traditional DCS, the fundamental spectral point spacing, which equals the comb repetition rate, can be excessively fine for many applications. While fine point spacing is not itself problematic, it comes with the penalty of excess acquisition time. Post-processing apodization (windowing) can be applied to tailor the resolution to the sample, but only with a deadtime penalty proportional to the degree of apodization. The excess acquisition time remains. With free-form DCS, this deadtime is avoided by programming a real-time apodization pattern that dynamically reverses the pulse periods between the dual frequency combs. In this way, one can tailor the spectrometer’s resolution and update rate to different applications without penalty. We show the operation of a free-form DCS system where the spectral resolution is varied from the intrinsic fine 160 MHz up to 822 GHz by applying tailored real-time apodization. Because there is no deadtime penalty, the spectral signal-to-noise ratio increases linearly with resolution by 5000× over this range, as opposed to the square root increase observed for post-processing apodization in traditional DCS. We explore the flexibility to change resolution and update rate to perform hyperspectral imaging at slow camera frame rates, where the penalty-free apodization allows for optimal use of each frame. We obtain dual-comb hyperspectral movies at a 20 Hz spectrum update rate with broad optical spectral coverage of over 10 THz.

Video frame interpolation for high dynamic range sequences captured with dual‐exposure sensors

AbstractVideo frame interpolation (VFI) enables many important applications such as slow motion playback and frame rate conversion. However, one major challenge in using VFI is accurately handling high dynamic range (HDR) scenes with complex motion. To this end, we explore the possible advantages of dual‐exposure sensors that readily provide sharp short and blurry long exposures that are spatially registered and whose ends are temporally aligned. This way, motion blur registers temporally continuous information on the scene motion that, combined with the sharp reference, enables more precise motion sampling within a single camera shot. We demonstrate that this facilitates a more complex motion reconstruction in the VFI task, as well as HDR frame reconstruction that so far has been considered only for the originally captured frames, not in‐between interpolated frames. We design a neural network trained in these tasks that clearly outperforms existing solutions. We also propose a metric for scene motion complexity that provides important insights into the performance of VFI methods at test time.

Metasurface-enhanced light detection and ranging technology

Deploying advanced imaging solutions to robotic and autonomous systems by mimicking human vision requires simultaneous acquisition of multiple fields of views, named the peripheral and fovea regions. Among 3D computer vision techniques, LiDAR is currently considered at the industrial level for robotic vision. Notwithstanding the efforts on LiDAR integration and optimization, commercially available devices have slow frame rate and low resolution, notably limited by the performance of mechanical or solid-state deflection systems. Metasurfaces are versatile optical components that can distribute the optical power in desired regions of space. Here, we report on an advanced LiDAR technology that leverages from ultrafast low FoV deflectors cascaded with large area metasurfaces to achieve large FoV (150°) and high framerate (kHz) which can provide simultaneous peripheral and central imaging zones. The use of our disruptive LiDAR technology with advanced learning algorithms offers perspectives to improve perception and decision-making process of ADAS and robotic systems.

Arm Movement Analysis Technology of Wushu Competition Image Based on Deep Learning.

In order to improve the recognition accuracy of action poses for athletes in martial arts competitions, it is considered that a single frame pose does not have the temporal features required for sequential actions. Based on deep learning, this paper proposes an image arm movement analysis technology in martial arts competitions. The motion features of the arm are extracted from the bone sequence. Taking human bone motion information as temporal dynamic information, combined with RGB spatial features and depth map, the spatiotemporal features of arm motion data are formed. In this paper, we set up a slow frame rate channel and a fast frame rate channel to detect sequential motion of images. The deep learning model takes 16 frames from each video as samples. The softmax classifier is used to get the classification result of which action category the human action in the video belongs to. The test results show that the accuracy and recall rate of the arm motion analysis technology based on deep learning in martial arts competitions are 95.477% and 92.948%, respectively, with good motion analysis performance.

RapidEELS: machine learning for denoising and classification in rapid acquisition electron energy loss spectroscopy

Recent advances in detectors for imaging and spectroscopy have afforded in situ, rapid acquisition of hyperspectral data. While electron energy loss spectroscopy (EELS) data acquisition speeds with electron counting are regularly reaching 400 frames per second with near-zero read noise, signal to noise ratio (SNR) remains a challenge owing to fundamental counting statistics. In order to advance understanding of transient materials phenomena during rapid acquisition EELS, trustworthy analysis of noisy spectra must be demonstrated. In this study, we applied machine learning techniques to denoise high frame rate spectra, benchmarking with slower frame rate “ground truths”. The results provide a foundation for reliable use of low SNR data acquired in rapid, in-situ spectroscopy experiments. Such a tool-set is a first step toward both automation in microscopy as well as use of these methods to interrogate otherwise poorly understood transformations.

Real-time Global Illumination Decomposition of Videos

We propose the first approach for the decomposition of a monocular color video into direct and indirect illumination components in real time. We retrieve, in separate layers, the contribution made to the scene appearance by the scene reflectance, the light sources, and the reflections from various coherent scene regions to one another. Existing techniques that invert global light transport require image capture under multiplexed controlled lighting or only enable the decomposition of a single image at slow off-line frame rates. In contrast, our approach works for regular videos and produces temporally coherent decomposition layers at real-time frame rates. At the core of our approach are several sparsity priors that enable the estimation of the per-pixel direct and indirect illumination layers based on a small set of jointly estimated base reflectance colors. The resulting variational decomposition problem uses a new formulation based on sparse and dense sets of non-linear equations that we solve efficiently using a novel alternating data-parallel optimization strategy. We evaluate our approach qualitatively and quantitatively and show improvements over the state-of-the-art in this field, in both quality and runtime. In addition, we demonstrate various real-time appearance editing applications for videos with consistent illumination.

High frame rate video mosaicking microendoscope to image large regions of intact tissue with subcellular resolution.

High-resolution microendoscopy (HRME) is a low-cost strategy to acquire images of intact tissue with subcellular resolution at frame rates ranging from 11 to 18 fps. Current HRME imaging strategies are limited by the small microendoscope field of view (∼0.5 mm2); multiple images must be acquired and reliably registered to assess large regions of clinical interest. Image mosaics have been assembled from co-registered frames of video acquired as a microendoscope is slowly moved across the tissue surface, but the slow frame rate of previous HRME systems made this approach impractical for acquiring quality mosaicked images from large regions of interest. Here, we present a novel video mosaicking microendoscope incorporating a high frame rate CMOS sensor and optical probe holder to enable high-speed, high quality interrogation of large tissue regions of interest. Microendoscopy videos acquired at >90 fps are assembled into an image mosaic. We assessed registration accuracy and image sharpness across the mosaic for images acquired with a handheld probe over a range of translational speeds. This high frame rate video mosaicking microendoscope enables in vivo probe translation at >15 millimeters per second while preserving high image quality and accurate mosaicking, increasing the size of the region of interest that can be interrogated at high resolution from 0.5 mm2 to >30 mm2. Real-time deployment of this high-frame rate system is demonstrated in vivo and source code made publicly available.

Asynchronous Semantic Background Subtraction.

The method of Semantic Background Subtraction (SBS), which combines semantic segmentation and background subtraction, has recently emerged for the task of segmenting moving objects in video sequences. While SBS has been shown to improve background subtraction, a major difficulty is that it combines two streams generated at different frame rates. This results in SBS operating at the slowest frame rate of the two streams, usually being the one of the semantic segmentation algorithm. We present a method, referred to as “Asynchronous Semantic Background Subtraction” (ASBS), able to combine a semantic segmentation algorithm with any background subtraction algorithm asynchronously. It achieves performances close to that of SBS while operating at the fastest possible frame rate, being the one of the background subtraction algorithm. Our method consists in analyzing the temporal evolution of pixel features to possibly replicate the decisions previously enforced by semantics when no semantic information is computed. We showcase ASBS with several background subtraction algorithms and also add a feedback mechanism that feeds the background model of the background subtraction algorithm to upgrade its updating strategy and, consequently, enhance the decision. Experiments show that we systematically improve the performance, even when the semantic stream has a much slower frame rate than the frame rate of the background subtraction algorithm. In addition, we establish that, with the help of ASBS, a real-time background subtraction algorithm, such as ViBe, stays real time and competes with some of the best non-real-time unsupervised background subtraction algorithms such as SuBSENSE.

Nucleation, Growth, and Grain Structure Control of Electrodeposited Graded Density Alloys

Driven by an interest in dynamic high-pressure research, we are developing methods to create thick (mm) electrodeposited films of alloys with graded composition and density. We are designing deposition processes to dynamically control nucleation and growth to investigate correlations between grain structure and dynamic properties. We are using a combination of electrochemical techniques with in situ atomic force microscopy (AFM) complemented by phase contrast imaging (PCI) to isolate variables controlling nucleation and growth which lead to the grain structure of the deposit.Gold is of interest as a model system in high-pressure and dynamic research because of its high compressibility and inert character, especially because purity, defects, and grain structure can be carefully controlled. Further, because of its potential as a graded density alloy, baseline characterization is important. Gold has complete solid solubility with silver, and similar deposition chemistries. Using several techniques, including control of several process variables, we have produced films of graded composition in silver and gold. To broaden the potential for high-pressure research, similar techniques can be applied to any material that can be electrodeposited, and we will discuss graded compositions varying from aluminum to rhenium.The grain structure of the deposit can have an influence on dynamic properties, so in addition to composition, we have been researching deposition parameters to control this structure. Controlling nucleation and growth is critical to influencing ultimate grain structure in the deposit, so we have also determined and applied some basic nucleation and growth parameters for our systems. Although there are several well-known aspects, nucleation and growth are some of the most fundamental rate dependent processes in many chemical and physical transformations and remain among the most difficult to measure and predict. Assessing and predicting nucleation and growth, and rate-dependent processes, such as first-order phase transitions, are widely applicable to a multitude of practical applications. In the context of metal deposition for alloy systems, characterizing and modelling nucleation and growth can lead to an ability to design compositionally varied alloys with controlled grain structures. Therefore, we also performed some real-time dynamic investigations of nucleation and growth in our systems using complementary techniques of real-time, in situ AFM and x-ray PCI. AFM offers the ability to dynamically monitor growth processes, and through a method to isolate nucleation sites, estimate nucleation processes with spatial resolution sufficient to capture individual nuclei (nm or better). However, the technique is relatively slow (frame rate of several minutes), and dynamic processes are therefore only estimated. PCI, on the other hand, has very high time resolution (frame rate of ~60 ps), and can capture aspects of dynamics to complement AFM, even if the spatial resolution is lower (~2-3um).We present a study of electroplated films with compositional gradients and varied microstructures up to thicknesses of several mm for gold and silver, with discussion of other alloys in development. We relate the composition and grain structure to dynamic response. Constant potential and pulsed plating techniques were used to create a variety of grain structures, including high aspect ratio columnar grains. These structures and their dynamic response were compared to cast and wrought microstructures. Microscopy was correlated with mechanical characterization of the films at several rates and scales, and impact characteristics are compared and related to growth techniques and their relation to nucleation and growth parameters.LA-UR-19-31897

Vibration distribution measurement using downsampling phase-shifting interferometer

Vibration distribution measurements using interferometers have been evaluated by many researchers. We have used a laser diode interferometer to measure two-dimensional vibration distributions using a combination of phase-shifting and downsampling techniques. Our system simultaneously measured nanometer-order sinusoidal vibration distributions of two objects vibrating at different frequencies, using a charged coupled device camera with a slower frame rate than either of the two harmonic frequencies being measured. This is a cost-efficient and compact system capable of successfully measuring high-frequency vibrations without requiring expensive high-frame-rate cameras.

DAViS Camera Optical Flow

Frame-based optical flow (OF) methods struggle in the presence of large motion and occlusion due to slow frame rates. Optical flow of dynamic vision sensor (DVS) has gained attention recently as a way to overcome these shortcomings. DVS reports “events” corresponding log intensity changes exceeding a specific threshold, with an accuracy of microsecond order. The increased temporal sampling rate are indeed helpful, but the poor spatial fidelity of DVS outputs make events-based OF less reliable overall. In this work, we consider a new sensor called DAViS that combines the conventional active pixel sensor (APS) and DVS circuitries, yielding a conventional intensity image frames as well as the events. We propose a novel optical flow method designed specifically for a DAViS camera that leverages the high spatial fidelity of intensity image frames and the high temporal resolution of events generated by DVS. Hence, the proposed DAViS-OF method yields reliable motion vector estimates while overcoming the fast motion and occlusion problems. The proposed DAViS-OF method is computationally efficient and is suitable for real-time implementation.

Using slow frame rate imaging to extract fast receptive fields

In functional imaging, large numbers of neurons are measured during sensory stimulation or behavior. This data can be used to map receptive fields that describe neural associations with stimuli or with behavior. The temporal resolution of these receptive fields has traditionally been limited by image acquisition rates. However, even when acquisitions scan slowly across a population of neurons, individual neurons may be measured at precisely known times. Here, we apply a method that leverages the timing of neural measurements to find receptive fields with temporal resolutions higher than the image acquisition rate. We use this temporal super-resolution method to resolve fast voltage and glutamate responses in visual neurons in Drosophila and to extract calcium receptive fields from cortical neurons in mammals. We provide code to easily apply this method to existing datasets. This method requires no specialized hardware and can be used with any optical indicator of neural activity.

Tracking and frame-rate enhancement for real-time 2D human pose estimation

We propose a near real-time solution for frame-rate enhancement that enables the use of existing sophisticated pose estimation solutions at elevated frame rates. Our approach couples a keypoint human pose estimator with optical flow using a multistage system of queues operating in a multi-threaded environment. As additional contributions, we propose a pose tracking solution and an approach to overcome errors caused by optical flow. A reduction in error in the range of 30–35% is observed at practical frame rates of pose estimator (4–6 frames per second) while processing $$1920\times 1080$$ resolution 30 frames-per-second videos at native frame rate. Slower frame rates have increased the reduction of error up to 50%, thereby promoting the use of cheaper hardware and sharing of expensive hardware. Thus, while improving accuracy by enabling sophisticated pose estimation models to operate at above-par frame rates, our approach reduces cost per frame by promoting efficient resource utilization.

Ultrafast polarization bio-imaging based on coherent detection and time-stretch techniques.

Optical polarization imaging has played an important role in many biological and biomedical applications, as it provides a label-free and non-invasive detection scheme to reveal the polarization information of optical rotation, birefringence, and photoelasticity distribution inherent in biological samples. However, the imaging speeds of the previously demonstrated polarization imaging techniques were often limited by the slow frame rates of the arrayed imaging detectors, which usually run at frame rates of several hundred hertz. By combining the optical coherent detection of orthogonal polarizations and the optical time-stretch imaging technique, we achieved ultrafast polarization bio-imaging at an extremely fast record line scanning rate up to 100 MHz without averaging. We experimentally demonstrated the superior performance of our method by imaging three slices of different kinds of biological samples with the retrieved Jones matrix and polarization-sensitive information including birefringence and diattenuation. The proposed system in this paper may find potential applications for ultrafast polarization dynamics in living samples or some other advanced biomedical research.

Ultrafast X-ray imaging of laser-metal additive manufacturing processes.

The high-speed synchrotron X-ray imaging technique was synchronized with a custom-built laser-melting setup to capture the dynamics of laser powder-bed fusion processes in situ. Various significant phenomena, including vapor-depression and melt-pool dynamics and powder-spatter ejection, were captured with high spatial and temporal resolution. Imaging frame rates of up to 10 MHz were used to capture the rapid changes in these highly dynamic phenomena. At the same time, relatively slow frame rates were employed to capture large-scale changes during the process. This experimental platform will be vital in the further understanding of laser additive manufacturing processes and will be particularly helpful in guiding efforts to reduce or eliminate microstructural defects in additively manufactured parts.

Reducing radiation exposure during procedures performed in the electrophysiology laboratory.

Expert societies recently published strong recommendations to reduce the exposure of patients and staff to ionizing radiation (IR) during interventional and electrophysiology (EP) procedures. However, adherence to these guidelines remains difficult and the impact of implementing such recommendations is poorly characterized. We conducted a single-center cohort study to quantify radiation exposure over time in three EP laboratories at the Montreal Heart Institute during 5,546 consecutive procedures from 2012 to 2015 by 11 primary operators. Overall, 2,618 (47.2%) procedures were catheter-based and 2,928 (52.8%) were device interventions. Interventions to reduce radiation exposure included educational initiatives to raise awareness (i.e., limiting cine acquisition, patient position, table height), slower frame rate, lower radiation dose per pulse, collimation, and integration with 3-D mapping systems and/or MediGuide technology. An 85% reduction in IR exposure was observed from 2012 to 2015, with the mean dose-area-product (DAP) decreasing from 7.65 ± 0.05Gy·cm2 to 1.15 ± 0.04Gy·cm2 (P<0.001). This was true for catheter-based procedures (mean DAP 16.99 ± 0.08 to 2.00 ± 0.06Gy·cm2 , P<0.001) and device interventions (mean DAP 4.18 ± 0.06 to 0.64 ± 0.05Gy·cm2 , P<0.001). The median effective dose of IR recorded per quarter by 282 cervical dosimeters on EP staff decreased from 0.57 (IQR 0.18, 1.03) mSv in 2012 to 0.00 (IQR 0.00, 0.19) mSv in 2015, P<0.001. Enforcing good clinical practices with simple measures and low-dose fluoroscopy settings are highly effective in reducing IR exposure in the EP lab. These promising results should encourage other EP labs to adopt similar protective measures.

A simple and reusable bilayer membrane-based microfluidic device for the study of gradient-mediated bacterial behaviors.

We have developed a user-friendly microfluidic device for the study of gradient-mediated bacterial behaviors, including chemotaxis. This device rapidly establishes linear concentration gradients by exploiting solute diffusion through porous membranes in the absence of convective flows. As such, the gradients are created rapidly and can be sustained for long time periods (e.g., hours), sufficient to evaluate cell phenotype. The device exploits a unique simple bilayer configuration that enables rapid setup and quick reproducible introduction of cells. Its reusability represents an additional advantage in that it need not be limited to settings with microfluidics expertise. We have successfully demonstrated the applicability of this tool in studying the chemotactic response of Escherichia coli to glucose. When coupled with our recent Python program, quantified metrics such as speed, ratio of tumble to run, and effective diffusivity can be obtained from slow frame rate videos. Moreover, we introduce a chemotaxis partition coefficient that conveniently scores swimming behavior on the single-cell level.

Flexible polygon-mirror based laser scanning microscope platform for multiphoton in-vivo imaging.

Commercial microscopy systems make use of tandem scanning i.e. either slow or fast scanning. We constructed, for the first time, an advanced control system capable of delivering a dynamic line scanning speed ranging from 2.7kHz to 27kHz and achieve variable frame rates from 5Hz to 50Hz (512 × 512). The dynamic scanning ability is digitally controlled by a new customized open-source software named PScan1.0. This permits manipulation of scanning rates either to gain higher fluorescence signal at slow frame rate without increasing laser power or increase frame rates to capture high speed events. By adjusting imaging speed from 40Hz to 160Hz, we capture a range of calcium waves and transient peaks from soma and dendrite of single fluorescence neuron (CAL-520AM). Motion artifacts arising from respiratory and cardiac motion in small animal imaging reduce quality of real-time images of single cells in-vivo. An image registration algorithm, integrated with PScan1.0, was shown to perform both real time and post-processed motion correction. The improvement is verified by quantification of blood flow rates. This work describes all the steps necessary to develop a high performance and flexible polygon-mirror based multiphoton microscope system for in-vivo biological imaging.

Coherent flow power Doppler (CFPD): flow detection using spatial coherence beamforming.

Power Doppler imaging is a widely used method of flow detection for tissue perfusion monitoring, inflammatory hyperemia detection, deep vein thrombosis diagnosis, and other clinical applications. However, thermal noise and clutter limit its sensitivity and ability to detect slow flow. In addition, large ensembles are required to obtain sufficient sensitivity, which limits frame rate and yields flash artifacts during moderate tissue motion. We propose an alternative method of flow detection using the spatial coherence of backscattered ultrasound echoes. The method enhances slow flow detection and frame rate, while maintaining or improving the signal quality of conventional power Doppler techniques. The feasibility of this method is demonstrated with simulations, flow-phantom experiments, and an in vivo human thyroid study. In comparison with conventional power Doppler imaging, the proposed method can produce Doppler images with 15- to 30-dB SNR improvement. Therefore, the method is able to detect flow with velocities approximately 50% lower than conventional power Doppler, or improve the frame rate by a factor of 3 with comparable image quality. The results show promise for clinical applications of the method.