Real-time 3D Imaging Research Articles

LIVE-streaming 3D images: A neuroscience approach to full-body illusions

Inspired by recent technological advances in the gaming industry, we used capture cards to create and LIVE-stream high quality 3D-images. With this novel technique, we developed a real-life stereoscopic 3D full-body illusion paradigm (3D projection). Unlike previous versions of the full-body illusion that rely upon unwieldy head-mounted displays, this paradigm enables the unobstructed investigation of such illusions with neuroscience methods (e.g., transcranial direct current stimulation, transcranial magnetic stimulation, electroencephalography, and near-infrared spectroscopy) and examination of their neural underpinnings. This paper has three aims: (i) to provide a step-by-step guide on how to implement 3D LIVE-streaming, (ii) to explain how this can be used to create a full-body illusion paradigm; and (iii) to present evidence that documents the effectiveness of our methods (de Boer et al., 2020), including suggestions for potential applications. Particularly significant is the fact that 3D LIVE-streaming is not GPU-intensive and can easily be applied to any device or screen that can display 3D images (e.g., TV, tablet, mobile phone). Therefore, these methods also have potential future clinical and commercial benefits. 3D LIVE-streaming could be used to enhance future clinical observations or educational tools, or potentially guide medical interventions with real-time high-quality 3D images. Alternatively, our methods can be used in future rehabilitation programs to aid recovery from nervous system injury (e.g., spinal cord injury, brain damage, limb loss) or in therapies aimed at alleviating psychosis symptoms. Finally, 3D LIVE-streaming could set a new standard for immersive online gaming as well as augmenting online and mobile experiences (e.g., video chat, social sharing/events).

Three-Dimensional Microscopic Image Reconstruction Based on Structured Light Illumination.

In this paper, we propose and experimentally demonstrate a three-dimensional (3D) microscopic system that reconstructs a 3D image based on structured light illumination. The spatial pattern of the structured light changes according to the profile of the object, and by measuring the change, a 3D image of the object is reconstructed. The structured light is generated with a digital micro-mirror device (DMD), which controls the structured light pattern to change in a kHz rate and enables the system to record the 3D information in real time. The working distance of the imaging system is 9 cm at a resolution of 20 μm. The resolution, working distance, and real-time 3D imaging enable the system to be applied in bridge and road crack examinations, and structure fault detection of transportation infrastructures.

Real‐Time 3D Imaging: GEOMScope: Large Field‐of‐View 3D Lensless Microscopy with Low Computational Complexity (Laser Photonics Rev. 15(8)/2021)

In article number 2100072, Feng Tian, Junjie Hu, and Weijian Yang demonstrate GEOMScope, a lensless microscope that can image 3D objects over a large volume with fine resolution. GEOMScope forms images through a thin layer of random microlens arrays, and reconstructs the 3D objects through an innovative, highly efficient algorithm combining geometrical optics and machine learning. This algorithm requires minimal computational resources and enables high-resolution, real-time 3D imaging over a large volume.

Real-Time Analysis of Oxygen Gradient in Oocyte Respiration Using a High-Density Microelectrode Array.

Physiological events related to oxygen concentration gradients provide valuable information to determine the state of metabolizing biological cells. The existing oxygen sensing methods (i.e., optical photoluminescence, magnetic resonance, and scanning electrochemical) are well-established and optimized for existing in vitro analyses. However, such methods also present various limitations in resolution, real-time sensing performance, complexity, and costs. An electrochemical imaging system with an integrated microelectrode array (MEA) would offer attractive means of measuring oxygen consumption rate (OCR) based on the cell’s two-dimensional (2D) oxygen concentration gradient. This paper presents an application of an electrochemical sensor platform with a custom-designed complementary-metal-oxide-semiconductor (CMOS)-based microchip and its Pt-coated surface MEA. The high-density MEA provides 16,064 individual electrochemical pixels that cover a 3.6 mm × 3.6 mm area. Utilizing the three-electrode configuration, the system is capable of imaging low oxygen concentration (18.3 µM, 0.58 mg/L, or 13.8 mmHg) at 27.5 µm spatial resolution and up to 4 Hz temporal resolution. In vitro oxygen imaging experiments were performed to analyze bovine cumulus-oocytes-complexes cells OCR and oxygen flux density. The integration of a microfluidic system allows proper bio-sample handling and delivery to the MEA surface for imaging. Finally, the imaging results are processed and presented as 2D heatmaps, representing the dissolved oxygen concentration in the immediate proximity of the MEA. This paper provides the results of real-time 2D imaging of OCR of live cells/tissues to gain spatial and temporal dynamics of target cell metabolism.

Robust real-time 3D imaging of moving scenes through atmospheric obscurant using single-photon LiDAR

Recently, time-of-flight LiDAR using the single-photon detection approach has emerged as a potential solution for three-dimensional imaging in challenging measurement scenarios, such as over distances of many kilometres. The high sensitivity and picosecond timing resolution afforded by single-photon detection offers high-resolution depth profiling of remote, complex scenes while maintaining low power optical illumination. These properties are ideal for imaging in highly scattering environments such as through atmospheric obscurants, for example fog and smoke. In this paper we present the reconstruction of depth profiles of moving objects through high levels of obscurant equivalent to five attenuation lengths between transceiver and target at stand-off distances up to 150 m. We used a robust statistically based processing algorithm designed for the real time reconstruction of single-photon data obtained in the presence of atmospheric obscurant, including providing uncertainty estimates in the depth reconstruction. This demonstration of real-time 3D reconstruction of moving scenes points a way forward for high-resolution imaging from mobile platforms in degraded visual environments.

Deep Learning-Based Denoising in High-Speed Portable Reflectance Confocal Microscopy.

Portable confocal microscopy (PCM) is a low-cost reflectance confocal microscopy technique that can visualize cellular details of human skin in vivo. When PCM images are acquired with a short exposure time to reduce motion blur and enable real-time 3D imaging, the signal-to-noise ratio (SNR) is decreased significantly, which poses challenges in reliably analyzing cellular features. In this paper, we evaluated deep learning (DL)-based approach for reducing noise in PCM images acquired with a short exposure time. Content-aware image restoration (CARE) network was trained with pairs of low-SNR input and high-SNR ground truth PCM images obtained from 309 distinctive regions of interest (ROIs). Low-SNR input images were acquired from human skin in vivo at the imaging speed of 180 frames/second. The high-SNR ground truth images were generated by registering 30 low-SNR input images obtained from the same ROI and summing them. The CARE network was trained using the Google Colaboratory Pro platform. The denoising performance of the trained CARE network was quantitatively and qualitatively evaluated by using image pairs from 45 unseen ROIs. CARE denoising improved the image quality significantly, increasing similarity with the ground truth image by 1.9 times, reducing noise by 2.35 times, and increasing SNR by 7.4 dB. Banding noise, prominent in input images, was significantly reduced in CARE denoised images. CARE denoising provided quantitatively and qualitatively better noise reduction than non-DL filtering methods. Qualitative image assessment by three confocal readers showed that CARE denoised images exhibited negligible noise more often than input images and non-DL filtered images. Results showed the potential of using a DL-based method for denoising PCM images obtained at a high imaging speed. The DL-based denoising method needs to be further trained and tested for PCM images obtained from disease-suspicious skin lesions.

Microinjection System to Enable Real-Time 3D Image Presentation Through Focal Position Adjustment

The microinjection technique, in which DNA is injected into cells, is widely applied in fields such as biology, medicine, and pharmacy. However, the implementation of the microinjection process requires skill and, thus, it is necessary to improve the operability. To this end, this study proposes a microinjection system that improves depth visibility through real-time 3D image presentation. An algorithm to estimate 3D position and size of micro-targets using a focal position adjustment mechanism is implemented to the system. In addition, a calibration method to adjust relative position and orientation between a target object and micromanipulators is developed. Then, the system presents reconstructed 3D images of a target and micromanipulators. Finally, the effectiveness of the proposed system is verified through manipulation experiments involving porcine embryos.

Deep learning model for simulating influence of natural organic matter in nanofiltration

Controlling membrane fouling in a membrane filtration system is critical to ensure high filtration performance. A forecast of membrane fouling could enable preliminary actions to relieve the development of membrane fouling. Therefore, we established a long short-term memory (LSTM) model to investigate the variations in filtration performance and fouling growth. For data acquisition, we first conducted lab-scale membrane fouling experiments to identify the diverse fouling mechanisms of natural organic matter (NOM) in nanofiltration (NF) systems. Four types of NOMs were considered as model foulants: humic acid, bovine-serum-albumin, sodium alginate, and tannic acid. In addition, real-time 2D images were acquired via optical coherence tomography (OCT) to quantify the cake layer formed on the membrane. Subsequently, experimental data were used to train the LSTM model to predict permeate flux and fouling layer thickness as output variables. The model performance exhibited root mean square errors of <1 L/m2/h for permeate flux and <10 µm for fouling layer thickness in both the training and validation steps. In this study, we demonstrated that deep learning can be used to simulate the influence of NOMs on the NF system and also be applied to simulate other membrane processes.

Technical Challenges of Real-Time Adaptive MR-Guided Radiotherapy.

In the past few years, radiotherapy (RT) has experienced a major technological innovation with the development of hybrid machines combining magnetic resonance (MR) imaging and linear accelerators. This new technology for MR-guided cancer treatment has the potential to revolutionize the field of adaptive RT due to the opportunity to provide high-resolution, real-time MR imaging before and during treatment application. However, from a technical point of view, several challenges remain which need to be tackled to ensure safe and robust real-time adaptive MR-guided RT delivery. In this manuscript, several technical challenges to MR-guided RT are discussed. Starting with magnetic field strength tradeoffs, the potential and limitations for purely MR-based RT workflows are discussed. Furthermore, the current status of real-time 3D MR imaging and its potential for real-time RT are summarized. Finally, the potential of quantitative MR imaging for future biological RT adaptation is highlighted.

Accelerating 3D single-molecule localization microscopy using blind sparse inpainting.

.Significance: Single-molecule localization-based super-resolution microscopy has enabled the imaging of microscopic objects beyond the diffraction limit. However, this technique is limited by the requirements of imaging an extremely large number of frames of biological samples to generate a super-resolution image, thus requiring a longer acquisition time. Additionally, the processing of such a large image sequence leads to longer data processing time. Therefore, accelerating image acquisition and processing in single-molecule localization microscopy (SMLM) has been of perennial interest.Aim: To accelerate three-dimensional (3D) SMLM imaging by leveraging a computational approach without compromising the resolution.Approach: We used blind sparse inpainting to reconstruct high-density 3D images from low-density ones. The low-density images are generated using much fewer frames than usually needed, thus requiring a shorter acquisition and processing time. Therefore, our technique will accelerate 3D SMLM without changing the existing standard SMLM hardware system and labeling protocol.Results: The performance of the blind sparse inpainting was evaluated on both simulation and experimental datasets. Superior reconstruction results of 3D SMLM images using up to 10-fold fewer frames in simulation and up to 50-fold fewer frames in experimental data were achieved.Conclusions: We demonstrate the feasibility of fast 3D SMLM imaging leveraging a computational approach to reduce the number of acquired frames. We anticipate our technique will enable future real-time live-cell 3D imaging to investigate complex nanoscopic biological structures and their functions.

Interactive effect of photoperiod and temperature on the induction and termination of embryonic diapause in the migratory locust.

Diapause plays a vital role in the survival and population reproduction of the migratory locust in temperate regions. Although some studies have suggested that photoperiod and temperature are two important environmental factors involved in the induction and termination of embryonic diapause in locusts, the independent or synergetic roles of these two factors are still unclear. We designed crossover experiments to investigate the effects of photoperiod, temperature and treatment time on the induction and termination of embryonic diapause in locusts by optical coherence tomography (OCT), which can track the entire embryonic developmental process via noninvasive 3D real-time imaging. Diapause induction experiments showed that a short photoperiod (in adults) and a low egg treatment temperature are the most important prerequisites for inducing egg diapause. The diapause rate reached 90.7% when the adults were reared under a short photoperiod (8 L:16 D) and the eggs were treated at 22 °C. However, the adult rearing temperature had no significant effect on diapause induction. Analysis of variance also confirmed that there was a significant interactive effect between parental photoperiod and egg treatment temperature. Moreover, diapause termination experiments showed that 13 °C (near the developmental threshold temperature of 15 °C) for 20 days is the most effective condition for terminating diapause. Our study provides accurate photoperiod and temperature thresholds for inducing and terminating diapause in the migratory locust, suggesting potential applications in the prediction and control of locust plagues. © 2021 Society of Chemical Industry.

Conceptual Design of Ukrainian Reusable Single-Stage Rocket with Vertical Takeoff & Landing Capability

In the paper we briefly describe conceptual design of Ukrainian reusable single-stage rocket demonstrator. The main objective consists in design, development and trials of a series of ‘Grasshopper’ demonstrators equipped with both the jet propulsion engine set and microwave/optical vision and measurement systems for the performing and control of its soft landing at the starting point. The greatest advantages of our design consist in availability of the results of previous developments of rocket engines and guiding systems for ‘Yuzhmash’ space ships, as well as advanced technique for real-time microwave 3D imaging using random/noise signals and MIMO1 radar operation mode with FPGA2-based signal processing.

An interactive holographic projection system that uses a hand-drawn interface with a consumer CPU

Holography is a promising technology for photo-realistic three-dimensional (3D) displays because of its ability to replay the light reflected from an object using a spatial light modulator (SLM). However, the enormous computational requirements for calculating computer-generated holograms (CGHs)—which are displayed on an SLM as a diffraction pattern—are a significant problem for practical uses (e.g., for interactive 3D displays for remote navigation systems). Here, we demonstrate an interactive 3D display system using electro-holography that can operate with a consumer’s CPU. The proposed system integrates an efficient and fast CGH computation algorithm for line-drawn 3D objects with inter-frame differencing, so that the trajectory of a line-drawn object that is handwritten on a drawing tablet can be played back interactively using only the CPU. In this system, we used an SLM with 1,920 times 1,080 pixels and a pixel pitch of 8 μm × 8 μm, a drawing tablet as an interface, and an Intel Core i9–9900K 3.60 GHz CPU. Numerical and optical experiments using a dataset of handwritten inputs show that the proposed system is capable of reproducing handwritten 3D images in real time with sufficient interactivity and image quality.

Complement receptor 3 mediates both sinking phagocytosis and phagocytic cup formation via distinct mechanisms.

A long-standing hypothesis is that complement receptors (CRs), especially CR3, mediate sinking phagocytosis, but evidence is lacking. Alternatively, CRs have been reported to induce membrane ruffles or phagocytic cups, akin to those induced by Fcγ receptors (FcγRs), but the details of these events are unclear. Here we used real-time 3D imaging and knockout mouse models to clarify how particles (human red blood cells) are internalized by resident peritoneal F4/80+ cells (macrophages) via CRs and/or FcγRs. We first show that FcγRs mediate highly efficient, rapid (2-3 min) phagocytic cup formation, which is completely abolished by deletion or mutation of the FcR γ-chain or conditional deletion of the signal transducer Syk. FcγR-mediated phagocytic cups robustly arise from any point of cell-particle contact, including filopodia. In the absence of CR3, FcγR-mediated phagocytic cups exhibit delayed closure and become aberrantly elongated. Independent of FcgRs, CR3 mediates sporadic ingestion of complement-opsonized particles by rapid phagocytic cup-like structures, typically emanating from membrane ruffles and largely prevented by deletion of the immunoreceptor tyrosine-based activation motif (ITAM) adaptors FcR γ-chain and DAP12 or Syk. Deletion of ITAM adaptors or Syk clearly revealed that there is a slow (10-25 min) sinking mode of phagocytosis via a restricted orifice. In summary, we show that (1) CR3 indeed mediates a slow sinking mode of phagocytosis, which is accentuated by deletion of ITAM adaptors or Syk, (2) CR3 induces phagocytic cup-like structures, driven by ITAM adaptors and Syk, and (3) CR3 is involved in forming and closing FcγR-mediated phagocytic cups.

Complement receptor 3 mediates both sinking phagocytosis and phagocytic cup formation via distinct mechanisms

A long-standing hypothesis is that complement receptors (CRs), especially CR3, mediate sinking phagocytosis, but evidence is lacking. Alternatively, CRs have been reported to induce membrane ruffles or phagocytic cups, akin to those induced by Fcγ receptors (FcγRs), but the details of these events are unclear. Here we used real-time 3D imaging and KO mouse models to clarify how particles (human red blood cells) are internalized by resident peritoneal F4/80+ cells (macrophages) via CRs and/or FcγRs. We first show that FcγRs mediate highly efficient, rapid (2–3 min) phagocytic cup formation, which is completely abolished by deletion or mutation of the FcR γ chain or conditional deletion of the signal transducer Syk. FcγR-mediated phagocytic cups robustly arise from any point of cell-particle contact, including filopodia. In the absence of CR3, FcγR-mediated phagocytic cups exhibit delayed closure and become aberrantly elongated. Independent of FcγRs, CR3 mediates sporadic ingestion of complement-opsonized particles by rapid phagocytic cup-like structures, typically emanating from membrane ruffles and largely prevented by deletion of the immunoreceptor tyrosine-based activation motif (ITAM) adaptors FcR γ chain and DAP12 or Syk. Deletion of ITAM adaptors or Syk clearly revealed that there is a slow (10–25 min) sinking mode of phagocytosis via a restricted orifice. In summary, we show that (1) CR3 indeed mediates a slow sinking mode of phagocytosis, which is accentuated by deletion of ITAM adaptors or Syk, (2) CR3 induces phagocytic cup-like structures, driven by ITAM adaptors and Syk, and (3) CR3 is involved in forming and closing FcγR-mediated phagocytic cups.

Enhanced Contactless Vital Sign Estimation from Real-Time Multimodal 3D Image Data.

The contactless estimation of vital signs using conventional color cameras and ambient light can be affected by motion artifacts and changes in ambient light. On both these problems, a multimodal 3D imaging system with an irritation-free controlled illumination was developed in this work. In this system, real-time 3D imaging was combined with multispectral and thermal imaging. Based on 3D image data, an efficient method was developed for the compensation of head motions, and novel approaches based on the use of 3D regions of interest were proposed for the estimation of various vital signs from multispectral and thermal video data. The developed imaging system and algorithms were demonstrated with test subjects, delivering a proof-of-concept.

Real-time three-dimensional histology-like imaging by label-free nonlinear optical microscopy.

The current gold-standard formalin-fixed and paraffin-embedded (FFPE) histology typically requires several days for tissue fixing, embedding, sectioning, and staining to provide depth-resolved tissue feature visualization. During these time- and labor- intense processes, the in vivo tissue dynamics and three-dimensional structures undergo inevitable loss and distortion. A simultaneous label-free autofluorescence multiharmonic (SLAM) microscope is used to conduct ex vivo and in vivo imaging of fresh human and rat tissues. Four nonlinear optical imaging modalities are integrated into this SLAM microscope, including second harmonic generation (SHG), two-photon fluorescence (2PF), third harmonic generation (THG), and three-photon fluorescence (3PF). By imaging fresh human and rat tissues without any tissue processing or staining, various biological tissue features are effectively visualized by one or multiple imaging modalities of the SLAM microscope. In particular, some of the most essential features in hematoxylin and eosin (H&E)-stained histology, such as collagen fibers and nuclei, are also present in the SLAM microscopy images with good contrast. Because nuclei are evident from negative contrast, the nuclei are segmented from the SLAM images using deep learning. Finally, a color-transforming algorithm is developed to convert the grey-scale images acquired by the SLAM microscope to the virtually H&E-stained histology-like images. The converted histology-like images are later compared with the FFPE histology at the same tissue site. In addition, the nuclear-to-cytoplasmic ratios (N/C ratios) of the cells in the SLAM image are quantified, which has diagnostic relevance for cancer. Various histological correlations are identified with high similarities for the color-converted histology-like SLAM microscopy images. By applying the color transforming algorithm on real-time SLAM image sequences and 3D SLAM image stacks, we report, for the first time and to the best our knowledge, real-time 3D histology-like imaging. Furthermore, the quantified N/C ratio of the cells in the SLAM image are overlaid on the converted histology-like image as a new image contrast. We demonstrated real-time 3D histology-like imaging and its future potential using SLAM microscopy aided by color remapping and deep-learning-based feature segmentation.

A Platform for Brain-wide Volumetric Functional Ultrasound Imaging and Analysis of Circuit Dynamics in Awake Mice

Imaging large-scale circuit dynamics is crucial to understanding brain function, but most techniques have a limited depth of field. Here, we describe volumetric functional ultrasound imaging (vfUSI), a platform for brain-wide vfUSI of hemodynamic activity in awake head-fixed mice. We combined a high-frequency 1,024-channel 2D-array transducer with advanced multiplexing and high-performance computing for real-time 3D power Doppler imaging at a high spatiotemporal resolution (220× 280× 175μm3, up to 6Hz). We developed a standardized software pipeline for registration, segmentation, and temporal analysis in 268 individual brain regions based on the Allen Mouse Common Coordinate Framework. We demonstrated the high sensitivity of vfUSI under multiple experimental conditions, and we successfully imaged stimulus-evoked activity when only a few trials were averaged. We also mapped neural circuits invivo across the whole brain during optogenetic activation of specific cell types. Moreover, we identified the sequential activation of sensory-motor networks during a grasping water-droplet task.

Conjugated Polymers: Solid‐State Low‐Temperature Thermoresponsive and Reversible Color Changes of Conjugated Polymer in Layered Structure: Beyond Infrared Thermography (Small 41/2020)

In article number 2004586, Yuya Oaki and co-workers report color changes originating from molecular motion of rigid conjugated polymer at unusual low-temperature range even in the solid crystalline state. The properties afford real-time 2D and 3D thermographic imaging in a temperature range much lower than typical infrared thermography.

Multifocal structured illumination optoacoustic microscopy

Optoacoustic (OA) imaging has the capacity to effectively bridge the gap between macroscopic and microscopic realms in biological imaging. High-resolution OA microscopy has so far been performed via point-by-point scanning with a focused laser beam, thus greatly restricting the achievable imaging speed and/or field of view. Herein we introduce multifocal structured illumination OA microscopy (MSIOAM) that attains real-time 3D imaging speeds. For this purpose, the excitation laser beam is shaped to a grid of focused spots at the tissue surface by means of a beamsplitting diffraction grating and a condenser and is then scanned with an acousto-optic deflector operating at kHz rates. In both phantom and in vivo mouse experiments, a 10 mm wide volumetric field of view was imaged with 15 Hz frame rate at 28 μm spatial resolution. The proposed method is expected to greatly aid in biological investigations of dynamic functional, kinetic, and metabolic processes across multiple scales.