Real-time Image Reconstruction Research Articles

Live cell imaging of cellular dynamics in poplar wood using computational cannula microscopy

This study presents significant advancements in computational cannula microscopy for live imaging of cellular dynamics in poplar wood tissues. Leveraging machine-learning models such as pix2pix for image reconstruction, we achieved high-resolution imaging with a field of view of 55µm using a 50µm-core diameter probe. Our method allows for real-time image reconstruction at 0.29 s per frame with a mean absolute error of 0.07. We successfully captured cellular-level dynamics in vivo, demonstrating morphological changes at resolutions as small as 3µm. We implemented two types of probabilistic neural network models to quantify confidence levels in the reconstructed images. This approach facilitates context-aware, human-in-the-loop analysis, which is crucial for in vivo imaging where ground-truth data is unavailable. Using this approach we demonstrated deep in vivo computational imaging of living plant tissue with high confidence (disagreement score ⪅0.2). This work addresses the challenges of imaging live plant tissues, offering a practical and minimally invasive tool for plant biologists.

Electrode module for EIT with a robust howland current source

Electrical Impedance Tomography (EIT) is a non-invasive imaging technique that reconstructs internal conductivity distributions of a body from electrical measurements taken on its boundary. This study contributes to the field by focusing on the technological intricacies of absolute EIT imaging, which is challenged by limitations such as the resolution capacity of the hardware and the complexities introduced by imaging capacitive bodies. The novel EIT system architecture proposed enhances the accuracy of measurement by integrating current sources and Analog-to-Digital Converters (ADCs) closer to the electrodes, employing alternating current excitations to accurately capture phase information. This system uses a dynamic arrangement of surface electrodes that continuously alter their roles between current injection and voltage measurement, in a synchronized sequence, to ensure the accuracy of the measurements. The paper describes the design and implementation of both the excitation and measurement subsystems, highlighting the use of digital signal demodulation near the electrode to reduce data transfer issues. Experimental results confirm the system’s capability for real-time image reconstruction at 50 frames per second with precision in phase delay measurements, suggesting significant potential for clinical and industrial applications. Future work will aim to further refine signal generation with higher-speed DACs and expand to image reconstruction with more channels.

Learned Full Waveform Inversion Incorporating Task Information for Ultrasound Computed Tomography.

Ultrasound computed tomography (USCT) is an emerging imaging modality that holds great promise for breast imaging. Full-waveform inversion (FWI)-based image reconstruction methods incorporate accurate wave physics to produce high spatial resolution quantitative images of speed of sound or other acoustic properties of the breast tissues from USCT measurement data. However, the high computational cost of FWI reconstruction represents a significant burden for its widespread application in a clinical setting. The research reported here investigates the use of a convolutional neural network (CNN) to learn a mapping from USCT waveform data to speed of sound estimates. The CNN was trained using a supervised approach with a task-informed loss function aiming at preserving features of the image that are relevant to the detection of lesions. A large set of anatomically and physiologically realistic numerical breast phantoms (NBPs) and corresponding simulated USCT measurements was employed during training. Once trained, the CNN can perform real-time FWI image reconstruction from USCT waveform data. The performance of the proposed method was assessed and compared against FWI using a hold-out sample of 41 NBPs and corresponding USCT data. Accuracy was measured using relative mean square error (RMSE), structural self-similarity index measure (SSIM), and lesion detection performance (DICE score). This numerical experiment demonstrates that a supervised learning model can achieve accuracy comparable to FWI in terms of RMSE and SSIM, and better performance in terms of task performance, while significantly reducing computational time.

Design of Nupix-A2, a Monolithic Active Pixel Sensor for heavy-ion physics

The High Intensity heavy-ion Accelerator Facility (HIAF) is being constructed to generate intense beams of primary and radioactive ion for a wide range of research fields. Hence, a Monolithic Active Pixel Sensor (MAPS) named Nupix-A2 has been developed in a 130-nm High Resistivity CMOS process. The Nupix-A2 can simultaneously measure the particle hit' energy, arrival time, and position. It consists of a 128 × 128 pixel array, a digital-to-analog converter array, and a digital control module. The Nupix-A2 can measure energy deposition from 300 e- to over 50 ke- and time duration from 13 μs to 140 μs. This sensor also offers full readout mode and fast readout mode. In full-readout mode, all pixels measure arrival time and energy, suitable for real-time beam monitoring and image reconstruction. In fast-readout mode, only particle hit's positions are detected and read out. This paper presents the design of the Nupix-A2.

Comparative Study of Ultrasound Tissue Motion Tracking Techniques for Effective Breast Ultrasound Elastography

Breast cancer is the most common and deadly cancer in women, where early detection is of the utmost importance as survival rates decrease with the advancement of the disease. Most available methods of breast cancer screening and evaluation lack the ability to effectively differentiate between benign and malignant lesions without a biopsy. Ultrasound elastography (USE) is a cost-effective method that can potentially provide an initial malignancy assessment at the bedside. One of the challenges, however, is the uncertainty of tissue displacement data when performing USE due to out-of-plane movement of the tissue during mechanical stimulation, in addition to the computational efficiency necessary for real-time image reconstruction. This work presents a comparison of four different theoretically sound displacement estimators for their ability in tissue Young’s modulus reconstruction level with an emphasis on quality-to-runtime ratio to determine which estimators are most suitable for real-time USE systems. The methods are known in literature as AM2D, GLUE, OVERWIND, and SOUL methods. The effectiveness of each method was assessed as a stand-alone method or in combination with a strain field enhancement technique known as STREAL, which was recently developed using tissue mechanics-based regularization. The study was performed using radiofrequency US data pertaining to in silico and tissue mimicking phantoms in addition to clinical data. This data was used to generate tissue displacement fields employed to generate axial and lateral strain images before Young’s modulus images were reconstructed. The study indicates that the AM2D displacement estimator, which is an older and computationally less involved method, along with a tissue-mechanics-based image processing algorithm, performs very well, with high CNR, SNR, and preservation of tumor heterogeneity obtained at both strain and stiffness image levels, while its computation run-time is much lower compared to other estimation methods. As such, it can be recommended for incorporation in real-time USE systems.

A deep neural network for real-time optoacoustic image reconstruction with adjustable speed of sound

A deep neural network for real-time optoacoustic image reconstruction with adjustable speed of sound

A frequency-domain model-based reconstruction method for transcranial photoacoustic imaging: A 2D numerical investigation

Phase aberration caused by the skull is a major barrier to achieving high quality photoacoustic images of human and non-human primates’ brains. To address this issue, time-reversal methods have been used but they are computationally demanding and slow due to relying on solving the full-wave equation. The proposed approach is based on model-based image reconstruction in the frequency-domain to achieve near real-time image reconstruction. The relationship between an imaging region and transducer array elements can be mathematically described as a model matrix and the image reconstruction can be performed by pseudo-inverse of the model matrix. The model matrix is numerically calculated due to the lack of analytical solutions for transcranial ultrasound. However, this calculation only needs to be performed once for a given experimental setup and the same acoustic medium, and is an offline process not affecting the actual image reconstruction time. This non-iterative mode-based method demonstrates a substantial improvement in image reconstruction time, being approximately 18 times faster than the time-reversal method, all while maintaining comparable image quality.

Hybrid spiral scanning in a double-clad fiber-based handheld confocal scanning light ophthalmoscope.

High-speed, accessible, and robust in vivo imaging of the human retina is critical for screening of retinal pathologies, such as diabetic retinopathy, age-related macular degeneration, and others. Scanning light ophthalmoscopy (SLO) is a retinal imaging modality that produces digital, en face images of the human retina with superior image gradability rates when compared to the current standard of care in screening for these diseases, namely the flood-illumination handheld fundus camera (HFC). However, current-generation commercial SLO systems are mostly tabletop devices, limiting their accessibility and utility in screening applications. Moreover, most existing SLO systems use raster scan patterns, which are both inefficient and lead to undesired subject gaze drift when used with visible or pseudo-visible illumination. Non-raster scan patterns, especially spiral scanning as described herein, promise advantages in both scan efficiency and reduced subject eye motion. In this work, we introduce a novel "hybrid spiral" scan pattern and the associated hardware design and real-time image reconstruction techniques necessary for its implementation in an SLO system. Building upon this core hybrid spiral scanning SLO (HSS-SLO) technology, we go on to present a complete handheld HSS-SLO system, featuring a fiber-coupled portable patient interface which leverages a dual-clad fiber (DCF) to form a single-path optical topology, thus ensuring mechanically robust co-alignment of illumination and collection apertures, a necessity for a handheld system. The feasibility of HSS-SLO for handheld, in vivo imaging is demonstrated by imaging eight human volunteers.

Structured illumination microscopy with a phase-modulated spinning disk for optical sectioning.

Among various super-resolution microscopic techniques, structured illumination microscopy (SIM) stands out for live-cell imaging because of its higher imaging speed. However, conventional SIM lacks optical sectioning capability. Here we demonstrate a new, to the best of our knowledge, approach using a phase-modulated spinning disk (PMSD) that enhances the optical sectioning capability of SIM. The PMSD consists of a pinhole array for confocal imaging and a transparent polymer layer for light phase modulation. The light phase modulation was designed to cancel the zeroth-order diffracted beam and create a sharp lattice illumination pattern using the interference of four first-order diffracted beams. In the detection optical path, the PMSD serves as a spatial filter to physically reject about 80% of the out-of-focus signals, an approach that allows for real-time optical reconstruction of super-resolved images with enhanced contrast. Furthermore, the simplicity of the design makes it easy to upgrade a conventional fluorescence microscope to a PMSD SIM system.

Real-time, deep-learning aided lensless microscope.

Traditional miniaturized fluorescence microscopes are critical tools for modern biology. Invariably, they struggle to simultaneously image with a high spatial resolution and a large field of view (FOV). Lensless microscopes offer a solution to this limitation. However, real-time visualization of samples is not possible with lensless imaging, as image reconstruction can take minutes to complete. This poses a challenge for usability, as real-time visualization is a crucial feature that assists users in identifying and locating the imaging target. The issue is particularly pronounced in lensless microscopes that operate at close imaging distances. Imaging at close distances requires shift-varying deconvolution to account for the variation of the point spread function (PSF) across the FOV. Here, we present a lensless microscope that achieves real-time image reconstruction by eliminating the use of an iterative reconstruction algorithm. The neural network-based reconstruction method we show here, achieves more than 10000 times increase in reconstruction speed compared to iterative reconstruction. The increased reconstruction speed allows us to visualize the results of our lensless microscope at more than 25 frames per second (fps), while achieving better than 7 µm resolution over a FOV of 10 mm2. This ability to reconstruct and visualize samples in real-time empowers a more user-friendly interaction with lensless microscopes. The users are able to use these microscopes much like they currently do with conventional microscopes.

Submerged single-photon LiDAR imaging sensor used for real-time 3D scene reconstruction in scattering underwater environments.

We demonstrate a fully submerged underwater LiDAR transceiver system based on single-photon detection technologies. The LiDAR imaging system used a silicon single-photon avalanche diode (SPAD) detector array fabricated in complementary metal-oxide semiconductor (CMOS) technology to measure photon time-of-flight using picosecond resolution time-correlated single-photon counting. The SPAD detector array was directly interfaced to a Graphics Processing Unit (GPU) for real-time image reconstruction capability. Experiments were performed with the transceiver system and target objects immersed in a water tank at a depth of 1.8 meters, with the targets placed at a stand-off distance of approximately 3 meters. The transceiver used a picosecond pulsed laser source with a central wavelength of 532 nm, operating at a repetition rate of 20 MHz and average optical power of up to 52 mW, dependent on scattering conditions. Three-dimensional imaging was demonstrated by implementing a joint surface detection and distance estimation algorithm for real-time processing and visualization, which achieved images of stationary targets with up to 7.5 attenuation lengths between the transceiver and the target. The average processing time per frame was approximately 33 ms, allowing real-time three-dimensional video demonstrations of moving targets at ten frames per second at up to 5.5 attenuation lengths between transceiver and target.

Silicon photonic integrated circuit for high-resolution multimode fiber imaging system

We propose and demonstrate a silicon photonic integrated circuit (PIC) for exciting different spatial modes launched into a multimode-fiber (MMF) speckle imaging system. The PIC consists of a 45-channel optical phased array and an array of nanoantennas to bridge the PIC and MMF. The nanoantenna array can excite a wide range of spatial modes in the MMF with a mode-group dependent loss of less than 3 dB. A high spatial resolution, which approaches the theoretical limit determined by the number of modes in the MMF, is realized by using the proposed PIC. An equivalent resolution of 1.75 µm is experimentally attained across a field of view of 105 µm. Two different algorithms for image reconstruction are compared. The algorithm based on truncated singular value decomposition is computationally efficient and suitable for real-time image reconstruction, whereas the algorithm based on total-variation regularization produces higher imaging quality. The number of resolvable points is derived to be ∼3000, which is more than the square of the number of phase shifters. These results represent the highest spatial resolution yet demonstrated in a PIC-based MMF imaging system.

A model-based image reconstruction algorithm for near real-time transcranial photoacoustic imaging

Photoacoustic imaging is a promising hybrid imaging modality that can visualize both microscopic and macroscopic structures and oxygenation changes in brain function. Phase aberration caused by the skull is a major barrier for high-quality photoacoustic imaging of the brain. Time-reversal methods have been used to address this issue, though they rely on solving the full-wave equation, and is therefore computationally heavy and slow. Herein, a near real-time model-based image reconstruction algorithm is proposed. The acoustic forward model is mathematically described as a model matrix and image reconstructions are performed by employing the pseudoinverse of the model matrix and calculating the initial pressure distribution. The pseudoinverse matrix only needs to be computed once for the same experimental setup and acoustic medium and is obtained offline prior to imaging. The proposed algorithm shows equivalent image quality but considerably faster reconstruction time (>40 times faster), compared to the time-reversal method and therefore could potentially enable near real-time photoacoustic imaging.

GPU-accelerated iterative method for FD-OCT image reconstruction with an image-level cross-domain regularizer.

The image reconstruction for Fourier-domain optical coherence tomography (FD-OCT) could be achieved by iterative methods, which offer a more accurate estimation than the traditional inverse discrete Fourier transform (IDFT) reconstruction. However, the existing iterative methods are mostly A-line-based and are developed on CPU, which causes slow reconstruction. Besides, A-line-based reconstruction makes the iterative methods incompatible with most existing image-level image processing techniques. In this paper, we proposed an iterative method that enables B-scan-based OCT image reconstruction, which has three major advantages: (1) Large-scale parallelism of the OCT dataset is achieved by using GPU acceleration. (2) A novel image-level cross-domain regularizer was developed, such that the image processing could be performed simultaneously during the image reconstruction; an enhanced image could be directly generated from the OCT interferogram. (3) The scalability of the proposed method was demonstrated for 3D OCT image reconstruction. Compared with the state-of-the-art (SOTA) iterative approaches, the proposed method achieves higher image quality with reduced computational time by orders of magnitude. To further show the image enhancement ability, a comparison was conducted between the proposed method and the conventional workflow, in which an IDFT reconstructed OCT image is later processed by a total variation-regularized denoising algorithm. The proposed method can achieve a better performance evaluated by metrics such as signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR), while the speed is improved by more than 30 times. Real-time image reconstruction at more than 20 B-scans per second was realized with a frame size of 4096 (axial) × 1000 (lateral), which showcases the great potential of the proposed method in real-world applications.

An End-to-End Network for Upright Adjustment of Panoramic Images

Nowadays, panoramic images can be easily obtained by panoramic cameras. However, when the panoramic camera orientation is tilted, a non-upright panoramic image will be captured. Existing upright adjustment models focus on how to estimate more accurate camera orientation, and attribute image reconstruction to offline or post-processing tasks. To this end, we propose an online end-to-end network for upright adjustment. Our network is designed to reconstruct the image while finding the angle. Our network consists of three modules: orientation estimation, LUT online generation, and upright reconstruction. Direction estimation estimates the tilt angle of the panoramic image. Then, a converter block with upsampling function is designed to generate angle to LUT. This module can output corresponding online LUT for different input angles. Finally, a lightweight generative adversarial network (GAN) aims to generate upright images from shallow features. The experimental results show that in terms of angles, we have improved the accuracy of small angle errors. In terms of image reconstruction, In image reconstruction, we have achieved the first real-time online upright reconstruction of panoramic images using deep learning networks.

Real-Time Reconstruction of 3D Videos From Single-Photon LiDaR Data in the Presence of Obscurants

Single-photon methods are emerging as a key approach to 3D Imaging. This paper introduces a two step statistical based approach for real-time image reconstruction applicable to a transmission medium with extreme light scattering conditions. The first step is an optional target detection method to select informative pixels which have photons reflected from the target, hence allowing data compression. The second is a reconstruction algorithm that exploits data statistics and multiscale information to deliver clean depth and reflectivity images together with associated uncertainty maps. Both methods involve independent operations that are implemented in parallel on graphics processing units (GPUs), which enables real-time data processing of moving scenes at more than 50 depth frames per second for an image of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$128 \times 128$</tex-math></inline-formula> pixels. Comparisons with state-of-the-art algorithms on simulated and real underwater data demonstrate the benefit of the proposed framework for target detection, and for fast and robust depth estimation at multiple frames per second.

Swept-source endoscopic optical coherence tomography real-time imaging system based on GPU acceleration for axial megahertz high-speed scanning.

In order to solve the problem of image real-time processing and correction for high-speed endoscopic swept-source optical coherence tomography (SS-OCT), we highly optimize a computer-unified device architecture-based platform and use a field-programmable gate array to summarize the application experience. We use the Half-Sync/Half-Asyn mode to optimize memory in order to build a high-throughput data thread pool for CPU. We use asynchronous streaming architecture to multiplex multiple threads at high speed to accelerate data processing. At the same time, we design a rotary scanning position information encoding feedback module to suppress image drift, which can realize 25ns logic-timing sequence synchronization control through FPGA 40MHz clock. The maximum complete attainable axial-scan-processing rate (including memory transfer and display of B-scan frames) is 3.52 MHz for a 16-bit pixel depth and A-scans/s of 1024 pixels. To our knowledge, this is the fastest processing rate reported to date with a single-chip graphical processing unit for SS-OCT. Finally, the established high-speed SS-OCT is used to image mouse esophagus and human fingers, and the output images are stable. When the image size is 1024 × 1024 pixels, the real-time imaging rate is 200 frames per second. This paper develops a real-time image processing and reconstruction technology suitable for high-throughput SS-OCT systems, which can have high-density operation and efficient parallelism, while suppressing high-speed image drift. It lays the foundation for the non-destructive, in vivo, non-staining, fast and convenient early tumor diagnosis of high-speed endoscopic SS-OCT.

Real-Time Integration Center of Mass (riCOM) Reconstruction for 4D STEM.

A real-time image reconstruction method for scanning transmission electron microscopy (STEM) is proposed. With an algorithm requiring only the center of mass of the diffraction pattern at one probe position at a time, it is able to update the resulting image each time a new probe position is visited without storing any intermediate diffraction patterns. The results show clear features at high spatial frequency, such as atomic column positions. It is also demonstrated that some common post-processing methods, such as band-pass filtering, can be directly integrated in the real-time processing flow. Compared with other reconstruction methods, the proposed method produces high-quality reconstructions with good noise robustness at extremely low memory and computational requirements. An efficient, interactive open source implementation of the concept is further presented, which is compatible with frame-based, as well as event-based camera/file types. This method provides the attractive feature of immediate feedback that microscope operators have become used to, for example, conventional high-angle annular dark field STEM imaging, allowing for rapid decision-making and fine-tuning to obtain the best possible images for beam-sensitive samples at the lowest possible dose.

Higher-order correlation based real-time beamforming in photoacoustic imaging.

Although a delay-and-sum (DAS) beamformer is best suited for real-time photoacoustic (PA) image formation, the reconstructed images are often afflicted by noises, sidelobes, and other intense artifacts due to inaccurate assumptions in PA signal correlation. The present work aims to develop a reconstruction method that reduces the occurrence of sidelobes and artifacts and thus improves the reconstructed image quality or imaging performance. This beamformer is fundamentally based on higher-order signal correlation wherein a higher number of delayed PA signals-compared to conventional delay-multiply-and-sum (DMAS)-are combined and summed up. The proposed technique provides significant improvements in resolution, contrast, and signal-to-noise ratio (SNR) compared to traditional beamformers. For real-time implementation, the proposed algorithms were simplified, and their computational complexities were shrunk to the order of DAS [O(N)]. A GPU based study was also performed to validate the real-time capability of the proposed beamformers. For validation studies, both numerical simulation and experiments were conducted. Quantitative evaluation studies involving SNR, contrast ratio, generalized contrast-to-noise ratio, and FWHM demonstrate that the proposed higher-order DMAS beamformer is superior in PA image reconstruction. Conclusively, the proposed beamformer uniquely facilitates real-time PA image reconstruction with an achievable frame rate close to DAS and DMAS but with better imaging performance, which holds promise for real-time PA imaging and its clinical applications.

Accurate Range Migration for Fast Quantitative Fourier-Based Image Reconstruction With Monostatic Radar

Range migration (or range focusing) techniques are widely used in optical, acoustic, and microwave real-time image reconstruction methods. They have been successfully applied to far-field 3-D imaging where they rely on plane-wave assumptions, which ignore the data amplitude variation over the acquisition aperture. The accuracy of the plane-wave assumption, however, quickly degrades in close-range imaging, where amplitude variations are significant and where the range to the target is on the order of the range sampling step. Here, we present a range-focusing method of improved accuracy, which is applicable to both far-zone and close-range monostatic radar. It refocuses the system point-spread function (PSF) to any range location, taking into account both magnitude and phase changes. The approach can be applied with any Fourier-based imaging algorithm utilizing the Lippmann–Schwinger equation as the underlying scattering model. Here, it is validated through examples based on simulated and measured data where the images are reconstructed with quantitative microwave holography (QMH). QMH employs measured PSFs to achieve quantitative imaging in real-time. The proposed range-migration method is applicable with measured PSFs, too, leading to reduced system-calibration effort and the ability to focus an image at any desired range.