Image Reconstruction Pipeline Research Articles

ChromaFlash: Snapshot Hyperspectral Imaging Using Rolling Shutter Cameras

Hyperspectral imaging captures scene information across narrow, contiguous bands of the electromagnetic spectrum. Despite its proven utility in industrial and biomedical applications, its ubiquity has been limited by bulky form factors, slow capture times, and prohibitive costs. In this work, we propose a generalized approach to snapshot hyperspectral imaging that only requires a standard rolling shutter camera and wavelength-adjustable lighting. The crux of this approach entails using the rolling shutter as a spatiotemporal mask, varying incoming light quicker than the camera's frame rate in order for the captured image to contain rows of pixels illuminated at different wavelengths. An image reconstruction pipeline then converts this coded image into a complete hyperspectral image using sparse optimization. We demonstrate the feasibility of this approach by deploying a low-cost system called ChromaFlash, which uses a smartphone's camera for image acquisition and a series of LEDs to change the scene's illumination. We evaluated ChromaFlash through simulations on two public hyperspectral datasets and assessed its spatial and spectral accuracy across various system parameters. We also tested the real-world performance of our prototype by capturing diverse scenes under varied ambient lighting conditions. In both experiments, ChromaFlash outperformed state-of-the-art techniques that use deep learning to convert RGB images into hyperspectral ones, achieving snapshot performance not demonstrated by prior attempts at accessible hyperspectral imaging.

Quantum radio astronomy: Data encodings and quantum image processing

We explore applications of quantum computing for radio interferometry and astronomy using recent developments in quantum image processing. We evaluate the suitability of different quantum image representations using a toy quantum computing image reconstruction pipeline, and compare its performance to the classical computing counterpart. For identifying and locating bright radio sources, quantum computing can offer an exponential speedup over classical algorithms, even when accounting for data encoding cost and repeated circuit evaluations. We also propose a novel variational quantum computing algorithm for self-calibration of interferometer visibilities, and discuss future developments and research that would be necessary to make quantum computing for radio astronomy a reality.

Phantom-based assessment of a deep-learning MR image reconstruction pipeline to inform optimisation of clinical protocols

Phantom-based assessment of a deep-learning MR image reconstruction pipeline to inform optimisation of clinical protocols

Fast optical coherence tomography angiography image acquisition and reconstruction pipeline for skin application.

Traditional high-quality OCTA images require multi-repeated scans (e.g., 4-8 repeats) in the same position, which may cause the patient to be uncomfortable. We propose a deep-learning-based pipeline that can extract high-quality OCTA images from only two-repeat OCT scans. The performance of the proposed image reconstruction U-Net (IRU-Net) outperforms the state-of-the-art UNet vision transformer and UNet in OCTA image reconstruction from a two-repeat OCT signal. The results demonstrated a mean peak-signal-to-noise ratio increased from 15.7 to 24.2; the mean structural similarity index measure improved from 0.28 to 0.59, while the OCT data acquisition time was reduced from 21 seconds to 3.5 seconds (reduced by 83%).

Live 4D-OCT denoising with self-supervised deep learning

By providing three-dimensional visualization of tissues and instruments at high resolution, live volumetric optical coherence tomography (4D-OCT) has the potential to revolutionize ophthalmic surgery. However, the necessary imaging speed is accompanied by increased noise levels. A high data rate and the requirement for minimal latency impose major limitations for real-time noise reduction. In this work, we propose a low complexity neural network for denoising, directly incorporated into the image reconstruction pipeline of a microscope-integrated 4D-OCT prototype with an A-scan rate of 1.2 MHz. For this purpose, we trained a blind-spot network on unpaired OCT images using a self-supervised learning approach. With an optimized U-Net, only a few milliseconds of additional latency were introduced. Simultaneously, these architectural adaptations improved the numerical denoising performance compared to the basic setup, outperforming non-local filtering algorithms. Layers and edges of anatomical structures in B-scans were better preserved than with Gaussian filtering despite comparable processing time. By comparing scenes with and without denoising employed, we show that neural networks can be used to improve visual appearance of volumetric renderings in real time. Enhancing the rendering quality is an important step for the clinical acceptance and translation of 4D-OCT as an intra-surgical guidance tool.

LRR-CED: low-resolution reconstruction-aware convolutional encoder–decoder network for direct sparse-view CT image reconstruction

Objective. Sparse-view computed tomography (CT) reconstruction has been at the forefront of research in medical imaging. Reducing the total x-ray radiation dose to the patient while preserving the reconstruction accuracy is a big challenge. The sparse-view approach is based on reducing the number of rotation angles, which leads to poor quality reconstructed images as it introduces several artifacts. These artifacts are more clearly visible in traditional reconstruction methods like the filtered-backprojection (FBP) algorithm. Approach. Over the years, several model-based iterative and more recently deep learning-based methods have been proposed to improve sparse-view CT reconstruction. Many deep learning-based methods improve FBP-reconstructed images as a post-processing step. In this work, we propose a direct deep learning-based reconstruction that exploits the information from low-dimensional scout images, to learn the projection-to-image mapping. This is done by concatenating FBP scout images at multiple resolutions in the decoder part of a convolutional encoder–decoder (CED). Main results. This approach is investigated on two different networks, based on Dense Blocks and U-Net to show that a direct mapping can be learned from a sinogram to an image. The results are compared to two post-processing deep learning methods (FBP-ConvNet and DD-Net) and an iterative method that uses a total variation (TV) regularization. Significance. This work presents a novel method that uses information from both sinogram and low-resolution scout images for sparse-view CT image reconstruction. We also generalize this idea by demonstrating results with two different neural networks. This work is in the direction of exploring deep learning across the various stages of the image reconstruction pipeline involving data correction, domain transfer and image improvement.

Self-navigated prospective motion correction for 3D-EPI acquisition.

Although 3D EPI is more susceptible to motion artifacts than 2D EPI, it presents some benefits for functional MRI, including the absence of spin-history artifacts, greater potential for parallel imaging acceleration, and better functional sensitivity in high-resolution imaging. Here we present a self-navigated 3D-EPI sequence suitable for prospective motion-corrected functional MRI without additional hardware or pulses. For each volume acquisition, the first 24 of the 52 partitions being acquired are accumulated to a new feedback block that was added to the image reconstruction pipeline. After zero-filling the remaining partitions, the feedback block constructs a volumetric self-navigator (vSNav), co-registers it to the reference vSNav acquired during the first volume acquisition, and sends motion estimates to the sequence. The sequence then updates its FOV and acquires subsequent partitions with the adjusted FOV, until the next update is received. The sequence was validated without and with intentional motion in phantom and in vivo on a 3T Skyra. For phantom scans, the FOV was updated 0.704 s after acquisition of the vSNav partitions, and for in vivo scans after 0.768 s. Both phantom and in vivo data demonstrated stable motion estimates in the absence of motion. For in vivo acquisitions, prospective head-pose estimates using the vSNav's and retrospective estimates with FLIRT (FMRIB's Linear Image Registration Tool) agreed to within 0.23 mm (< 10% of the slice thickness) and 0.14° in all directions. Depending when motion occurs during a volume acquisition, the proposed method fully corrects the FOV and recovers image quality within one volume acquisition.

Integrated Array Tomography for 3D Correlative Light and Electron Microscopy.

Volume electron microscopy (EM) of biological systems has grown exponentially in recent years due to innovative large-scale imaging approaches. As a standalone imaging method, however, large-scale EM typically has two major limitations: slow rates of acquisition and the difficulty to provide targeted biological information. We developed a 3D image acquisition and reconstruction pipeline that overcomes both of these limitations by using a widefield fluorescence microscope integrated inside of a scanning electron microscope. The workflow consists of acquiring large field of view fluorescence microscopy (FM) images, which guide to regions of interest for successive EM (integrated correlative light and electron microscopy). High precision EM-FM overlay is achieved using cathodoluminescent markers. We conduct a proof-of-concept of our integrated workflow on immunolabelled serial sections of tissues. Acquisitions are limited to regions containing biological targets, expediting total acquisition times and reducing the burden of excess data by tens or hundreds of GBs.

MR-guided motion-corrected PET image reconstruction for cardiac PET-MR.

Simultaneous PET-MR imaging has shown potential for the comprehensive assessment of myocardial health from a single examination. Furthermore, MR-derived respiratory motion information has been shown to improve PET image quality by incorporating this information into the PET image reconstruction. Separately, MR-based anatomically guided PET image reconstruction has been shown to perform effective denoising, but this has been so far demonstrated mainly in brain imaging. To date the combined benefits of motion compensation and anatomical guidance have not been demonstrated for myocardial PET-MR imaging. This work addresses this by proposing a single cardiac PET-MR image reconstruction framework which fully utilises MR-derived information to allow both motion compensation and anatomical guidance within the reconstruction. Methods: Fifteen patients underwent a 18F-FDG cardiac PET-MR scan with a previously introduced acquisition framework. The MR data processing and image reconstruction pipeline produces respiratory motion fields and a high-resolution respiratory motion-corrected MR image with good tissue contrast. This MR-derived information was then included in a respiratory motion-corrected, cardiac-gated, anatomically guided image reconstruction of the simultaneously acquired PET data. Reconstructions were evaluated by measuring myocardial contrast and noise and compared to images from several comparative intermediate methods using the components of the proposed framework separately. Results: Including respiratory motion correction, cardiac gating, and anatomical guidance significantly increased contrast. In particular, myocardium-to-blood pool contrast increased by 143% on average (p<0.0001) compared to conventional uncorrected, non-guided PET images. Furthermore, anatomical guidance significantly reduced image noise compared to non-guided image reconstruction by 16.1% (p<0.0001). Conclusion: The proposed framework for MR-derived motion compensation and anatomical guidance of cardiac PET data was shown to significantly improve image quality compared to alternative reconstruction methods. Each component of the reconstruction pipeline was shown to have a positive impact on the final image quality. These improvements have the potential to improve clinical interpretability and diagnosis based on cardiac PET-MR images.

PET Image Reconstruction Using a Cascading Back-Projection Neural Network

Positron emission tomography (PET) imaging is a noninvasive technique that makes it possible to probe biological metabolic processes in vivo . However, PET image reconstruction is challenging due to the ill-posedness of the inverse problem. Many image reconstruction methods have been proposed over the past few years to improve diagnostic performance. However, most of these methods can compromise the reconstruction of important high-frequency structural details after aggressive denoising. To address this problem, in this work, we propose a novel deep learning method that reconstructs PET images using a cascading back-projection neural network (bpNet). This network consists of a domain translation operation, which acts as prior knowledge, cascaded with a modified encoder-decoder network. The image reconstruction pipeline ranges from the sinogram to the back-projection image and then to the PET image. Quantitative results from simulation data and Derenzo phantom experiments with the small animal PET prototype system developed in our laboratory clearly demonstrate that our proposed method provides favorable reconstructed image quality, especially for low-count PET image reconstruction.

Rapid dealiasing of undersampled, non-Cartesian cardiac perfusion images using U-net.

Compressed sensing (CS) is a promising method for accelerating cardiac perfusion MRI to achieve clinically acceptable image quality with high spatial resolution (1.6 × 1.6 × 8 mm3 ) and extensive myocardial coverage (6-8 slices per heartbeat). A major disadvantage of CS is its relatively lengthy processing time (~8 min per slice with 64 frames using a graphics processing unit), thereby making it impractical for clinical translation. The purpose of this study was to implement and test whether an image reconstruction pipeline including a neural network is capable of reconstructing 6.4-fold accelerated, non-Cartesian (radial) cardiac perfusion k-space data at least 10 times faster than CS, without significant loss in image quality. We implemented a 3D (2D + time) U-Net and trained it with 132 2D + time datasets (coil combined, zero filled as input; CS reconstruction as reference) with 64 time frames from 28 patients (8448 2D images in total). For testing, we used 56 2D + time coil-combined, zero-filled datasets (3584 2D images in total) from 12 different patients as input to our trained U-Net, and compared the resulting images with CS reconstructed images using quantitative metrics of image quality and visual scores (conspicuity of wall enhancement, noise, artifacts; each score ranging from 1 (worst) to 5 (best), with 3 defined as clinically acceptable) evaluated by readers. Including pre- and post-processing steps, compared with CS, U-Net significantly reduced the reconstruction time by 14.4-fold (32.1 ± 1.4 s for U-Net versus 461.3 ± 16.9 s for CS, p < 0.001), while maintaining high data fidelity (structural similarity index = 0.914 ± 0.023, normalized root mean square error = 1.7 ± 0.3%, identical mean edge sharpness of 1.2 mm). The median visual summed score was not significantly different (p = 0.053) between CS (14; interquartile range (IQR) = 0.5) and U-Net (12; IQR = 0.5). This study shows that the proposed pipeline with a U-Net is capable of reconstructing 6.4-fold accelerated, non-Cartesian cardiac perfusion k-space data 14.4 times faster than CS, without significant loss in data fidelity or image quality.

Filling Adeno-Associated Virus Capsids: Estimating Success by Cryo-Electron Microscopy.

Adeno-associated viruses (AAVs) have been employed successfully as gene therapy vectors in treating various genetic diseases for almost two decades. However, transgene packaging is usually imperfect, and developing a rapid and accurate method for measuring the proportion of DNA encapsidation is an important step for improving the downstream process of large scale vector production. In this study, we used two-dimensional class averages and three-dimensional classes, intermediate outputs in the single particle cryo-electron microscopy (cryo-EM) image reconstruction pipeline, to determine the proportion of DNA-packaged and empty capsid populations. Two different preparations of AAV3 were analyzed to estimate the minimum number of particles required to be sampled by cryo-EM in order for robust calculation of the proportion of the full versus empty capsids in any given sample. Cost analysis applied to the minimum amount of data required for a valid ratio suggests that cryo-EM is an effective approach to analyze vector preparations.

A cone-beam X-ray computed tomography data collection designed for machine learning

Unlike previous works, this open data collection consists of X-ray cone-beam (CB) computed tomography (CT) datasets specifically designed for machine learning applications and high cone-angle artefact reduction. Forty-two walnuts were scanned with a laboratory X-ray set-up to provide not only data from a single object but from a class of objects with natural variability. For each walnut, CB projections on three different source orbits were acquired to provide CB data with different cone angles as well as being able to compute artefact-free, high-quality ground truth images from the combined data that can be used for supervised learning. We provide the complete image reconstruction pipeline: raw projection data, a description of the scanning geometry, pre-processing and reconstruction scripts using open software, and the reconstructed volumes. Due to this, the dataset can not only be used for high cone-angle artefact reduction but also for algorithm development and evaluation for other tasks, such as image reconstruction from limited or sparse-angle (low-dose) scanning, super resolution, or segmentation.

On the benefit of adversarial training for monocular depth estimation

In this paper we address the benefit of adding adversarial training to the task of monocular depth estimation. A model can be trained in a self-supervised setting on stereo pairs of images, where depth (disparities) are an intermediate result in a right-to-left image reconstruction pipeline. For the quality of the image reconstruction and disparity prediction, a combination of different losses is used, including L1 image reconstruction losses and left–right disparity smoothness. These are local pixel-wise losses, while depth prediction requires global consistency. Therefore, we extend the self-supervised network to become a Generative Adversarial Network (GAN), by including a discriminator which should tell apart reconstructed (fake) images from real images. We evaluate Vanilla GANs, LSGANs and Wasserstein GANs in combination with different pixel-wise reconstruction losses. Based on extensive experimental evaluation, we conclude that adversarial training is beneficial if and only if the reconstruction loss is not too constrained. Even though adversarial training seems promising because it promotes global consistency, non-adversarial training outperforms (or is on par with) any method trained with a GAN when a constrained reconstruction loss is used in combination with batch normalisation. Based on the insights of our experimental evaluation we obtain state-of-the art monocular depth estimation results by using batch normalisation and different output scales.

All-passive pixel super-resolution of time-stretch imaging

Based on image encoding in a serial-temporal format, optical time-stretch imaging entails a stringent requirement of state-of-the-art fast data acquisition unit in order to preserve high image resolution at an ultrahigh frame rate — hampering the widespread utilities of such technology. Here, we propose a pixel super-resolution (pixel-SR) technique tailored for time-stretch imaging that preserves pixel resolution at a relaxed sampling rate. It harnesses the subpixel shifts between image frames inherently introduced by asynchronous digital sampling of the continuous time-stretch imaging process. Precise pixel registration is thus accomplished without any active opto-mechanical subpixel-shift control or other additional hardware. Here, we present the experimental pixel-SR image reconstruction pipeline that restores high-resolution time-stretch images of microparticles and biological cells (phytoplankton) at a relaxed sampling rate (≈2–5 GSa/s)—more than four times lower than the originally required readout rate (20 GSa/s) — is thus effective for high-throughput label-free, morphology-based cellular classification down to single-cell precision. Upon integration with the high-throughput image processing technology, this pixel-SR time-stretch imaging technique represents a cost-effective and practical solution for large scale cell-based phenotypic screening in biomedical diagnosis and machine vision for quality control in manufacturing.

Abstract 235: Integration of Patient-specific Computational Hemodynamics and Vessel Wall Shear Stress Into MRI Diagnosis of Vascular Diseases

The research objective is to expand the capability of current MRI imaging technique in assessing the overall risk and predicted outcomes of atherosclerotic diseases through the quantification of individual patient-specific hemodynamics, including flow, pressure, and wall-shear stress. A unique computational modeling technique, named InVascular, is integrated directly into clinical MRI scanners as the extension of the image reconstruction and post-processing pipeline so that velocity, pressure, vorticity, and WSS can be available immediately with other diagnostic images. InVascular is a unified and GPU accelerated computation platform to model and simulate patient-specific hemodynamics and flow-vessel interaction based on MRI imaging data. In this study, we validate the efficiency and accuracy of InVascular through quantitative hemodynamics in vertebral and carotid arteries. A group of five volunteers participated in the scanning of high resolution time-of-flight (TOF) and low resolution electrocardiogram (ECG) gated phase contrast (PC) MR angiogram (MRA) images. For each case, InVascular successively processes the images to get vessel geometry from TOF MRA and velocity slices from PC MRA and solve the fluid dynamics inside the carotid arteries with PC MRA measured velocity at the inlet and outlet (Fig. 1 a-c). The velocity profiles from Invascular and PC MRA are compared at the same location (Fig. 1 d-g ). We conclude that integration of MRAs and InVascular can well captured the velocity fields as MRI measures. InVascular can provide quantitative pressure and WSS (Fig. 1h ) information as well.

Dynamic structure and protein expression of the live embryonic heart captured by 2-photon light sheet microscopy and retrospective registration.

We present an imaging and image reconstruction pipeline that captures the dynamic three-dimensional beating motion of the live embryonic zebrafish heart at subcellular resolution. Live, intact zebrafish embryos were imaged using 2-photon light sheet microscopy, which offers deep and fast imaging at 70 frames per second, and the individual optical sections were assembled into a full 4D reconstruction of the beating heart using an optimized retrospective image registration algorithm. This imaging and reconstruction platform permitted us to visualize protein expression patterns at endogenous concentrations in zebrafish gene trap lines.

Parallel computing on graphics processing units and heterogeneous platforms

Parallel computing on graphics processing units and heterogeneous platforms

Implementation of a generalized heterogeneous image reconstruction system for clinical magnetic resonance

SummaryThis paper describes the development of a novel online, heterogeneous image reconstruction system for magnetic resonance data. The system integrates an external computer equipped with a graphic processing unit card into the magnetic resonance scanner's image reconstruction pipeline. The system promotes fast online reconstruction for computationally intensive algorithms making them feasible in a busy clinical service. Analysis and improvement of execution time of the complex, iterative reconstruction algorithm is presented. Specifically, optimizations were examined that allowed fast reconstruction of data that were acquired in an arbitrary manner. The system was successfully used in research and clinical studies requiring high data throughput. Copyright © 2014 John Wiley &amp; Sons, Ltd.

Generic software framework for a line-buffer-based image processing pipeline

The majority of the current affordable mobile devices contain a camera with simple optics and a low-cost camera sensor. In these devices, the quality of the captured images is made acceptable with various image processing algorithms that together form an image reconstruction pipeline. The best performance is often achieved with a hardware pipeline, but software implementations can be preferred to minimize production costs and to maximize flexibility. This paper presents a generic software framework for a line-buffer-based image reconstruction pipeline. The presented framework is capable of operating in low-memory environments and significantly eases algorithm insertions, changes of processing order, and other pipeline management tasks. The savings in development time can be even months. In addition, our experiments show that it offers over 99% memory savings compared with traditional implementations using a ping-pong buffer scheme with full-sized image buffers. The implemented framework also enhances processing performance due to better cache usage and increases flexibility with various pipeline configurations.