Computational Imaging Approach Research Articles

Design of on-chip coded high-resolution 2D imaging via 3D compressed sensing

This manuscript introduces a computational imaging approach where a static coded aperture is integrated with the image sensor to replace the extra Fourier optical elements or dynamic modulation in the previous computational high/super-resolution tasks. A two-dimensional (2D) high-resolution image with N×N pixels is formulated as a three-dimensional (3D) low-resolution cube with Nk×Nk×k2 voxels in the forward model where N is the spatial resolution and k is a factor, respectively. Thus, the 2D image reconstruction can be performed with a 3D compressed sensing model in the snapshot-compressive-imaging format which has been mathematically proved to be convergent. Our proposed method successfully performs on the scaling factors such as 4× enlargement and the PSNR gains for natural images, remote sensing images and infrared image are 1.8 dB, 3.5 dB and 1.3 dB, respectively with only 1000 paired training images.

Design and realization of a miniaturized high resolution computed tomography imaging spectrometer

The computed tomography imaging spectrometer (CTIS) is a relatively unknown snapshot hyperspectral camera. It utilizes computational imaging approaches to gain the hyperspectral image from a spatio-spectral smeared sensor image. We present a strongly miniaturized system with a dimension of only 36 × 40.5 × 52.8 mm and a diagonal field of view of 29°. We achieve this using a Galilean beam expander and a combination of off-the-shelf lenses, a highly aspherical imaging system from a commercial smartphone, and a 13 MP monochrome smartphone image sensor. The reconstructed hyperspectral image has a spatial resolution of 400 × 300 pixel with 39 spectral channels.

Computational Imaging to Compensate for Soft-Tissue Deformations in Image-Guided Breast Conserving Surgery.

During breast conserving surgery (BCS), magnetic resonance (MR) images aligned to accurately display intraoperative lesion locations can offer improved understanding of tumor extent and position relative to breast anatomy. Unfortunately, even under consistent supine conditions, soft tissue deformation compromises image-to-physical alignment and results in positional errors. A finite element inverse modeling technique has been developed to nonrigidly register preoperative supine MR imaging data to the surgical scene for improved localization accuracy during surgery. Registration is driven using sparse data compatible with acquisition during BCS, including corresponding surface fiducials, sparse chest wall contours, and the intra-fiducial skin surface. Deformation predictions were evaluated at surface fiducial locations and subsurface tissue features that were expertly identified and tracked. Among n = 7 different human subjects, an average of 22 ± 3 distributed subsurface targets were analyzed in each breast volume. The average target registration error (TRE) decreased significantly when comparing rigid registration to this nonrigid approach (10.4 ± 2.3 mm vs 6.3 ± 1.4 mm TRE, respectively). When including a single subsurface feature as additional input data, the TRE significantly improved further (4.2 ± 1.0 mm TRE), and in a region of interest within 15 mm of a mock biopsy clip TRE was 3.9 ± 0.9 mm. These results demonstrate accurate breast deformation estimates based on sparse-data-driven model predictions. The data suggest that a computational imaging approach can account for image-to-surgery shape changes to enhance surgical guidance during BCS.

Snapshot compressive imaging based digital image correlation: temporally super-resolved full-resolution deformation measurement.

The limited throughput of a digital image correlation (DIC) system hampers measuring deformations at both high spatial resolution and high temporal resolution. To address this dilemma, in this paper we propose to integrate snapshot compressive imaging (SCI)-a recently proposed computational imaging approach-into DIC for high-speed, high-resolution deformation measurement. Specifically, an SCI-DIC system is established to encode a sequence of fast changing speckle patterns into a snapshot and a high-accuracy speckle decompress SCI (Sp-DeSCI) algorithm is proposed for computational reconstruction of the speckle sequence. To adapt SCI reconstruction to the unique characteristics of speckle patterns, we propose three techniques under SCI reconstruction framework to secure high-precision reconstruction, including the normalized sum squared difference criterion, speckle-adaptive patch search strategy, and adaptive group aggregation. For efficacy validation of the proposed Sp-DeSCI, we conducted extensive simulated experiments and a four-point bending SCI-DIC experiment on real data. Both simulation and real experiments verify that the Sp-DeSCI successfully removes the deviations of reconstructed speckles in DeSCI and provides the highest displacement accuracy among existing algorithms. The SCI-DIC system together with the Sp-DeSCI algorithm can offer temporally super-resolved deformation measurement at full spatial resolution, and can potentially replace conventional high-speed DIC in real measurements.

Tomographic single pixel spatial frequency projection imaging

Conventional methods for three-dimensional (3D) imaging frequently rely on voxel-by-voxel data acquisition, which restricts the range of specimens in which they can be effectively employed. While advances in imaging technology now permit the routine acquisition of 3D images approaching video rates, there are other limitations to image formation in fluorescent microscopy that prohibit studies in large volume samples, highly scattering media, and dynamic environments. Some approaches to 3D image collection circumvent this need by the use of tomographic imaging, where sub-3D projections are collected at varying illumination angles and reconstructed through an inversion algorithm to compute an estimate of the 3D fluorophore distribution. Many such methods rely on spatially coherent light, and thus prohibit the use of fluorescent light. By employing unique spatio-temporally varying illumination patterns in conjunction with computational imaging approaches to image reconstruction, we show that some limitations of laser scanning and wide-field imaging can be overcome. We outline several approaches that utilize tomographic projections with patterned illumination to collect 3D image data. All three dimensional optical imaging exploits projection of the desired 3D information into a lower-dimensional subspace, and then a full three dimensional object is estimated from these data. We discuss a number of such single pixel strategies that project object information onto a zero-dimensional, usually a power, measurement. Further, we outline computational image reconstruction approaches that enhance the object estimates by employing a forward model for the image formation process.

The Modulo Radon Transform: Theory, Algorithms, and Applications

Recently, experiments have been reported where researchers were able to perform high dynamic range (HDR) tomography in a heuristic fashion, by fusing multiple tomographic projections. This approach to HDR tomography has been inspired by HDR photography and inherits the same disadvantages. Taking a computational imaging approach to the HDR tomography problem, we here suggest a new model based on the Modulo Radon Transform (MRT), which we rigorously introduce and analyze. By harnessing a joint design between hardware and algorithms, we present a single-shot HDR tomography approach, which to our knowledge, is the only approach that is backed by mathematical guarantees. On the hardware front, instead of recording the Radon Transform projections that my potentially saturate, we propose to measure modulo values of the same. This ensures that the HDR measurements are folded into a lower dynamic range. On the algorithmic front, our recovery algorithms reconstruct the HDR images from folded measurements. Beyond mathematical aspects such as injectivity and inversion of the MRT for different scenarios including band-limited and approximately compactly supported images, we also provide a first proof-of-concept demonstration. To do so, we implement MRT by experimentally folding tomographic measurements available as an open source data set using our custom designed modulo hardware. Our reconstruction clearly shows the advantages of our approach for experimental data. In this way, our MRT based solution paves a path for HDR acquisition in a number of related imaging problems.

Fourier ptychography multi-parameunter neural network with composite physical priori optimization.

Fourier ptychography microscopy(FPM) is a recently developed computational imaging approach for microscopic super-resolution imaging. By turning on each light-emitting-diode (LED) located on different position on the LED array sequentially and acquiring the corresponding images that contain different spatial frequency components, high spatial resolution and quantitative phase imaging can be achieved in the case of large field-of-view. Nevertheless, FPM has high requirements for the system construction and data acquisition processes, such as precise LEDs position, accurate focusing and appropriate exposure time, which brings many limitations to its practical applications. In this paper, inspired by artificial neural network, we propose a Fourier ptychography multi-parameter neural network (FPMN) with composite physical prior optimization. A hybrid parameter determination strategy combining physical imaging model and data-driven network training is proposed to recover the multi layers of the network corresponding to different physical parameters, including sample complex function, system pupil function, defocus distance, LED array position deviation and illumination intensity fluctuation, etc. Among these parameters, LED array position deviation is recovered based on the features of brightfield to darkfield transition low-resolution images while the others are recovered in the process of training of the neural network. The feasibility and effectiveness of FPMN are verified through simulations and actual experiments. Therefore FPMN can evidently reduce the requirement for practical applications of FPM.

Design and realization of a miniaturized high resolution computed tomography imaging spectrometer

The computed tomography imaging spectrometer (CTIS) is a relatively unknown snapshot hyperspectral camera. It utilizes computational imaging approaches to gain the hyperspectral image from a spatiospectral smeared sensor image. We present a strongly miniaturized system with a dimension of only 36mm x 40.5mm x 52.8mm and a diagonal field of view of 29°. We achieve this using a Galilean beam expander and a combination of off-the-shelf lenses, a highly aspherical imaging system from a commercial smartphone and a 13 MP monochrome smartphone image sensor. The reconstructed hyperspectral image has a spatial resolution of 400 x 300 pixel with 39 spectral channels.

Acousto-optic ptychography

Acousto-optic imaging (AOI) enables optical-contrast imaging deep inside scattering samples via localized ultrasound-modulation of scattered light. While AOI allows optical investigations at depths, its imaging resolution is inherently limited by the ultrasound wavelength, prohibiting microscopic investigations. Here, we propose a computational imaging approach that allows optical diffraction-limited imaging using a conventional AOI system. We achieve this by extracting diffraction-limited imaging information from speckle correlations in the conventionally detected ultrasound-modulated scattered-light fields. Specifically, we identify that since “memory-effect” speckle correlations allow estimation of the Fourier magnitude of the field inside the ultrasound focus, scanning the ultrasound focus enables robust diffraction-limited reconstruction of extended objects using ptychography (i.e., we exploit the ultrasound focus as the scanned spatial-gate probe required for ptychographic phase retrieval). Moreover, we exploit the short speckle decorrelation-time in dynamic media, which is usually considered a hurdle for wavefront-shaping- based approaches, for improved ptychographic reconstruction. We experimentally demonstrate noninvasive imaging of targets that extend well beyond the memory-effect range, with a 40-times resolution improvement over conventional AOI.

Holographic optical field recovery using a regularized untrained deep decoder network

Image reconstruction using minimal measured information has been a long-standing open problem in many computational imaging approaches, in particular in-line holography. Many solutions are devised based on compressive sensing (CS) techniques with handcrafted image priors or supervised deep neural networks (DNN). However, the limited performance of CS methods due to lack of information about the image priors and the requirement of an enormous amount of per-sample-type training resources for DNNs has posed new challenges over the primary problem. In this study, we propose a single-shot lensless in-line holographic reconstruction method using an untrained deep neural network which is incorporated with a physical image formation algorithm. We demonstrate that by modifying a deep decoder network with simple regularizers, a Gabor hologram can be inversely reconstructed via a minimization process that is constrained by a deep image prior. The outcoming model allows to accurately recover the phase and amplitude images without any training dataset, excess measurements, or specific assumptions about the object’s or the measurement’s characteristics.

Alveolar epithelial cell fate is maintained in a spatially restricted manner to promote lung regeneration after acute injury.

SUMMARYAlveolar epithelial type 2 (AT2) cells integrate signals from multiple molecular pathways to proliferate and differentiate to drive regeneration of the lung alveolus. Utilizing in vivo genetic and ex vivo organoid models, we investigated the role of Fgfr2 signaling in AT2 cells across the lifespan and during adult regeneration after influenza infection. We show that, although dispensable for adult homeostasis, Fgfr2 restricts AT2 cell fate during postnatal lung development. Using an unbiased computational imaging approach, we demonstrate that Fgfr2 promotes AT2 cell proliferation and restrains differentiation in actively regenerating areas after injury. Organoid assays reveal that Fgfr2-deficient AT2 cells remain competent to respond to multiple parallel proliferative inputs. Moreover, genetic blockade of AT2 cell cytokinesis demonstrates that cell division and differentiation are uncoupled during alveolar regeneration. These data reveal that Fgfr2 maintains AT2 cell fate, balancing proliferation and differentiation during lung alveolar regeneration.

Neural network model assisted Fourier ptychography with Zernike aberration recovery and total variation constraint.

.Significance: Fourier ptychography (FP) is a computational imaging approach that achieves high-resolution reconstruction. Inspired by neural networks, many deep-learning-based methods are proposed to solve FP problems. However, the performance of FP still suffers from optical aberration, which needs to be considered.Aim: We present a neural network model for FP reconstructions that can make proper estimation toward aberration and achieve artifact-free reconstruction.Approach: Inspired by the iterative reconstruction of FP, we design a neural network model that mimics the forward imaging process of FP via TensorFlow. The sample and aberration are considered as learnable weights and optimized through back-propagation. Especially, we employ the Zernike terms instead of aberration to decrease the optimization freedom of pupil recovery and perform a high-accuracy estimation. Owing to the auto-differentiation capabilities of the neural network, we additionally utilize total variation regularization to improve the visual quality.Results: We validate the performance of the reported method via both simulation and experiment. Our method exhibits higher robustness against sophisticated optical aberrations and achieves better image quality by reducing artifacts.Conclusions: The forward neural network model can jointly recover the high-resolution sample and optical aberration in iterative FP reconstruction. We hope our method that can provide a neural-network perspective to solve iterative-based coherent or incoherent imaging problems.

Learning Wavefront Coding for Extended Depth of Field Imaging.

Depth of field is an important factor of imaging systems that highly affects the quality of the acquired spatial information. Extended depth of field (EDoF) imaging is a challenging ill-posed problem and has been extensively addressed in the literature. We propose a computational imaging approach for EDoF, where we employ wavefront coding via a diffractive optical element (DOE) and we achieve deblurring through a convolutional neural network. Thanks to the end-to-end differentiable modeling of optical image formation and computational post-processing, we jointly optimize the optical design, i.e., DOE, and the deblurring through standard gradient descent methods. Based on the properties of the underlying refractive lens and the desired EDoF range, we provide an analytical expression for the search space of the DOE, which is instrumental in the convergence of the end-to-end network. We achieve superior EDoF imaging performance compared to the state of the art, where we demonstrate results with minimal artifacts in various scenarios, including deep 3D scenes and broadband imaging.

Imaging reconstruction comparison of different ghost imaging algorithms

As an indirect and computational imaging approach, imaging reconstruction efficiency is critical for ghost imaging (GI). Here, we compare different GI algorithms, including logarithmic GI and exponential GI we proposed, by numerically analysing their imaging reconstruction efficiency and error tolerance. Simulation results show that compressive GI algorithm has the highest reconstruction efficiency due to its global optimization property. Error tolerance studies further manifest that compressive GI and exponential GI are sensitive to the error ratio. By replacing the bucket input of compressive GI with different bucket object signal functions, we integrate compressive GI with other GI algorithms and discuss their imaging efficiency. With the combination between the differential GI (or normalized GI) and compressive GI, both reconstruction efficiency and error tolerance will present the best performance. Moreover, an optical encryption is proposed by combining logarithmic GI, exponential GI and compressive GI, which can enhance the encryption security based on GI principle.

X-ray imaging of fast dynamics with single-pixel detector.

We demonstrate experimentally the ability to use a single-pixel detector for two-dimensional high-resolution x-ray imaging of fast dynamics. We image the rotation of a spinning chopper at 100 kHz and at spatial resolution of about 40 microns by using the computational ghost imaging approach. The technique we develop can be used for the imaging of fast dynamics of periodic and periodically stimulated effects with a large field of view and at low dose.

Computational imaging with low-order OAM beams at microwave frequencies

With the distinguished wavefront characteristics of vortex electromagnetic wave carrying orbital angular momentum (OAM), the OAM beams have been exploited for radar imaging in recent years. In this paper, the computational imaging model is built using OAM wave, which enables the target reconstruction with limited measurements. The measurement matrix is designed, and the target reconstruction method is proposed in the Cartesian coordinate. Simulation results indicate that the proposed computational imaging approach is robust against noise influence. Furthermore, the outdoor experiments are carried out, for the first time, to validate the super-resolution imaging ability of this novel technique. Experimental results show good agreement with theoretical analyses. This work can advance the development of OAM-based sensing technology.

Super-resolution reconstruction of real infrared images acquired with unmanned aerial vehicle.

Super-resolution (SR) technology provides a far promising computational imaging approach in obtaining a high-resolution (HR) image (or image sequences) from observed multiple low-resolution (LR) images by incorporating complementary information. In this paper, a three-stage SR method is proposed to generate a HR image from infrared (IR) LR Images acquired with Unmanned Aerial Vehicle (UAV). The proposed method integrates a high-level image capturing process and a low-level SR process. In this integrated process, we incorporate UAV path optimization, sub-pixel image registration, and sparseness constraint into a computational imaging framework of a region of interest (ROI). To refine ROI complementary feathers, we design an optimal flight control scheme to acquire adequate image sequences from multi-angles. In particular, a phase correlation approach achieving reliable sub-pixel image feature matching is adapted, on the basis of which an effective sparseness regularization model is built to enhance the fine structures of the IR image. Unlike most traditional multiple-frame SR algorithms that mainly focus on signal processing and achieve good performances when using standard test datasets, the performed experiments with real-life IR sequences indicate the three-stage SR method can also deal with practical LR IR image sequences collected by UAVs. The experimental results demonstrate that the proposed method is capable of generating HR images with good performance in terms of edge preservation and detail enhancement.

Multi-frame motion deblurring using coded exposure imaging with complementary fluttering sequences

Motion blurring is one of the most common but unpleasant phenomenons along with the camera imaging process, resulting in image resolution degradation. Coded exposure technique is an effective computational imaging approach to handle with this problem by introducing extra shutter control within the exposure time and help the deblurring problem become well-posed. In this paper, a multi-frame motion deblurring framework is presented using coded exposure imaging with complementary fluttering sequences. A set of complementary sequences are well designed to compensate for the spatial losses due to motion blurring in each frame based on the complementary theory of Golay matrix pair. Hadamard matrix is served as the Golay sequence pair to generate complementary fluttering sequences to capture multi-frame images for the following deblurring processing. The joint modulation transfer function (MTF) of the complementary code sequences has a more flat spectrum curve with smaller variance value and higher MTF value. For multi-frame deblurring implementation, a unified multi-image deblurring framework is performed by introducing Total Variation (TV) regularization prior with each selected code sequence in the complementary set integrated in the energy function. A real imaging system is built and various groups of synthetic and real-world imaging experiments are carried out to verify the effectiveness of the method. Experimental results demonstrate that the deblurrring images obtained by the proposed approach are of high quality with improved objective image evaluation values and better negative artifacts suppression.

Influence of the source’s energy fluctuation on computational ghost imaging and effective correction approaches

We investigate the influence of the source’s energy fluctuation on both computational ghost imaging and computational ghost imaging via sparsity constraint, and if the reconstruction quality will decrease with the increase of the source’s energy fluctuation. In order to overcome the problem of image degradation, a correction approach against the source’s energy fluctuation is proposed by recording the source’s fluctuation with a monitor before modulation and correcting the echo signal or the intensity of computed reference light field with the data recorded by the monitor. Both the numerical simulation and experimental results demonstrate that computational ghost imaging via sparsity constraint can be enhanced by correcting the echo signal or the intensity of computed reference light field, while only correcting the echo signal is valid for computational ghost imaging.

Do domain-general executive resources play a role in linguistic prediction? Re-evaluation of the evidence and a path forward

Most current accounts of language comprehension agree on a role for prediction, but they disagree on the importance of domain-general executive resources in predictive behavior. In this opinion piece, we briefly review the evidence for linguistic prediction, and the findings that have been used to argue that prediction draws on domain-general executive resources. The most compelling evidence is an apparent reduction in predictive behavior during language comprehension in populations with lower executive resources, such as children, older adults, and second language (L2) learners. We propose that these between-population differences can be explained without invoking executive resources. Instead, differences in the quantity and kind of language experience that these populations bring to bear may affect the probability of engaging in predictive behavior, or simply make prediction effects more difficult to detect in paradigms designed for young adult native speakers. Thus, domain-specific prediction mechanisms remain a viable possibility. We discuss ways to further test accounts of linguistic prediction that do vs. do not require domain-general executive resources, using behavioral, computational, and brain imaging approaches.