Computational Imaging Technique Research Articles

Single-shot memory-effect video

Imaging through opaque scattering media is critically important in applications ranging from biological and astronomical imaging to metrology and security. While the random process of scattering in turbid media produces scattered light that appears uninformative to the human eye, a wealth of information is contained in the signal and can be recovered using computational post-processing techniques. Recent studies have shown that statistical correlations present in the scattered light, known as ‘memory effects’, allow for diffraction-limited imaging through opaque media without detailed knowledge of (or access to) the source or scatterer. However, previous methods require that the object and/or scatterer be static during the measurement. We overcome this limitation by combining traditional memory effect imaging with coded-aperture-based computational imaging techniques, which enables us to realize for the first time single-shot video of arbitrary dynamic scenes through dynamic, opaque media. This has important implications for a wide range of real-world imaging scenarios.

Fast and robust misalignment correction of Fourier ptychographic microscopy for full field of view reconstruction.

Fourier ptychographic microscopy (FPM) is a newly developed computational imaging technique that can provide gigapixel images with both high resolution (HR) and wide field of view (FOV). However, there are two possible reasons for position misalignment, which induce a degradation of the reconstructed image. The first one is the position misalignment of the LED array, which can largely be eliminated during the experimental system building process. The more important one is the segment-dependent position misalignment. Note that, this segment-dependent positional misalignment still exists, even after we correct the central coordinates of every small segment. In this paper, we carefully analyze this segment-dependent misalignment and find that this global shift matters more, compared with the rotational misalignments. According to this fact, we propose a robust and fast method to correct the two factors of position misalignment of the FPM, termed as misalignment correction for the FPM misalignment correction (mcFPM). Although different regions in the FOV have different sensitivities to the position misalignment, the experimental results show that the mcFPM is robust with respect to the elimination of each region. Compared with the state-of-the-art methods, the mcFPM is much faster.

Subwavelength resolution Fourier ptychography with hemispherical digital condensers.

Fourier ptychography (FP) is a promising computational imaging technique that overcomes the physical space-bandwidth product (SBP) limit of a conventional microscope by applying angular-varied illuminations. However, to date, the effective imaging numerical aperture (NA) achievable with a commercial LED board is still limited to the range of 0.3-0.7 with a 4 × /0.1NA objective due to the geometric constraint with the declined illumination intensities and attenuated signal-to-noise ratio (SNR). Thus the highest achievable half-pitch resolution is usually constrained between 500-1000 nm, which cannot meet the requirements of high-resolution biomedical imaging applications. Although it is possible to improve the resolution by using a high-NA objective lens, the FP approach is less appealing as the decrease of field-of-view (FOV) will far exceed the improvement of spatial resolution in this case. In this paper, we initially present a subwavelength resolution Fourier ptychography (SRFP) platform with a hemispherical digital condenser to provide high-angle programmable plane-wave illuminations of 0.95NA, attaining a 4 × /0.1NA objective with the final effective imaging performance of 1.05NA at a half-pitch resolution of 244 nm with the incident wavelength of 465 nm across a wide FOV of 14.60 mm2, corresponding to a SBP of 245 megapixels. Our work provides an essential step of FP towards high-throughput imaging applications.

Adaptive-window angular spectrum algorithm for near-field ptychography

Abstract Near-field ptychography provides advantages over far-field ptychography that large field-of-view (FOV) can be imaged with fewer diffraction images, and with weaker requirements on the detector dynamic range and beam coherence, which has drawn attention recently. However, the propagation distance of traditional angular spectrum (AS) method is limited and the reconstruction of the smallest resolvable object detail is restricted by the sensor’s pixel size. To this end, we propose an adaptive-window angular spectrum (AWAS) algorithm to solve both problems by adding the window adaptively associated with the propagation distance and avoid the extra computations via extra scaling factors. Meanwhile, it features validity for the independent sample size and the sample number on the observation plane. This algorithm is strictly deduced from the Rayleigh–Sommerfeld formula and based on the linear convolution, which can be evaluated by fast Fourier transform effectively. The burden of calculations is comparable to traditional AS method. The performance has been achieved both in two-dimensional and three-dimensional near-field ptychography with simulations and experiments. This method will make near-field ptychography more practical and can be used in X-ray or electron-microscopy and other computational imaging techniques.

Analysis of Fourier ptychographic microscopy with half of the captured images

Fourier ptychographic microscopy (FPM) is a new computational imaging technique that can provide gigapixel images with both a high-resolution and a wide field of view. However, the data-acquisition process is time-consuming, which is a critical issue. In this study, we analyze the FPM imaging system with the half number of captured images; i.e. if the size of the light-emitting diode (LED) array is 15 × 15, we use 15 × 8 LEDs to capture images. According to the image analysis of the conventional FPM system, we then compare the reconstructed images with different captured data. When the sample is close to amplitude-only, simulation and experimental results show that the resolution of the reconstructed image with the half number of captured data does not differ significantly from that with all the captured data.

Computational complex optical field imaging using a designed metasurface diffuser

Various speckle-based computational imaging techniques that exploit the ability of scattering media to transfer hidden information into the speckle pattern have recently been demonstrated. Current implementations suffer from several drawbacks associated with the use of conventional scattering media (CSM), such as their time-consuming characterization, instability with time, and limited memory-effect range. Here we show that by using a random dielectric metasurface diffuser (MD) with known scattering properties, many of these issues can be addressed. We experimentally demonstrate an imaging system with the ability to retrieve complex field values using a MD and the speckle-correlation scattering matrix method. We explore the mathematical properties of the MD transmission matrix such as its correlation and singular value spectrum to expand the understanding about both MDs and the speckle-correlation scattering matrix approach. In addition to a large noise tolerance, reliable reproducibility, and robustness against misalignments, using the MD allows us to substitute the laborious experimental characterization procedure of the CSM with a simple simulation process. Moreover, dielectric MDs with identical scattering properties can easily be mass-produced, thus enabling real-world applications. Representing a bridge between metasurface optics and speckle-based computational imaging, this work paves the way to extending the potentials of diverse speckle-based computational imaging methods for various applications such as biomedical imaging, holography, and optical encryption.

Optical image encryption via high-quality computational ghost imaging using iterative phase retrieval

A novel computational ghost imaging scheme based on specially designed phase-only masks, which can be efficiently applied to encrypt an original image into a series of measured intensities, is proposed in this paper. First, a Hadamard matrix with a certain order is generated, where the number of elements in each row is equal to the size of the original image to be encrypted. Each row of the matrix is rearranged into the corresponding 2D pattern. Then, each pattern is encoded into the phase-only masks by making use of an iterative phase retrieval algorithm. These specially designed masks can be wholly or partially used in the process of computational ghost imaging to reconstruct the original information with high quality. When a significantly small number of phase-only masks are used to record the measured intensities in a single-pixel bucket detector, the information can be authenticated without clear visualization by calculating the nonlinear correlation map between the original image and its reconstruction. The results illustrate the feasibility and effectiveness of the proposed computational ghost imaging mechanism, which will provide an effective alternative for enriching the related research on the computational ghost imaging technique.

Revealing hidden scenes by photon-efficient occlusion-based opportunistic active imaging.

The ability to see around corners, i.e., recover details of a hidden scene from its reflections in the surrounding environment, is of considerable interest in a wide range of applications. However, the diffuse nature of light reflected from typical surfaces leads to mixing of spatial information in the collected light, precluding useful scene reconstruction. Here, we employ a computational imaging technique that opportunistically exploits the presence of occluding objects, which obstruct probe-light propagation in the hidden scene, to undo the mixing and greatly improve scene recovery. Importantly, our technique obviates the need for the ultrafast time-of-flight measurements employed by most previous approaches to hidden-scene imaging. Moreover, it does so in a photon-efficient manner (i.e., it only requires a small number of photon detections) based on an accurate forward model and a computational algorithm that, together, respect the physics of three-bounce light propagation and single-photon detection. Using our methodology, we demonstrate reconstruction of hidden-surface reflectivity patterns in a meter-scale environment from non-time-resolved measurements. Ultimately, our technique represents an instance of a rich and promising new imaging modality with important potential implications for imaging science.

Computational Imaging and Displays: Capturing and displaying richer representations of the world.

Computational imaging techniques allow capturing richer, more complete representations of a scene through the introduction of novel computational algorithms, overcoming the limitations imposed by hardware and optics. The areas of application range from medical imaging to security, to areas in engineering, to name a few. Computational displays combine optics, hardware and computation to faithfully reproduce the world as seen by our eyes, something that current displays still cannot do. This article looks at a few examples of both, aiming to convey the power of the joint co-design of hardware and computational algorithms, also taking into account visual perception.

1000 fps computational ghost imaging using LED-based structured illumination.

Single-pixel imaging uses a single-pixel detector, rather than a focal plane detector array, to image a scene. It provides advantages for applications such as multi-wavelength, three-dimensional imaging. However, low frame rates have been a major obstacle inhibiting the use of computational ghost imaging technique in wider applications since its invention one decade ago. To address this problem, a computational ghost imaging scheme, which utilizes an LED-based, high-speed illumination module is presented in this work. At 32 × 32 pixel resolution, the proof-of-principle system achieved continuous imaging with 1000 fps frame rate, approximately two orders larger than those of other existing ghost imaging systems. The proposed scheme provides a cost-effective and high-speed imaging technique for dynamic imaging applications.

Analysis for Science Librarians of the 2017 Nobel Prize in Chemistry: Cryo-electron Microscopy

ABSTRACTThe 2017 Nobel Prize in Chemistry recognized three scientists for their contributions to developing the technique of cryo-electron microscopy (cryo-EM) for the imaging and structure elucidation of biomolecules. Richard Henderson’s dogged pursuit led to the first Ångstrom-level visualization of a biomacromolecule. Joachim Frank developed computational imaging techniques to produce 3D visualizations from collections of low-contrast EM images. Jacques Dubochet was the first to use vitrified, noncrystalline water as a medium for cryo-EM of biological specimens. Each of these scientists has contributed a key technique or insight into using cryo-EM to visualize biological particles that heretofore had been elusive to visualization. Recent improvements in electron detector technology have helped realize the fuller potential of cryo-EM.

Virtual Image Points Based Geometrical Parameters’ Calibration for Focused Light Field Camera

In the light field imaging, the geometrical parameters’ calibration of the light field camera is the foundation of computational imaging techniques, such as the calibration of the light field data, digital refocusing, depth reconstruction, all-in-focus imaging, and so on. In this paper, we propose a virtual image points based geometrical parameters’ calibration for focused light field camera. In the forward model, the mapping from the object points to the virtual image points with respect to the main lens as well as the mapping from the virtual image points to the image points on the detector with respect to the microlens array are established. While calibrating geometrical parameters, the inverse problem of calculating the virtual image points from the corner on the detector is established and the model of calculating the geometrical parameters of the focused light field camera is established. Levenberg–Marquardt algorithm is used to solve the geometrical parameters iteratively. We use the checkerboard for the calibration experiment and validate the calculation via the reprojection of the checkerboard corners.

Data preprocessing methods for robust Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) is a recently proposed computational imaging technique with both high resolution and wide field-of-view. In current FP experimental setup, the dark-field images with high-angle illuminations are easily submerged by stray light and background noise due to the low signal-to-noise ratio, thus significantly degrading the reconstruction quality and also imposing a major restriction on the synthetic numerical aperture (NA) of the FP approach. To this end, an overall and systematic data preprocessing scheme for noise removal from FP's raw dataset is provided, which involves sampling analysis as well as underexposed/overexposed treatments, then followed by the elimination of unknown stray light and suppression of inevitable background noise, especially Gaussian noise and CCD dark current in our experiments. The reported non-parametric scheme facilitates great enhancements of the FP's performance, which has been demonstrated experimentally that the benefits of noise removal by these methods far outweigh its defects of concomitant signal loss. In addition, it could be flexibly cooperated with the existing state-of-the-art algorithms, producing a stronger robustness of the FP approach in various applications.

Computational passive imaging of thermal sources with a leaky chaotic cavity

Computational imaging techniques are of great interest to simplify the architecture of imaging devices since random illuminations of a scene enable its reconstruction from few measurements by solving an inverse problem. Here, we present a passive system for imaging of thermal sources in the microwave range from the cross-correlation of noise signals recorded by only two channels. The channels are attached to a high Q-factor chaotic cavity with a leaky aperture on its front side. The spatial distribution of noise sources is encoded onto the broadband spectrum of the cross-correlation and can be reconstructed from the sensing matrix mapping the uncorrelated far-field speckle patterns of the cavity onto a set of frequencies. We demonstrate imaging of localized and extended thermal sources and show that the polarization of those radiations can be discriminated. Moreover, we exhibit the effectiveness of the proposed system as a compressive imaging device which exploits the natural randomness of the speckle patterns. We believe that these results are a promising step for the design of real time and low cost microwave radiometers.

Computational imaging with a frequency-diverse metasurface aperture

We demonstrate image formation with radio frequency (RF) waves using a frequency-diverse metasurface antenna. The metasurface antenna consists of a planar cavity—formed from a double-layer, copper-clad printed circuit board—fed from one side by an RF source, and with a series of irises patterned into the opposite side. At any particular frequency, the cavity mode excited by the feed in turn excites the irises, each of which radiates as a magnetic dipole element. The cavity walls (fabricated using vias patterned into the circuit board) are irregular, so that the excited cavity modes exhibit considerable variation as a function of frequency. Thus, the radiated fields from the cavity-backed metasurface tend to have a series of randomly directed lobes, the number and directions of which vary significantly as a function of frequency. These radiation patterns can be used to form measurements of a scene, with the scene then estimated using computational imaging techniques. We have applied the metasurface aperture in a prototype imaging system capable of acquiring high resolution images of human-scale targets. The metasurface aperture design and reconstruction methods are general, and can be easily adapted for acoustic imaging.

A color-corrected strategy for information multiplexed Fourier ptychographic imaging

Fourier ptychography (FP) is a novel computational imaging technique that provides both wide field of view (FoV) and high-resolution (HR) imaging capacity for biomedical imaging. Combined with information multiplexing technology, wavelength multiplexed (or color multiplexed) FP imaging can be implemented by lighting up R/G/B LED units simultaneously. Furthermore, a HR image can be recovered at each wavelength from the multiplexed dataset. This enhances the efficiency of data acquisition. However, since the same dataset of intensity measurement is used to recover the HR image at each wavelength, the mean value in each channel would converge to the same value. In this paper, a color correction strategy embedded in the multiplexing FP scheme is demonstrated, which is termed as color corrected wavelength multiplexed Fourier ptychography (CWMFP). Three images captured by turning on a LED array in R/G/B are required as priori knowledge to improve the accuracy of reconstruction in the recovery process. Using the reported technique, the redundancy requirement of information multiplexed FP is reduced. Moreover, the accuracy of reconstruction at each channel is improved with correct color reproduction of the specimen.

Label-free 3D computational imaging of spermatozoon locomotion, head spin and flagellum beating over a large volume

We report a high-throughput and label-free computational imaging technique that simultaneously measures in three-dimensional (3D) space the locomotion and angular spin of the freely moving heads of microswimmers and the beating patterns of their flagella over a sample volume more than two orders-of-magnitude larger compared to existing optical modalities. Using this platform, we quantified the 3D locomotion of 2133 bovine sperms and determined the spin axis and the angular velocity of the sperm head, providing the perspective of an observer seated at the moving and spinning sperm head. In this constantly transforming perspective, flagellum-beating patterns are decoupled from both the 3D translation and spin of the head, which provides the opportunity to truly investigate the 3D spatio-temporal kinematics of the flagellum. In addition to providing unprecedented information on the 3D locomotion of microswimmers, this computational imaging technique could also be instrumental for micro-robotics and sensing research, enabling the high-throughput quantification of the impact of various stimuli and chemicals on the 3D swimming patterns of sperms, motile bacteria and other micro-organisms, generating new insights into taxis behaviors and the underlying biophysics.

3D imaging of optically cleared tissue using a simplified CLARITY method and on-chip microscopy.

High-throughput sectioning and optical imaging of tissue samples using traditional immunohistochemical techniques can be costly and inaccessible in resource-limited areas. We demonstrate three-dimensional (3D) imaging and phenotyping in optically transparent tissue using lens-free holographic on-chip microscopy as a low-cost, simple, and high-throughput alternative to conventional approaches. The tissue sample is passively cleared using a simplified CLARITY method and stained using 3,3'-diaminobenzidine to target cells of interest, enabling bright-field optical imaging and 3D sectioning of thick samples. The lens-free computational microscope uses pixel super-resolution and multi-height phase recovery algorithms to digitally refocus throughout the cleared tissue and obtain a 3D stack of complex-valued images of the sample, containing both phase and amplitude information. We optimized the tissue-clearing and imaging system by finding the optimal illumination wavelength, tissue thickness, sample preparation parameters, and the number of heights of the lens-free image acquisition and implemented a sparsity-based denoising algorithm to maximize the imaging volume and minimize the amount of the acquired data while also preserving the contrast-to-noise ratio of the reconstructed images. As a proof of concept, we achieved 3D imaging of neurons in a 200-μm-thick cleared mouse brain tissue over a wide field of view of 20.5 mm2. The lens-free microscope also achieved more than an order-of-magnitude reduction in raw data compared to a conventional scanning optical microscope imaging the same sample volume. Being low cost, simple, high-throughput, and data-efficient, we believe that this CLARITY-enabled computational tissue imaging technique could find numerous applications in biomedical diagnosis and research in low-resource settings.

Computational imaging of sperm locomotion.

Not only essential for scientific research, but also in the analysis of male fertility and for animal husbandry, sperm tracking and characterization techniques have been greatly benefiting from computational imaging. Digital image sensors, in combination with optical microscopy tools and powerful computers, have enabled the use of advanced detection and tracking algorithms that automatically map sperm trajectories and calculate various motility parameters across large data sets. Computational techniques are driving the field even further, facilitating the development of unconventional sperm imaging and tracking methods that do not rely on standard optical microscopes and objective lenses, which limit the field of view and volume of the semen sample that can be imaged. As an example, a holographic on-chip sperm imaging platform, only composed of a light-emitting diode and an opto-electronic image sensor, has emerged as a high-throughput, low-cost and portable alternative to lens-based traditional sperm imaging and tracking methods. In this approach, the sample is placed very close to the image sensor chip, which captures lensfree holograms generated by the interference of the background illumination with the light scattered from sperm cells. These holographic patterns are then digitally processed to extract both the amplitude and phase information of the spermatozoa, effectively replacing the microscope objective lens with computation. This platform has further enabled high-throughput 3D imaging of spermatozoa with submicron 3D positioning accuracy in large sample volumes, revealing various rare locomotion patterns. We believe that computational chip-scale sperm imaging and 3D tracking techniques will find numerous opportunities in both sperm related research and commercial applications.

N-Desmethyldauricine Induces Autophagic Cell Death in Apoptosis-Defective Cells via Ca2+ Mobilization.

Resistance of cancer cells to chemotherapy remains a significant problem in oncology. Mechanisms regulating programmed cell death, including apoptosis, autophagy or necrosis, in the treatment of cancers have been extensively investigated over the last few decades. Autophagy is now emerging as an important pathway in regulating cell death or survival in cancer therapy. Recent studies demonstrated variety of natural small-molecules could induce autophagic cell death in apoptosis-resistant cancer cells, therefore, discovery of novel autophagic enhancers from natural products could be a promising strategy for treatment of chemotherapy-resistant cancer. By computational virtual docking analysis, biochemical assays, and advanced live-cell imaging techniques, we have identified N-desmethyldauricine (LP-4), isolated from rhizoma of Menispermum dauricum DC as a novel inducer of autophagy. LP-4 was shown to induce autophagy via the Ulk-1-PERK and Ca2+/Calmodulin-dependent protein kinase kinase β (CaMKKβ)-AMPK-mTOR signaling cascades, via mobilizing calcium release through inhibition of SERCA, and importantly, lead to autophagic cell death in a panel of cancer cells, apoptosis-defective and apoptosis-resistant cells. Taken together, this study provides detailed insights into the cytotoxic mechanism of a novel autophagic compound that targeting the apoptosis resistant cancer cells, and new implication on drug discovery from natural products for drug resistant cancer therapy.