Computational Imaging Applications Research Articles

Learning Nonlocal Sparse and Low-Rank Models for Image Compressive Sensing: Nonlocal sparse and low-rank modeling

The compressive sensing (CS) scheme exploits many fewer measurements than suggested by the Nyquist–Shannon sampling theorem to accurately reconstruct images, which has attracted considerable attention in the computational imaging community. While classic image CS schemes employ sparsity using analytical transforms or bases, the learning-based approaches have become increasingly popular in recent years. Such methods can effectively model the structure of image patches by optimizing their sparse representations or learning deep neural networks while preserving the known or modeled sensing process. Beyond exploiting local image properties, advanced CS schemes adopt nonlocal image modeling by extracting similar or highly correlated patches at different locations of an image to form a group to process jointly. More recent learning-based CS schemes apply nonlocal structured sparsity priors using group sparse (and related) representation (GSR) and/or low-rank (LR) modeling, which have demonstrated promising performance in various computational imaging and image processing applications.

Simultaneous spectral recovery and CMOS micro-LED holography with an untrained deep neural network

Lensless holography promises compact, low-cost optical apparatus designs with similar performance to traditional imaging setups. Here, we propose the use of a silicon micro-LED fabricated in a commercial CMOS microelectronics process as the illumination source in a lensless holographic microscope. Its small emission area ( < 4 µ m 2 ) ensures high spatial coherence without the need for a pinhole and results in a large NA setup, circumventing the limits to the source-to-sample distance encountered by conventional lensless holography apparatus. The scene is reconstructed using an untrained deep neural network architecture that simultaneously performs spectral recovery by learning from the given single experimental diffraction intensity. We envision this synergetic combination of CMOS micro-LEDs and the machine learning framework can be used in other computational imaging applications, such as a compact microscope for live-cell tracking or spectroscopic imaging of biological materials.

DO: A Differentiable Engine for Deep Lens Design of Computational Imaging Systems

Computational imaging systems algorithmically post-process acquisition images either to reveal physical quantities of interest or to increase image quality, e.g., deblurring. Designing a computational imaging system requires co-design of optics and algorithms, and recently Deep Lens systems have been proposed in which both components are end-to-end designed using data-driven end-to-end training. However, progress on this exciting concept has so far been hampered by the lack of differentiable forward simulations for complex optical design spaces. Here, we introduce <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathbf{dO}$</tex-math></inline-formula> (DiffOptics) to provide derivative insights into the design pipeline to chain variable parameters and their gradients to an error metric through differential ray tracing. However, straightforward back-propagation of many millions of rays requires unaffordable device memory, and is not resolved by prior works. <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathbf{dO}$</tex-math></inline-formula> alleviates this issue using two customized memory-efficient techniques: differentiable ray-surface intersection and adjoint back-propagation. Broad application examples demonstrate the versatility and flexibility of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathbf{dO}$</tex-math></inline-formula> , including classical lens designs in asphere, double-Gauss, and freeform, reverse engineering for metrology, and joint designs of optics-network in computational imaging applications. We believe <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathbf{dO}$</tex-math></inline-formula> enables a radically new approach to computational imaging system designs and relevant research domains.

Spatial diversity improvement in frequency-diverse computational imaging with a multi-port antenna

In this paper, we study an additional spatial diversity mechanism added to a cavity-backed frequency-diverse aperture for computational imaging applications at X-band frequencies. It is shown that the dynamic variation of cavity modes achieved by means of a simple, multi-port excitation mechanism can significantly simplify the diversity constraints on the frequency-diversity technique for computational imaging and can substantially improve the fidelity of the reconstructed microwave images. Leveraging the proposed technique, we present that, for the selected number of measurement modes, the condition number of the studied computational imaging problem is improved by 8.5 times in comparison to using the frequency-diversity as the only diversity mechanism. The reconstructed X-band images of various targets, including the letters “Q”, “U” and “B” reflect on this improvement and exhibit a higher reconstruction quality, significantly assisting in the identification of the imaged objects. We also present that the resolution limits of the frequency-diverse computational imaging system synthesized using the proposed multi-port cavity are in excellent agreement with the theoretical resolution limits.

Deep Coded Aperture Design: An End-to-End Approach for Computational Imaging Tasks

Covering from photography to depth and spectral estimation, diverse computational imaging (CI) applications benefit from the versatile modulation of coded apertures (CAs). The lightwave fields as space, time, or spectral can be modulated to obtain projected encoded information at the sensor that is then decoded by efficient methods, such as the modern deep learning decoders. Although the CA can be fabricated to produce an analog modulation, a binary CA is preferred since more straightforward calibration, higher speed, and lower storage are achieved. As the performance of the decoder mainly depends on the structure of the CA, several works optimize the CA ensembles by customizing regularizers for a particular application without considering the critical physical constraints of the CAs. This work presents an end-to-end (E2E) deep learning-based optimization of CAs for CI tasks. The CA design method aims to cover a wide range of CI problems, easily changing the loss function of the deep approach. The designed loss function includes regularizers to fulfill the widely used sensing requirements of the CI applications. Mainly, the regularizers can be selected to optimize the transmittance, the compression ratio, and the correlation among measurements. At the same time, a binary CA solution is encouraged, and the performance of the CI task is maximized in applications such as restoration, classification, and semantic segmentation.

Close Encounters of the Binary Kind: Signal Reconstruction Guarantees for Compressive Hadamard Sampling With Haar Wavelet Basis

We investigate the problems of 1-D and 2-D signal recovery from subsampled Hadamard measurements using Haar wavelet as a sparsity inducing prior. These problems are of interest in, e.g., computational imaging applications relying on optical multiplexing or single-pixel imaging. However, the realization of such modalities is often hindered by the coherence between the Hadamard and Haar bases. The variable and multilevel density sampling strategies solve this issue by adjusting the subsampling process to the local and multilevel coherence, respectively, between the two bases; hence enabling successful signal recovery. In this work, we compute an explicit sample-complexity bound for Hadamard-Haar systems as well as uniform and non-uniform recovery guarantees; a seemingly missing result in the related literature. We explore the faithfulness of the numerical simulations to the theoretical results and show in a practically relevant instance, e.g., single-pixel camera, that the target signal can be recovered from a few Hadamard measurements.

Implementation and Characterization of a Two-Dimensional Printed Circuit Dynamic Metasurface Aperture for Computational Microwave Imaging

We present the design, fabrication, and experimental characterization of a 2-D, dynamically tuned, metasurface aperture, emphasizing its potential performance in computational imaging applications. The dynamic metasurface aperture (DMA) consists of an irregular, planar cavity that feeds a multitude of tunable metamaterial elements, all fabricated in a compact, multilayer printed circuit board process. The design considerations for the metamaterial element as a tunable radiator, the associated biasing circuitry, as well as cavity parameters are examined and discussed. A sensing matrix can be constructed from the measured transmit patterns, the singular value spectrum of which provides insight into the information capacity of the apertures. We investigate the singular value spectra of the sensing matrix over a variety of operating parameters, such as the number of metamaterial elements, number of masks (aka tuning states), and number of radiating elements. After optimizing over these key parameters, we demonstrate computational microwave imaging of simple test objects.

Simulating anisoplanatic turbulence by sampling intermodal and spatially correlated Zernike coefficients

Simulating atmospheric turbulence is an essential task for evaluating turbulence mitigation algorithms and training learning-based methods. Advanced numerical simulators for atmospheric turbulence are available, but they require evaluating wave propagation, which is computationally expensive. We present a propagation-free method for simulating imaging through turbulence with applications in ground-to-ground imaging. The key idea behind our work is a method to draw intermodal and spatially correlated Zernike coefficients. By establishing the equivalence of the angle-of-arrival correlation by Basu, McCrae, and Fiorino (2015) with the multiaperture correlation by Chanan (1992), we show that the Zernike coefficients can be drawn according to a covariance matrix defining the correlations. We propose fast and scalable sampling strategies to draw these samples. The method allows us to compress the wave propagation problem into a sampling problem, hence making the simulator significantly faster than existing ones. Experimental results show that the simulator has an excellent match with the theory and real turbulence data. We anticipate the simulator, which balances speed and accuracy, will be a useful tool for various computational imaging applications.

Artificial intelligence radiogenomics for advancing precision and effectiveness in oncologic care (Review).

The new era of artificial intelligence (AI) has introduced revolutionary data-driven analysis paradigms that have led to significant advancements in information processing techniques in the context of clinical decision-support systems. These advances have created unprecedented momentum in computational medical imaging applications and have given rise to new precision medicine research areas. Radiogenomics is a novel research field focusing on establishing associations between radiological features and genomic or molecular expression in order to shed light on the underlying disease mechanisms and enhance diagnostic procedures towards personalized medicine. The aim of the current review was to elucidate recent advances in radiogenomics research, focusing on deep learning with emphasis on radiology and oncology applications. The main deep learning radiogenomics architectures, together with the clinical questions addressed, and the achieved genetic or molecular correlations are presented, while a performance comparison of the proposed methodologies is conducted. Finally, current limitations, potentially understudied topics and future research directions are discussed.

XUAVs: Towards Efficient Approximate Computing for UAVs—Low Power Approximate Adders With Single LUT Delay for FPGA-Based Aerial Imaging Optimization

High Definition (HD) image processing and real-time analytics over live video feeds have always been the key requirements for Intelligence, Surveillance and Reconnaissance (ISR) applications. With the evolution of optics and image enhancement techniques, computational loads of HD ISR systems are also rising exponentially. On the contrary, the slow-down of Moore's Law has recently posed challenging bounds over the level of achievable miniaturization for emerging processing and storage units. Field Programmable Gate Arrays (FPGAs) offer a popular choice of implementing ISR algorithms over resource-constrained platforms, such as Unmanned Aerial Vehicles (UAVs), due to favorable features of reconfigurability and rapid prototyping. A promising solution to bridge the gap between resource-constrained host platforms and computation-intensive FPGA applications is the paradigm of Approximate Computing. It compromises on the accuracy of processed results to offer significant performance gains for error-tolerant applications, such as video and image processing. In this paper, we present a novel approximate adder design methodology, for FPGA-based systems with improved SWaP performance, besides preserving the accuracy requirements within acceptable thresholds. The design methodology proposed in this paper focuses on the FPGA-specific Look-Up Table (LUT) architecture to introduce approximations while splitting the carry chain into LUT-based sub-adders, with flexible overlap to tune the adder's accuracy and achieve the overall latency of a single LUT. The paper presents several variants of the proposed design and offers application-oriented flexibility to adjust for optimal SWaP vs accuracy trade-off. We have further devised a comprehensive assessment approach to verify functional viability of the proposed atomic arithmetic blocks at system level, through their implementation into dense computational imaging applications, such as 2-dimensional Discrete Cosine Transform (DCT), airborne self-localization and moving object tracking algorithms, in comparison with other state-of-the-art adders. Our most accurate design performs at least 9.9% better in power consumption when compared with existing approximate adders, which proves that the proposed methodology holds promising potential to improve SWaP-index for computation-intensive UAV applications.

Reflection and transmission by large inhomogeneous media. Validity of born, rytov and beam propagation methods

The ability to model the backward and forward scattering by large inhomogeneous samples is of major importance especially for computational imaging applications. Unfortunately, when the sample dimensions exceed hundreds of wavelengths, a rigorous solving of the Maxwell Equations becomes difficult. In this work, we compare the scattered field estimated by a rigorous Maxwell equation solver, based on the discrete dipole approximation, with that given by several approached methods, Rytov, Born and a revisited Beam Propagation Method. We show on many examples, that BPM outperforms all the other techniques and accurately handles field distortion and moderate multiple scattering.

Dual-Tap Computational Photography Image Sensor With Per-Pixel Pipelined Digital Memory for Intra-Frame Coded Multi-Exposure

A coded-exposure-pixel image sensor for computational imaging applications is presented. Each frame exposure time is divided into ${N}$ subframes. Within each subframe, each pixel sorts photo-generated charge into two charge taps depending on that pixel’s 1-bit binary code. ${N}$ global updates of arbitrary pixel-wise codes are implemented in each frame to enable ${N}$ short global pixel-specific subexposures within one frame. To make these subexposures global, two latches per pixel are utilized in a pipelined fashion. The code for the next subframe is loaded into latch 1 in a row parallel fashion, while the code for the current subframe is being applied by latch 2 globally for photo-generated charge sorting during the current subexposure. A $280^{H}\times 176^{V}$ image sensor prototype with $11.2{\text{-}}\mu \text{m}$ pixel pitch has been fabricated in a $0.11{\text{-}}\mu \text{m}$ CMOS image sensor (CIS) technology. The image sensor has been demonstrated in two computational photography applications, each using only a single frame of a video: 1) computing both albedo (a measure of reflectivity) and 3-D depth maps by means of structured-light imaging and 2) computing surface normals (3-D orientations) map by means of photometric stereo imaging. These demonstrations experimentally validate some of the unique capabilities of this computational image sensor, such as accurate 3-D visual scene reconstruction using only one camera, while maintaining its native specifications: the full spatial resolution and the maximum frame rate.

On the Coherence Relationship between Measurement Matrices and Equivalent Radiation Sources in Microwave Computational Imaging Applications

In recent years, computational imaging, which encodes scene information into a set of measurements, has become a research focus in the field of microwave imaging. As with other typical inverse problems, the key challenge is to reduce the mutual coherences in the measurement matrix which is composed of measurement modes. Since the modes are synthesized by antennas, there is a great deal of interest in the antenna optimization for the reduction. The mechanism underlying the generation of the coherences is critical for the optimization; however, relevant research is still inadequate. In this paper, we try to address the research gap by relating the coherences to the antennas’ equivalent radiation sources via spectral Green’s dyad. We demonstrate that the coherences in the measurement matrix are dependent on the spatial spectral coherences of the sources, while in this relationship the imaging scenario acts as a spectral low-pass filter. Increasing the imaging range narrows the spectral constraint, which eventually increases the coherences in the measurement matrix. Full-wave electromagnetic simulations are performed for validation. We hope that our work provides a possible direction for the antenna optimization in microwave computational imaging (MCI) applications and motivates further research in this field.

Accurate Pixel-to-Pixel Alignment Method With Six-Axis Adjustment for Computational Photography

Computational photography has been a new research focus over the last two decades. Most of the computational imaging applications involve pixel-to-pixel correspondence adjustment between the spatial light modulator and the sensor. In this study, an accuracy pixel-to-pixel alignment method with six-axis adjustment is proposed. Specifically, the relations between the moire fringe distribution and the six degree of freedom (DoF) displacements are characterized. A special pattern called five-grating pattern is designed for monitoring and adjusting the six DoF displacements in a four-step alignment procedure. Finally, the experimental results verify the performance of the proposed method by discussing the alignment accuracy.

Information-Theoretic Characterization and Undersampling Ratio Determination for Compressive Radar Imaging in a Simulated Environment

Assuming sparsity or compressibility of the underlying signals, compressed sensing or compressive sampling (CS) exploits the informational efficiency of under-sampled measurements for increased efficiency yet acceptable accuracy in information gathering, transmission and processing, though it often incurs extra computational cost in signal reconstruction. Shannon information quantities and theorems, such as source rate-distortion, trans-information and rate distortion theorem concerning lossy data compression, provide a coherent framework, which is complementary to classic CS theory, for analyzing informational quantities and for determining the necessary number of measurements in CS. While there exists some information-theoretic research in the past on CS in general and compressive radar imaging in particular, systematic research is needed to handle issues related to scene description in cluttered environments and trans-information quantification in complex sparsity-clutter-sampling-noise settings. The novelty of this paper lies in furnishing a general strategy for information-theoretic analysis of scene compressibility, trans-information of radar echo data about the scene and the targets of interest, respectively, and limits to undersampling ratios necessary for scene reconstruction subject to distortion given sparsity-clutter-noise constraints. A computational experiment was performed to demonstrate informational analysis regarding the scene-sampling-reconstruction process and to generate phase transition diagrams showing relations between undersampling ratios and sparsity-clutter-noise-distortion constraints. The strategy proposed in this paper is valuable for information-theoretic analysis and undersampling theorem developments in compressive radar imaging and other computational imaging applications.

Aperture access and manipulation for computational imaging

Many computational imaging applications involve manipulating the incoming light beam in the aperture and image planes. However, accessing the aperture, which conventionally stands inside the imaging lens, is still challenging. In this paper, we present an approach that allows access to the aperture plane and enables dynamic control of its transmissivity, position, and orientation. Specifically, we present two kinds of compound imaging systems (CIS), CIS1 and CIS2, to reposition the aperture in front of and behind the imaging lens respectively. CIS1 repositions the aperture plane in front of the imaging lens and enables the dynamic control of the light beam coming to the lens. This control is quite useful in panoramic imaging at the single viewpoint. CIS2 uses a rear-attached relay system (lens) to replace the aperture plane behind the imaging lens, and enables the dynamic control of the imaging light jointly formed by the imaging lens and the relay lens. In this way, the common imaging beam can be coded or split in the aperture plane to achieve many imaging functions, such as coded aperture imaging, high dynamic range (HDR) imaging and light field sampling. In addition, CIS2 repositions the aperture behind, instead of inside, the relay lens, which allows the employment of the optimized relay lens to preserve the high imaging quality. Finally, we present the physical implementations of CIS1 and CIS2, to demonstrate (1) their effectiveness in providing access to the aperture and (2) the advantages of aperture manipulation in computational imaging applications.