Single-pixel Camera Research Articles

Single-pixel image reconstruction from experimental data using neural networks.

Single-pixel cameras that measure image coefficients have various promising applications, in particular for hyper-spectral imaging. Here, we investigate deep neural networks that when fed with experimental data can output high-quality images in real time. Assuming that the measurements are corrupted by mixed Poisson-Gaussian noise, we propose to map the raw data from the measurement domain to the image domain based on a Tikhonov regularization. This step can be implemented as the first layer of a deep neural network, followed by any architecture of layers that acts in the image domain. We also describe a framework for training the network in the presence of noise. In particular, our approach includes an estimation of the image intensity and experimental parameters, together with a normalization scheme that allows varying noise levels to be handled during training and testing. Finally, we present results from simulations and experimental acquisitions with varying noise levels. Our approach yields images with improved peak signal-to-noise ratios, even for noise levels that were foreseen during the training of the networks, which makes the approach particularly suitable to deal with experimental data. Furthermore, while this approach focuses on single-pixel imaging, it can be adapted for other computational optics problems.

Single-pixel scatter-plate microscopy.

Based on the optical memory effect of scattered light, we developed a new single-pixel camera concept. The retrieved images contain both 3D and spectral information about the sample. A spatial light modulator (SLM) generates a random intensity modulation. The signal recorded by the single-pixel detector is cross correlated by the calculated point spread function (PSF) signals of the SLM to retrieve the image. In this publication, both simulations and experimental results are presented.

Single-pixel camera with a large-area microstrip superconducting single photon detector on a multimode fiber

High sensitivity imaging at the level of single photons is an invaluable tool in many areas, ranging from microscopy to astronomy. However, development of single-photon sensitive detectors with high spatial resolution is very non-trivial. Here we employ the single-pixel imaging approach and demonstrate a proof-of-principle single-pixel single-photon imaging setup. We overcome the problem of low light gathering efficiency by developing a large-area microstrip superconducting single photon detector coupled to a multi-mode optical fiber interface. We show that the setup operates well in the visible and near infrared spectrum, and is able to capture images at the single-photon level.

Two-step training deep learning framework for computational imaging without physics priors.

Deep learning (DL) is a powerful tool in computational imaging for many applications. A common strategy is to use a preprocessor to reconstruct a preliminary image as the input to a neural network to achieve an optimized image. Usually, the preprocessor incorporates knowledge of the physics priors in the imaging model. One outstanding challenge, however, is errors that arise from imperfections in the assumed model. Model mismatches degrade the quality of the preliminary image and therefore affect the DL predictions. Another main challenge is that many imaging inverse problems are ill-posed and the networks are over-parameterized; DL networks have flexibility to extract features from the data that are not directly related to the imaging model. This can lead to suboptimal training and poorer image reconstruction results. To solve these challenges, a two-step training DL (TST-DL) framework is proposed for computational imaging without physics priors. First, a single fully-connected layer (FCL) is trained to directly learn the inverse model with the raw measurement data as the inputs and the images as the outputs. Then, this pre-trained FCL is fixed and concatenated with an un-trained deep convolutional network with a U-Net architecture for a second-step training to optimize the output image. This approach has the advantage that does not rely on an accurate representation of the imaging physics since the first-step training directly learns the inverse model. Furthermore, the TST-DL approach mitigates network over-parameterization by separately training the FCL and U-Net. We demonstrate this framework using a linear single-pixel camera imaging model. The results are quantitatively compared with those from other frameworks. The TST-DL approach is shown to perform comparable to approaches which incorporate perfect knowledge of the imaging model, to be robust to noise and model ill-posedness, and to be more robust to model mismatch than approaches which incorporate imperfect knowledge of the imaging model. Furthermore, TST-DL yields better results than end-to-end training while suffering from less overfitting. Overall, this TST-DL framework is a flexible approach for image reconstruction without physics priors, applicable to diverse computational imaging systems.

Compressive imaging for defending deep neural networks from adversarial attacks.

Despite their outstanding performance, convolutional deep neural networks (DNNs) are vulnerable to small adversarial perturbations. In this Letter, we introduce a novel approach to thwart adversarial attacks. We propose to employ compressive sensing (CS) to defend DNNs from adversarial attacks, and at the same time to encode the image, thus preventing counterattacks. We present computer simulations and optical experimental results of object classification in adversarial images captured with a CS single pixel camera.

Multispectral compressive fluorescence lifetime imaging microscopy with a SPAD array detector.

Multispectral/hyperspectral fluorescence lifetime imaging microscopy (λFLIM) is a promising tool for studying functional and structural biological processes. The rich information content provided by a multidimensional dataset is often in contrast with the acquisition speed. In this work, we develop and experimentally demonstrate a wide-field λFLIM setup, based on a novel time-resolved 18×1 single-photon avalanche diode array detector working in a single-pixel camera scheme, which parallelizes the spectral detection, reducing measurement time. The proposed system, which implements a single-pixel camera with a compressive sensing scheme, represents an optimal microscopy framework towards the design of λFLIM setups.

Non-uniform Recovery Guarantees for Binary Measurements and Infinite-Dimensional Compressed Sensing

Due to the many applications in Magnetic Resonance Imaging (MRI), Nuclear Magnetic Resonance (NMR), radio interferometry, helium atom scattering etc., the theory of compressed sensing with Fourier transform measurements has reached a mature level. However, for binary measurements via the Walsh transform, the theory has long been merely non-existent, despite the large number of applications such as fluorescence microscopy, single pixel cameras, lensless cameras, compressive holography, laser-based failure-analysis etc. Binary measurements are a mainstay in signal and image processing and can be modelled by the Walsh transform and Walsh series that are binary cousins of the respective Fourier counterparts. We help bridging the theoretical gap by providing non-uniform recovery guarantees for infinite-dimensional compressed sensing with Walsh samples and wavelet reconstruction. The theoretical results demonstrate that compressed sensing with Walsh samples, as long as the sampling strategy is highly structured and follows the structured sparsity of the signal, is as effective as in the Fourier case. However, there is a fundamental difference in the asymptotic results when the smoothness and vanishing moments of the wavelet increase. In the Fourier case, this changes the optimal sampling patterns, whereas this is not the case in the Walsh setting.

Inherent dose-reduction potential of classical ghost imaging

Classical ghost imaging is a computational imaging technique that employs patterned illumination. It is very similar in concept to the single-pixel camera in that an image may be reconstructed from a set of measurements even though all imaging quanta that pass through that sample are never recorded with a position resolving detector. The method was first conceived and applied for visible-wavelength photons and was subsequently translated to other probes such as x rays, atomic beams, electrons and neutrons. In the context of ghost imaging using penetrating probes that enable transmission measurement, we here consider several questions relating to the achievable signal-to-noise ratio (SNR). This is compared with the SNR for conventional imaging under scenarios of constant radiation dose and constant experiment time, considering both photon shot-noise and per-measurement electronic read-out noise. We show that inherent improved SNR capabilities of ghost imaging are limited to a subset of these scenarios and are actually due to increased dose (Fellgett advantage). An explanation is also presented for recent results published in the literature that are not consistent with these findings.

Deep Compressed Imaging via Optimized-Pattern Scanning.

The need for high-speed imaging in applications such as biomedicine, surveillance and consumer electronics has called for new developments of imaging systems. While the industrial effort continuously pushes the advance of silicon focal plane array image sensors, imaging through a single-pixel detector has gained significant interests thanks to the development of computational algorithms. Here, we present a new imaging modality, Deep Compressed Imaging via Optimized-Pattern Scanning (DeCIOPS), which can significantly increase the acquisition speed for a single-detector-based imaging system. We project and scan an illumination pattern across the object and collect the sampling signal with a single-pixel detector. We develop an innovative end-to-end optimized auto-encoder, using a deep neural network and compressed sensing algorithm, to optimize the illumination pattern, which allows us to reconstruct faithfully the image from a small number of samples, and with a high frame rate. Compared with the conventional switching-mask based single-pixel camera and point scanning imaging systems, our method achieves a much higher imaging speed, while retaining a similar imaging quality. We experimentally validated this imaging modality in the settings of both continuous-wave (CW) illumination and pulsed light illumination and showed high-quality image reconstructions with a high compressed sampling rate. This new compressed sensing modality could be widely applied in different imaging systems, enabling new applications which require high imaging speed.

Fluorescence lifetime imaging via spatio-temporal speckle patterns in a single-pixel camera configuration.

Photoluminescence (PL) spectroscopy offers excellent methods for mapping the PL decay on the nanosecond time scale. However, capturing maps of emission dynamics on the microsecond timescale can be highly time-consuming. We present a new approach to fluorescence lifetime imaging (FLIM), which combines the concept of random temporal speckles excitation (RATS) with the concept of a single-pixel camera based on spatial speckles. The spatio-temporal speckle pattern makes it possible to map PL dynamics with unmatched simplicity. Moreover, the method can acquire all the data necessary to map PL decay on the microsecond timescale within minutes. We present proof-of-principle measurements for two samples and compare the reconstructed decays to the non-imaging measurements. Finally, we discuss the effect of the preprocessing routine and other factors on the reconstruction noise level. The presented method is suitable for lifetime imaging processes in several samples, including monitoring charge carrier dynamics in perovskites or monitoring solid-state luminophores with a long lifetime of PL.

Single pixel camera methodologies for spatially resolved acoustic spectroscopy

Spatially resolved acoustic spectroscopy (SRAS) is a laser ultrasound technique used to determine the crystallographic orientation (i.e., microstructure) of materials through the generation and measurement of surface acoustic wave velocity on a sample. Previous implementations have used a grating pattern imaged onto the surface to control the frequency of the generated wave in a single direction—grain orientation can be computed by acquiring wave velocities in different directions on the surface (gathered by physically rotating the grating pattern). This paper reports an advance to this methodology, inspired by single pixel cameras, using a coded grating pattern, created using a spatial light modulator, to excite surface acoustic waves in multiple directions simultaneously. This change to the optical arrangement can simplify the overall system alignment, remove mechanical complexities, and is well suited for point-by-point full orientation imaging, potentially allowing for faster orientation imaging using SRAS microscopy. Improvements to the robustness of measurement may be expected to extend the applicability of SRAS in the materials science field. To demonstrate this methodology, experiments were conducted on isotropic and anisotropic samples.

Programmable spatially variant single-pixel imaging based on compressive sensing

Single-pixel camera is developed to mitigate the constraints faced by the conventional cameras especially in invisible wavelengths and low light conditions. Nyquist–Shannon theorem requires as many measurements as the image pixels to reconstruct images flawlessly. In practice, obtaining more measurements increases the cost and acquisition time, which are the major drawbacks of single-pixel imaging (SPI). Therefore, compressive sensing was proposed to enable image reconstruction with fewer measurements. We present a design of sensing patterns to obtain image information by utilizing spatially variant resolution (SVR) technique in SPI. The proposed method reduces the measurements by prioritizing the resolution in the region of interest (ROI). It successfully achieves the programmable imaging concept where multiresolution adaptively optimizes the balance between the image quality and the measurements number. Results show that SVR images can be reconstructed from significantly fewer measurements yet able to achieve better image quality than uniform resolution images. In addition, the SVR images can be further enhanced by integrating the dynamic supersampling technique. Consequently, the concerns of image quality, long acquisition, and processing time can be addressed. The proposed method potentially benefits imaging applications where the target ROI is prioritized over the background and most importantly it requires fewer measurements.

Compressive Sampling Using a Pushframe Camera

The recently described pushframe imager, a parallelized single pixel camera capturing with a pushbroom-like motion, is intrinsically suited to both remote-sensing and compressive sampling. It optically applies a 2D mask to the imaged scene, before performing light integration along a single spatial axis, but previous work has not made use of the architecture's potential for taking measurements sparsely. In this paper we develop a strongly performing static binarized noiselet compressive sampling mask design, tailored to pushframe hardware, allowing both a single exposure per motion time-step, and retention of 2D correlations in the scene. Results from simulated and real-world captures are presented, with performance shown to be similar to that of immobile and hence inappropriate for satellite use whole-scene imagers. A particular feature of our sampling approach is that the degree of compression can be varied without altering the pattern, and we demonstrate the utility of this for efficiently storing and transmitting multi-spectral images.

Hyperspectral Imaging Based on Compressive Sensing: Determining Cancer Margins in Human Pancreatic Tissue <i>ex Vivo</i>, a Pilot Study

Cancer is the second-leading cause of death in the United State and surgery remains the primary treatment for most solid mass tumors. However, accurately identifying tumor margins in real-time remains a challenge. In this study, the design and testing of hyperspectral imaging (HSI) system based on a single-pixel camera engine is discussed. The primary advantage of a single pixel architecture over traditional scanning HSI techniques is its high sensitivity and potential to function at low light levels. The objective for the imaging system described here is to detect changes in the reflectance spectra of tissue and to use these differences to delineate tumor margins. This paper presents the results of a 19-patient pilot study that assesses the ability of the HSI system to use reflectance imaging to delineate adenocarcinoma tumor margins in human pancreatic tissue imaged ex vivo. Pancreatic tissue excised during pancreatectomy was imaged immediately after being sent to the pathology lab. A pathologist sectioned the tissue and placed samples into standard tissue embedding cassettes. These tissue samples were then imaged using the HSI system. After imaging, the samples were returned to the pathologist for processing and analysis. The HSI was later compared to the histological analysis. The spectral angle mapping (SAM) and support vector machine (SVM) algorithms were used to classify pixels in the HSI images as healthy or unhealthy in order to delineate margins. Good agreement between margins determined via HSI (using both SAM and SVM) and histology/white light imaging was found.

Single-photon quantum imaging via single-photon illumination

Quantum imaging based on entangled light, laser light, and thermal light has been extensively studied. However, single-photon quantum imaging is still in its infancy. In this paper, we perform quantum imaging using heralded single-photon sources and achieve single-photon illumination computational ghost imaging and single-photon illumination single-pixel camera imaging.

High-fidelity off-axis digital optical phase conjugation with transmission matrix assisted calibration.

The spatial information carried by light is scrambled when it propagates through a scattering medium, such as frosted glass, biological tissue, turbulent air, or multimode optical fibres. Digital optical phase conjugation (DOPC) is a technique that 'pre-aberrates' an illuminating wavefront to compensate for scatterer induced distortion. DOPC systems act as phase-conjugate mirrors: they require a camera to holographically record a distorted wavefront emanating from the scatterer and a spatial light modulator (SLM) to synthesize a phase conjugate of the measured wavefront, which is sent back through the scatterer thus creating a time-reversed copy of the original optical field. High-fidelity DOPC can be technically challenging to achieve as it typically requires pixel-perfect alignment between the camera and SLM. Here we describe a DOPC system in which the normally stringent alignment criteria are relaxed. In our system the SLM and camera are placed in-line in the same optical path from the sample, and the SLM is used in an off-axis configuration. This means high-precision alignment can be achieved by measurement of the transmission matrix (TM) mapping optical fields from the SLM to the camera and vice-versa, irrespective of their relative position. The TM also absorbs and removes other aberrations in the optical system, such as the curvature of the SLM and camera chips. Using our system we demonstrate high-fidelity focussing of light through two ground glass diffusers with a peak-intensity to mean-background ratio of ∼700. We provide a step-by-step guide detailing how to align this system and discuss the trade-offs with alternative configurations. We also describe how our setup can be used as a 'single-pixel camera' based DOPC system, offering potential for DOPC at wavelengths in which cameras are not available or are prohibitively expensive.

Image Compression Based on Compressive Sensing: End-to-End Comparison With JPEG

We present an end-to-end image compression system based on compressive sensing. The presented system integrates the conventional scheme of compressive sampling and reconstruction with quantization and entropy coding. The compression performance, in terms of decoded image quality versus data rate, is shown to be comparable with JPEG and significantly better at the low rate range. We study the parameters that influence the system performance, including (i) the choice of sensing matrix, (ii) the trade-off between quantization and compression ratio, and (iii) the reconstruction algorithms. We propose an effective method to jointly control the quantization step and compression ratio in order to achieve near optimal quality at any given bit rate. Furthermore, our proposed image compression system can be directly used in the compressive sensing camera, e.g. the single pixel camera, to construct a hardware compressive sampling system.

Close Encounters of the Binary Kind: Signal Reconstruction Guarantees for Compressive Hadamard Sampling With Haar Wavelet Basis

We investigate the problems of 1-D and 2-D signal recovery from subsampled Hadamard measurements using Haar wavelet as a sparsity inducing prior. These problems are of interest in, e.g., computational imaging applications relying on optical multiplexing or single-pixel imaging. However, the realization of such modalities is often hindered by the coherence between the Hadamard and Haar bases. The variable and multilevel density sampling strategies solve this issue by adjusting the subsampling process to the local and multilevel coherence, respectively, between the two bases; hence enabling successful signal recovery. In this work, we compute an explicit sample-complexity bound for Hadamard-Haar systems as well as uniform and non-uniform recovery guarantees; a seemingly missing result in the related literature. We explore the faithfulness of the numerical simulations to the theoretical results and show in a practically relevant instance, e.g., single-pixel camera, that the target signal can be recovered from a few Hadamard measurements.

Compact multispectral pushframe camera for nanosatellites

In this paper we present an evolution of the single-pixel camera architecture, called "pushframe," which addresses the limitations of pushbroom cameras in space-based applications. In particular, it is well-suited to observing fast-moving scenes while retaining high spatial resolution and sensitivity. We show that the system is capable of producing color images with good fidelity and scalable resolution performance. The principle of our design broadens the choice of spectral ranges that can be captured, making it suitable for wide spectral ranges of infrared imaging.

Single-pixel imaging 12 years on: a review.

Modern cameras typically use an array of millions of detector pixels to capture images. By contrast, single-pixel cameras use a sequence of mask patterns to filter the scene along with the corresponding measurements of the transmitted intensity which is recorded using a single-pixel detector. This review considers the development of single-pixel cameras from the seminal work of Duarte et al. up to the present state of the art. We cover the variety of hardware configurations, design of mask patterns and the associated reconstruction algorithms, many of which relate to the field of compressed sensing and, more recently, machine learning. Overall, single-pixel cameras lend themselves to imaging at non-visible wavelengths and with precise timing or depth resolution. We discuss the suitability of single-pixel cameras for different application areas, including infrared imaging and 3D situation awareness for autonomous vehicles.