Compressive Imaging System Research Articles

Mask-Guided Spatial-Spectral MLP Network for High-Resolution Hyperspectral Image Reconstruction.

Hyperspectral image (HSI) reconstruction is a critical and indispensable step in spectral compressive imaging (CASSI) systems and directly affects our ability to capture high-quality images in dynamic environments. Recent research has increasingly focused on deep unfolding frameworks for HSI reconstruction, showing notable progress. However, these approaches have to break the optimization task into two sub-problems, solving them iteratively over multiple stages, which leads to large models and high computational overheads. This study presents a simple yet effective method that passes the degradation information (sensing mask) through a deep learning network to disentangle the degradation and the latent target's representations. Specifically, we design a lightweight MLP block to capture non-local similarities and long-range dependencies across both spatial and spectral domains, and investigate an attention-based mask modelling module to achieve the spatial-spectral-adaptive degradation representationthat is fed to the MLP-based network. To enhance the information flow between MLP blocks, we introduce a multi-level fusion module and apply reconstruction heads to different MLP features for deeper supervision. Additionally, we combine the projection loss from compressive measurements with reconstruction loss to create a dual-domain loss, ensuring consistent optical detection during HS reconstruction. Experiments on benchmark HS datasets show that our method outperforms state-of-the-art approaches in terms of both reconstruction accuracy and efficiency, reducing computational and memory costs.

Degradation-aware deep unfolding network with transformer prior for video compressive imaging

In video snapshot compressive imaging (SCI) systems, video reconstruction methods are used to recover spatial–temporal-correlated video frame signals from a compressed measurement. While unfolding methods have demonstrated promising performance, they encounter two challenges: (1) They lack the ability to estimate degradation patterns and the degree of ill-posedness from video SCI, which hampers guiding and supervising the iterative learning process. (2) The prevailing reliance on 3D-CNNs in these methods limits their capacity to capture long-range dependencies. To address these concerns, this paper introduces the Degradation-Aware Deep Unfolding Network (DADUN). DADUN leverages estimated priors from compressed frames and the physical mask to guide and control each iteration. We also develop a novel Bidirectional Propagation Convolutional Recurrent Neural Network (BiP-CRNN) that simultaneously captures both intra-frame contents and inter-frame dependencies. By plugging BiP-CRNN into DADUN, we establish a novel end-to-end (E2E) and data-dependent deep unfolding method, DADUN with transformer prior (TP), for video sequence reconstruction. Experimental results on various video sequences show the effectiveness of our proposed approach, which is also robust to random masks and has wide generalization bounds.

Compressive single-pixel spectral imaging with spatial-spectral modulation optimization via coherence minimization

Compressive spectral imaging (CSI) obtains spatial-spectral information from under-sampled measurements, which solves the contradiction between imaging speed and resolution in conventional scanning-based spectral imaging systems. The spatial-spectral modulation and the corresponding sparse prior in reconstruction algorithm jointly determine the reconstruction quality of CSI systems. In this paper, a compressive single-pixel spectral imaging system with the spatial-spectral modulation and the sparse prior simultaneously optimized by coherence minimization is proposed. By formulating the modulation of the single-pixel spectral imaging system and the sensing coherence in the differentiable matrix notation, the gradients of the modulation and the sparse prior with respect to the sensing coherence are derived for both spatial and spectral dimensions. The spatial-spectral modulation and the sparse prior are optimized via gradient descent to minimize the sensing coherence, for both the decoupled CSI systems with one dimension completely sampled and the coupled CSI system with both spatial-spectral dimensions compressively sampled. For the optimized spatial-spectral modulation, high relative mutual differences in the spectral dimension as well as low redundancy patterns in the spatial dimension are achieved, and by combining with the optimized spatial-spectral sparse prior, the enhancement of 5.7 dB for PSNR compared to the unoptimized system is realized.

Supervised-unsupervised combined transformer for spectral compressive imaging reconstruction

To solve the low spatial and/or temporal resolution problem which the conventional hyperspectral cameras often suffer from, spectral compressive imaging systems (SCI) have attracted more attention recently. Recovering a hyperspectral image (HSI) from its corresponding 2D coded image is an ill-posed inverse problem, and learning accurate prior from HSI and 2D coded image is essential to solve this inverse problem. Existing methods only use supervised networks that focus on learning generalized prior from training datasets, or only use unsupervised networks that focus on learning specific prior from 2D coded image, resulting in the inability to learn both generalized and specific priors. Also, when learning the priors, existing methods cannot simultaneously give consideration to both global and local scales, as well as both spatial and spectral dimensions. To cope with this problem, in this paper, we propose a Supervised-Unsupervised Combined Transformer Network (SUCTNet) composed by a supervised Spatio-spectral Transformer network (SSTNet) and an Unsupervised Multi-level Feature Refinement network (UMFRNet). Specifically, we first develop the SSTNet to learn generalized prior and obtain a preliminary HSI. In SSTNet, the proposed spatial encoding and spectral decoding network architecture enables it to simultaneously consider both spatial and spectral dimensions, and a proposed Global and Local Multi head Self Attention block (GL-MSA) enables it simultaneously to consider both global and local scales. Then, the preliminary HSI is fed into the proposed UMFRNet to learn specific prior and obtain the target HSI. In UMFRNet, a proposed multi-level feature refinement mechanism and the physical imaging model of SCI are used to improve reconstruction accuracy and generalization performance. Extensive experiments show that our method significantly outperforms state-of-the-art (SOTA) methods on simulated and real datasets. Codes will be available at https://github.com/Vzhouhan/SUCTNet.

Simulation of compressed ultrafast imaging system based on streak tube

Simulation of compressed ultrafast imaging system based on streak tube

Video snapshot compressive imaging using adaptive progressive coding for high-quality reconstruction under different illumination circumstances.

We consider capturing high-speed color video under different illumination conditions using a video snapshot compressive imaging system (video SCI). An adaptive progressive coding method is proposed, and we conduct an integrated design of the imaging system in terms of optics, mechanics, and control. Compared to previous video SCI systems, this adaptive progressive coding method mitigates the image stability issues in various illumination conditions, ensuring high-quality imaging while greatly improving the light throughput of the system. Based on the analysis of both simulation and real experimental results, we found that this imaging system can achieve color video shooting under an illumination range of 2 lux to 60 lux.

High-Fidelity compressive spectral image reconstruction through a novel Non-Convex Non-Local Low-Rank tensor approximation model

Low-quality reconstruction is one main factor that greatly limits the development of snapshot compressive imaging (SCI) systems. The common algorithms usually unfold a 3D hyperspectral image (HSI) into a vector or matrix which inevitably corrupts the intrinsic structure of the HSI and insufficiently discovers its prior information. To fully exploit the underlying structures and internal correlations of the HSIs, this paper proposes a novel non-convex model based on Non-Local low-rank Tensor Approximation with Hyper-Laplacian priori (HL), named HL-NLTA, which effectively combines local and non-local similarity information from fused spectral and spatial perspectives. Specifically, a non-local low-rank tensor model based on the minimax concave plus (MCP) penalty and log sum (LS) penalty separately is developed which can simultaneously exploit two intrinsic properties of HSI, i.e., global correlation along the spectral domain (GCAS) and non-local self-similarity along the spatial domain (NSAS). Furthermore, the Hyper-Laplacian priori regularized through a non-convex ℓp (0 < p < 1) norm can preserve well the spectral and spatial structure. An optimization algorithm based on the alternating direction multiplier method (ADMM) framework is designed and accelerated by the Generalized shrinkage/thresholding (GST) algorithm and the fast Fourier transform (FFT) to solve the new nonconvex model. Extensive simulation results on public datasets and experimental results on our real CASSI system indicate that the proposed method enables high-fidelity HSI reconstructions in terms of simultaneous spatial structure detail recovery and spectral preservation. These results verify the effectiveness of the proposed model for snapshot compressive imaging (e.g., Coded aperture snapshot spectral imaging CASSI).

High precision reconstruction for compressed ultrafast off-axis holography

Existing compressed ultrafast photography (CUP) mainly acquires the light intensity and loses the phase information that characterizes the spatial features. In this paper, we propose a compressed ultrafast holographic imaging system to achieve phase measurement in ultrafast dynamic samples. In the proposed imaging system, the Mach-Zehnder interference structure is incorporated into compressed ultrafast photography. Based on this imaging system, a reconstruction method combining the plug-and-play generalized alternating projection (PnP-GAP) framework and the depth filtering of prior FastDVDNet is proposed to recover the hologram sequence in ultrafast temporal resolution. FastDVDNet employs the correlation between frames to enhance denoising performance and maintain temporal continuity, thus obtaining better results for ultrafast hologram sequence reconstruction. The simulation results indicate that the proposed method outperforms other reconstruction methods in terms of average peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

Compressive video via IR-pulsed illumination.

We propose and demonstrate a compressive temporal imaging system based on pulsed illumination to encode temporal dynamics into the signal received by the imaging sensor during exposure time. Our approach enables >10x increase in effective frame rate without increasing camera complexity. To mitigate the complexity of the inverse problem during reconstruction, we introduce two keyframes: one before and one after the coded frame. We also craft what we believe to be a novel deep learning architecture for improved reconstruction of the high-speed scenes, combining specialized convolutional and transformer architectures. Simulation and experimental results clearly demonstrate the reconstruction of high-quality, high-speed videos from the compressed data.

Dual-Channel Switchable Metasurface Filters for Compact Spectral Imaging with Deep Compressive Reconstruction.

Spectral imaging technology, which aims to capture images across multiple spectral channels and create a spectral data cube, has been widely utilized in various fields. However, conventional spectral imaging systems face challenges, such as slow acquisition speed and large size. The rapid development of optical metasurfaces, capable of manipulating light fields versatilely and miniaturizing optical components into ultrathin planar devices, offers a promising solution for compact hyperspectral imaging (HSI). This study proposes a compact snapshot compressive spectral imaging (SCSI) system by leveraging the spectral modulations of metasurfaces with dual-channel switchable metasurface filters and employing a deep-learning-based reconstruction algorithm. To achieve compactness, the proposed system integrates dual-channel switchable metasurface filters using twisted nematic liquid crystals (TNLCs) and anisotropic titanium dioxide (TiO2) nanostructures. These thin metasurface filters are closely attached to the image sensor, resulting in a compact system. The TNLCs possess a broadband linear polarization conversion ability, enabling the rapid switching of the incidence polarization state between x-polarization and y-polarization by applying different voltages. This polarization conversion facilitates the generation of two groups of transmittance spectra for wavelength-encoding, providing richer information for spectral data cube reconstruction compared to that of other snapshot compressive spectral imaging techniques. In addition, instead of employing classic iterative compressive sensing (CS) algorithms, an end-to-end residual neural network (ResNet) is utilized to reconstruct the spectral data cube. This neural network leverages the 2-frame snapshot measurements of orthogonal polarization channels. The proposed hyperspectral imaging technology demonstrates superior reconstruction quality and speed compared to those of the traditional compressive hyperspectral image recovery methods. As a result, it is expected that this technology will have substantial implications in various domains, including but not limited to object detection, face recognition, food safety, biomedical imaging, agriculture surveillance, and so on.

Bi-channel compressive hyperspectral imager based on polarization conversion metasurface

Metasurfaces have demonstrated remarkable capabilities in manipulating light fields and reducing the size of conventional optical elements to ultrathin devices, and compressive hyperspectral imaging systems can effectively utilize the spectral modulations of metasurfaces through spectral transmission tuning. In this paper, a bi-channel compressive hyperspectral imager based on polarization conversion metasurfaces is proposed. The metasurface layer of this device is able to modulate and convert the polarized incidence into two outputs of different transmission spectra with orthogonal polarizations, providing two channels of hyperspectral information that can be detected by a polarization image sensor in one snapshot. Additionally, this design offers a compact size, high speed, and accurate reconstruction through the utilization of a macro-pixel segmentation method for compressive sensing algorithms, and it is firmly demonstrated that the proposed hyperspectral imaging technology surpasses other traditional compressive hyperspectral image recovery methods in terms of improved reconstruction quality. Therefore, it is envisioned that this technology will find significant applications in various fields, including food safety, biomedical imaging, precision agriculture, object detection and so on.

Provable deep video denoiser using spatial–temporal information for video snapshot compressive imaging: Algorithm and convergence analysis

Video snapshot compressive imaging (SCI) is a new compressive imaging system that aims to recover multiple video frames from a single measurement. Recent plug-and-play (PnP) imaging methods have been used for solving SCI inverse problems by leveraging pre-trained deep Gaussian denoisers. Under the condition of diminishing noise levels, a necessary assumption for the fixed-point convergence of PnP imaging algorithms is that the plugged denoisers are bounded denoisers. However, it is difficult to prove that existing deep Gaussian denoisers meet this assumption due to the complex network architectures, limiting the convergence analysis of these algorithms. This paper aims to elaborate a bounded deep video denoiser to remedy such a gap. Concretely, we propose a denoiser using double tight frames dubbed as VDTF and plug it into the PnP framework to construct the PnP-VDTF algorithm. In VDTF, a constant network (CNet) is designed, wherein a Swin Transformer-based module and a 3D convolution module extract the spatial information of the global features and the temporal information of the input video, respectively. Theoretically, we provide a strict boundary proof of VDTF, and show that PnP-VDTF can generate fixed-point convergent trajectories. Experiments show that PnP-VDTF can achieve higher-quality reconstructions compared with benchmark SCI algorithms.

Redundant compressed single-pixel hyperspectral imaging system

As single-pixel imaging technology is highly sensitive and low-cost, a large number of hyperspectral imaging methods based on single-pixel have been proposed. Although the image of each band can be reconstructed independently, the parameters that need to be computed for reconstructing hyperspectral images will increase in parallel with the number of bands. Therefore, in this paper, we propose a redundant compressed single-pixel hyperspectral imaging system (RCSHI), which takes full advantage of the spectral redundancy of hyperspectral images by jointly reconstructing images from different bands and compressing the spectral dimension of the data. RCSHI reduces the amount of redundant hyperspectral information stacking and the number of parameters that need to be computed to reconstruct hyperspectral images. In addition, we use progressive fusion enhancement to recover the compressed spectral dimension and enhance the spatial dimension of the images. Experimental results show that RCSHI is able to acquire hyperspectral data with high spatial and spectral resolution and performs well on various datasets and actual scenarios, especially at lower compressed sampling rates.

Hybrid compressive spectral imaging with coded aperture optimization based on coherence minimization

Compressive spectral imaging (CSI) obtains spectral images through reconstruction from under-sampled low-dimensional projections with spatial and spectral coding. However, the simultaneous realization of high spatial–spectral resolution is challenged due to limitations of a single CSI architecture. In this paper, a hybrid CSI system combining the coded aperture snapshot spectral imaging (CASSI) and the spectrometer-based single-pixel spectral imaging (SPSI), with the corresponding coded aperture optimization for both constituent systems are proposed. The hybrid CSI system sustains the complementary advantages of the spatial resolution in CASSI and the spectral fidelity in SPSI, ensuring high reconstruction quality in both spatial and spectral dimensions with increased sampling efficiency. The derivable matrix operation of the hybrid CSI system is formulated; meanwhile, based on coherence minimization, the individual optimization on the coded apertures in the CASSI or the SPSI system, and the joint optimization on the coded apertures in the hybrid system are demonstrated. The reconstruction is further improved with the optimization, indicated by the quantitative image quality metrics and the point spectra. Compared to the conventional unoptimized CASSI, the optimized hybrid CSI system improves the PSNR for more than 7 dB. This work may introduce opportunities for multipath compressive imaging systems toward high-efficiency, high-resolution future.

Deep Equilibrium Models for Snapshot Compressive Imaging

The ability of snapshot compressive imaging (SCI) systems to efficiently capture high-dimensional (HD) data has led to an inverse problem, which consists of recovering the HD signal from the compressed and noisy measurement. While reconstruction algorithms grow fast to solve it with the recent advances of deep learning, the fundamental issue of accurate and stable recovery remains. To this end, we propose deep equilibrium models (DEQ) for video SCI, fusing data-driven regularization and stable convergence in a theoretically sound manner. Each equilibrium model implicitly learns a nonexpansive operator and analytically computes the fixed point, thus enabling unlimited iterative steps and infinite network depth with only a constant memory requirement in training and testing. Specifically, we demonstrate how DEQ can be applied to two existing models for video SCI reconstruction: recurrent neural networks (RNN) and Plug-and-Play (PnP) algorithms. On a variety of datasets and real data, both quantitative and qualitative evaluations of our results demonstrate the effectiveness and stability of our proposed method. The code and models are available at: https://github.com/IndigoPurple/DEQSCI.

Superpixel-based linear reconstruction method for dual-camera compressed hyperspectral imaging system

Compressive spectral imaging systems have received significant research attention in recent years. Among these systems, the dual-camera compressive hyperspectral imaging (DCCHI) design, has the potential to improve imaging resolution while maintaining the snapshot advantage. In this paper, we propose superpixel-based linear reconstruction methods for DCCHI system. Specifically, we assume that the spectral intensity of the superpixels generated from the RGB image varies linearly in the spectral dimension. Instead of addressing an ill-posed problem, we reconstruct hyperspectral images via solving overdetermined equations using the least squares method, resulting in a significant reduction in both computation time and memory usage. To further enhance the reconstruction accuracy, we extend the spectra of superpixels from a 1D linear space into a 3D space with an RGB spectral response matrix. Despite the simplicity of our method, it offers improved reconstruction accuracy of hyperspectral images compared to other state-of-the-art methods. Furthermore, it is more efficient in terms of memory usage and computational time.

Frequency-domain compression imaging for extending the field of view of infrared thermometers.

We propose a computational imaging technique for expanding the field of view of infrared thermometers. The contradiction between the field of view and the focal length has always been a chief problem for researchers, especially in infrared optical systems. Large-area infrared detectors are expensive and technically arduous to be manufactured, which enormously limits the performance of the infrared optical system. On the other hand, the extensive use of infrared thermometers in COVID-19 has created a considerable demand for infrared optical systems. Therefore, improving the performance of infrared optical systems and increasing the utilization of infrared detectors is vital. This work proposes a multi-channel frequency-domain compression imaging method based on point spread function (PSF) engineering. Compared with conventional compressed sensing, the submitted method images once without an intermediate image plane. Furthermore, phase encoding is used without loss of illumination of the image surface. These facts can significantly reduce the volume of the optical system and improve the energy efficiency of the compressed imaging system. Therefore, its application in COVID-19 is of great value. We design a dual-channel frequency-domain compression imaging system to verify the proposed method's feasibility. Then, the wavefront coded PSF and optical transfer function (OTF) are used, and the two-step iterative shrinkage/thresholding (TWIST) algorithm is used to restore the image to get the final result. This compression imaging method provides a new idea for the large field of view monitoring systems, especially in infrared optical systems.

Experimental Study on Measuring Petzval Image Plane of Streak Tube with Single Image

In the process of image reconstruction of compressed sensing algorithms, building a measurement matrix related to the parameters of the imaging system is necessary to improve its imaging quality. Additionally, building a compressed ultrafast imaging system based on a streak camera, which includes aberrations in the imaging system, is necessary. However, the aberration coefficient of the streak tube can be obtained only by numerical calculation based on the known internal structure of the streak tube, and it does not apply to a tube with an unknown structure. Based on the Lagrange–Helmholtz relation, which is widely established in electronic optical imaging systems, this study proposes a method to obtain the Petzval image plane of a streak tube by measuring only a single image without considering the internal structure of the streak tube. This method provides a reference for the construction of the measurement matrix in the application of the compressed sensing algorithm; additionally, it provides a test scheme for the performance index of the streak tube after assembly in commercial production to further optimize the assembly process and improve the yield of production.

3D compressive imaging system with a single photon-counting detector.

For photon-counting based compressive imaging systems, it is difficult to obtain 3D image with intensity and depth information precisely due to the dead time and shot noise effect of photon-counting detectors. In this study, we design and achieve a 3D compressive imaging system using a single photon-counting detector. To overcome the radiometric distortion arising from the dead time and shot noise, considering the response mechanism of photon-counting detectors, a Bayesian posterior model is derived and a Reversible jump Markov chain Monte Carlo (RJMCMC)-based method is proposed to iteratively obtain model parameters. Experimental and simulation results indicate that the 3D image of targets can be effectively and accurately reconstructed with a smaller number of repeated illuminations and no longer restricted by the photon flux conditions (i.e., breaking through the upper limit of the received signal level). The proposed Bayesian RJMCMC-based radiometric correction method is not only beneficial to single-photon 3D compressive imaging system, but also to any other photon-counting based systems, e.g., photon-counting lidars. In addition, limiting condition of recovering the actual photon number for photon-counting imaging or lidar systems is also quantitatively analyzed, which is of great significance to the system scheme design.

Feasibility of a Real-Time Embedded Hyperspectral Compressive Sensing Imaging System.

Hyperspectral imaging has been attracting considerable interest as it provides spectrally rich acquisitions useful in several applications, such as remote sensing, agriculture, astronomy, geology and medicine. Hyperspectral devices based on compressive acquisitions have appeared recently as an alternative to conventional hyperspectral imaging systems and allow for data-sampling with fewer acquisitions than classical imaging techniques, even under the Nyquist rate. However, compressive hyperspectral imaging requires a reconstruction algorithm in order to recover all the data from the raw compressed acquisition. The reconstruction process is one of the limiting factors for the spread of these devices, as it is generally time-consuming and comes with a high computational burden. Algorithmic and material acceleration with embedded and parallel architectures (e.g., GPUs and FPGAs) can considerably speed up image reconstruction, making hyperspectral compressive systems suitable for real-time applications. This paper provides an in-depth analysis of the required performance in terms of computing power, data memory and bandwidth considering a compressive hyperspectral imaging system and a state-of-the-art reconstruction algorithm as an example. The results of the analysis show that real-time application is possible by combining several approaches, namely, exploitation of system matrix sparsity and bandwidth reduction by appropriately tuning data value encoding.