Compressive Spectral Imaging Research Articles

Freeform-surface-based optical design of a broadband compressive spectral imager with co-aperture coding

Compressive spectral imaging (CSI) is an advanced computational imaging approach to reconstruct the three-dimensional (3D) spatio-spectral data cube of a target scene through a single or a few snapshots. However, limited by the response range of the image detector, the existing CSI systems mostly work within narrow spectral bands, such as the visible or shortwave-infrared (SWIR) spectral band. The work band of the CSI system constrains the detection capacity for the targets under complex environments (such as rain, snow, haze, etc.). In addition, most of the current CSI prototypes lack engineering design for practical applications. This paper develops a novel, to the best of our knowledge, optical design scheme of a broadband CSI system with co-aperture coding to simultaneously realize visible multi-spectral imaging (10 channels) and SWIR super-resolution imaging (16×). The freeform surfaces are used to design the front-end reflective objective lens, thus significantly improving the image quality and spatial modulation precision of the system. By means of performance evaluation and tolerance analysis, excellent image quality and manufacturability of the proposed system are demonstrated.

Compressive single-pixel spectral imaging with spatial-spectral modulation optimization via coherence minimization

Compressive spectral imaging (CSI) obtains spatial-spectral information from under-sampled measurements, which solves the contradiction between imaging speed and resolution in conventional scanning-based spectral imaging systems. The spatial-spectral modulation and the corresponding sparse prior in reconstruction algorithm jointly determine the reconstruction quality of CSI systems. In this paper, a compressive single-pixel spectral imaging system with the spatial-spectral modulation and the sparse prior simultaneously optimized by coherence minimization is proposed. By formulating the modulation of the single-pixel spectral imaging system and the sensing coherence in the differentiable matrix notation, the gradients of the modulation and the sparse prior with respect to the sensing coherence are derived for both spatial and spectral dimensions. The spatial-spectral modulation and the sparse prior are optimized via gradient descent to minimize the sensing coherence, for both the decoupled CSI systems with one dimension completely sampled and the coupled CSI system with both spatial-spectral dimensions compressively sampled. For the optimized spatial-spectral modulation, high relative mutual differences in the spectral dimension as well as low redundancy patterns in the spatial dimension are achieved, and by combining with the optimized spatial-spectral sparse prior, the enhancement of 5.7 dB for PSNR compared to the unoptimized system is realized.

Fast matrix inversion in compressive spectral imaging based on a tensorial representation

Fast matrix inversion in compressive spectral imaging based on a tensorial representation

Anti-laser interference methods for compressive spectral imaging based on grayscale coded aperture

Coded aperture snapshot spectral imaging (CASSI) captures three-dimensional (3D) spectral images by one or several snapshots two-dimensional (2D) measurements. At present, in practical application of spectral imaging systems, artificial interference such as laser irradiation is wide-spread. When CASSI system is irradiated by intense laser, optoelectronic image sensor will inevitably be dazzled, leading to the loss of compressive sampling information, thereby reducing the reconstruction accuracy of spectral images. This paper establishes the imaging model of CASSI system under laser interference and proposes a grayscale coding design method. Through the design of a reasonable and feasible feedback model, the grayscale coded aperture is calculated according to the measurements of CASSI system in real time feedback. The intensity of the incident light of the system is modulated to reduce the power density on the sensor, thus avoiding sensor saturation, and improving the anti-laser interference threshold of the CASSI system. Simulation results show the improvement of reconstruction accuracy when grayscale coded apertures are employed.

Degradation Estimation Recurrent Neural Network With Local and Non-Local Priors for Compressive Spectral Imaging

Degradation Estimation Recurrent Neural Network With Local and Non-Local Priors for Compressive Spectral Imaging

High-Fidelity compressive spectral image reconstruction through a novel Non-Convex Non-Local Low-Rank tensor approximation model

Low-quality reconstruction is one main factor that greatly limits the development of snapshot compressive imaging (SCI) systems. The common algorithms usually unfold a 3D hyperspectral image (HSI) into a vector or matrix which inevitably corrupts the intrinsic structure of the HSI and insufficiently discovers its prior information. To fully exploit the underlying structures and internal correlations of the HSIs, this paper proposes a novel non-convex model based on Non-Local low-rank Tensor Approximation with Hyper-Laplacian priori (HL), named HL-NLTA, which effectively combines local and non-local similarity information from fused spectral and spatial perspectives. Specifically, a non-local low-rank tensor model based on the minimax concave plus (MCP) penalty and log sum (LS) penalty separately is developed which can simultaneously exploit two intrinsic properties of HSI, i.e., global correlation along the spectral domain (GCAS) and non-local self-similarity along the spatial domain (NSAS). Furthermore, the Hyper-Laplacian priori regularized through a non-convex ℓp (0 < p < 1) norm can preserve well the spectral and spatial structure. An optimization algorithm based on the alternating direction multiplier method (ADMM) framework is designed and accelerated by the Generalized shrinkage/thresholding (GST) algorithm and the fast Fourier transform (FFT) to solve the new nonconvex model. Extensive simulation results on public datasets and experimental results on our real CASSI system indicate that the proposed method enables high-fidelity HSI reconstructions in terms of simultaneous spatial structure detail recovery and spectral preservation. These results verify the effectiveness of the proposed model for snapshot compressive imaging (e.g., Coded aperture snapshot spectral imaging CASSI).

3D-Memory efficient listless set partitioning in hierarchical trees for hyperspectral image sensors

Hyperspectral (HS) images contain rich spatial and spectral information. Due to its large size, it is difficult to store, process, analyze, or transmit the critical information contained in it. The compression of hyperspectral images is inevitable. Many transform based Hyper Spectral Image Compression Algorithms (HSICAs) have been proposed in the past that work for both lossy and lossless compression processes. The transform based HSICA uses linked lists or dedicated markers or array structure to keep track of significant and insignificant sets or coefficients of a transformed HS image. However, these algorithms either suffered from low coding efficiency, high memory requirements, or high coding complexity. This work proposes a transform based HSICA using a curvelet transform to improve the directional elements and the ability to represent edges and other singularities along curves. The proposed HSICA aims to provide superior quality compressed HS images by representing HS images at different scales and directions and to achieve a high compression ratio. Experimental results show that the proposed algorithm has a low coding memory requirement with a 2% to 5% increase in coding gain compared to the other state of art compression algorithms.

Dual-Channel Switchable Metasurface Filters for Compact Spectral Imaging with Deep Compressive Reconstruction.

Spectral imaging technology, which aims to capture images across multiple spectral channels and create a spectral data cube, has been widely utilized in various fields. However, conventional spectral imaging systems face challenges, such as slow acquisition speed and large size. The rapid development of optical metasurfaces, capable of manipulating light fields versatilely and miniaturizing optical components into ultrathin planar devices, offers a promising solution for compact hyperspectral imaging (HSI). This study proposes a compact snapshot compressive spectral imaging (SCSI) system by leveraging the spectral modulations of metasurfaces with dual-channel switchable metasurface filters and employing a deep-learning-based reconstruction algorithm. To achieve compactness, the proposed system integrates dual-channel switchable metasurface filters using twisted nematic liquid crystals (TNLCs) and anisotropic titanium dioxide (TiO2) nanostructures. These thin metasurface filters are closely attached to the image sensor, resulting in a compact system. The TNLCs possess a broadband linear polarization conversion ability, enabling the rapid switching of the incidence polarization state between x-polarization and y-polarization by applying different voltages. This polarization conversion facilitates the generation of two groups of transmittance spectra for wavelength-encoding, providing richer information for spectral data cube reconstruction compared to that of other snapshot compressive spectral imaging techniques. In addition, instead of employing classic iterative compressive sensing (CS) algorithms, an end-to-end residual neural network (ResNet) is utilized to reconstruct the spectral data cube. This neural network leverages the 2-frame snapshot measurements of orthogonal polarization channels. The proposed hyperspectral imaging technology demonstrates superior reconstruction quality and speed compared to those of the traditional compressive hyperspectral image recovery methods. As a result, it is expected that this technology will have substantial implications in various domains, including but not limited to object detection, face recognition, food safety, biomedical imaging, agriculture surveillance, and so on.

Mixture-Net: Low-rank deep image prior inspired by mixture models for spectral image recovery

This paper proposes a non-data-driven deep neural network for spectral image recovery problems such as denoising, single hyperspectral image super-resolution, and compressive spectral imaging reconstruction. Unlike previous methods, the proposed approach, dubbed Mixture-Net, implicitly learns the prior information through the network. Mixture-Net consists of a deep generative model whose layers are inspired by the linear and non-linear low-rank mixture models, where the recovered image is composed of a weighted sum between the linear and non-linear decomposition. Mixture-Net also provides a low-rank decomposition interpreted as the spectral image abundances and endmembers, helpful in achieving remote sensing tasks without running additional routines. The experiments show the Mixture-Net effectiveness outperforming state-of-the-art methods in recovery quality with the advantage of architecture interpretability.

A method of reconstructing compressive spectral imaging with a complementary prior constraint

Compressed spectral imaging (CSI) is a technique for acquiring a cube of spectral image data in a single snapshot. In this paper, we propose a reconstruction method for the CSI system that integrates complementary prior constraints on spectral data to significantly enhance the accuracy of reconstructed data. The fundamental concepts are as follows: (1) Firstly, by conducting a thorough analysis of the corresponding feature maps of spectral data on the convolution dictionaries, we confirm the feasibility of utilizing the stationary response characteristics of spectral data on the convolution dictionaries as a reconstruction constraint to enhance spectral accuracy. (2) To compensate for the limitations of Convolutional Sparse Coding in reconstructing low-frequency signals, this paper proposes utilizing iterative Total Variation operators to estimate the low-frequency portion of spectral data. This approach results in improved performance in reconstructing low-frequency data and noise suppression. (3) Convolutional Sparse Coding and Total Variation are utilized as constraints for high and low-frequency complementary reconstruction, resulting in a multi-constraint solution problem within the Plug-and-Play framework that simplifies the overall reconstruction process. The proposed method surpasses the state-of-the-art reconstruction methods in both spatial reconstruction quality and spectral accuracy, while also significantly enhancing the level of detail in reconstructed spectral images.

Prior Image Guided Snapshot Compressive Spectral Imaging.

Spectral images with rich spatial and spectral information have wide usage, however, traditional spectral imaging techniques undeniably take a long time to capture scenes. We consider the computational imaging problem of the snapshot spectral spectrometer, i.e., the Coded Aperture Snapshot Spectral Imaging (CASSI) system. For the sake of a fast and generalized reconstruction algorithm, we propose a prior image guidance-based snapshot compressive imaging method. Typically, the prior image denotes the RGB measurement captured by the additional uncoded panchromatic camera of the dual-camera CASSI system. We argue that the RGB image as a prior image can provide valuable semantic information. More importantly, we design the Prior Image Semantic Similarity (PIDS) regularization term to enhance the reconstructed spectral image fidelity. In particular, the PIDS is formulated as the difference between the total variation of the prior image and the recovered spectral image. Then, we solve the PIDS regularized reconstruction problem by the Alternating Direction Method of Multipliers (ADMM) optimization algorithm. Comprehensive experiments on various datasets demonstrate the superior performance of our method.

Hybrid compressive spectral imaging with coded aperture optimization based on coherence minimization

Compressive spectral imaging (CSI) obtains spectral images through reconstruction from under-sampled low-dimensional projections with spatial and spectral coding. However, the simultaneous realization of high spatial–spectral resolution is challenged due to limitations of a single CSI architecture. In this paper, a hybrid CSI system combining the coded aperture snapshot spectral imaging (CASSI) and the spectrometer-based single-pixel spectral imaging (SPSI), with the corresponding coded aperture optimization for both constituent systems are proposed. The hybrid CSI system sustains the complementary advantages of the spatial resolution in CASSI and the spectral fidelity in SPSI, ensuring high reconstruction quality in both spatial and spectral dimensions with increased sampling efficiency. The derivable matrix operation of the hybrid CSI system is formulated; meanwhile, based on coherence minimization, the individual optimization on the coded apertures in the CASSI or the SPSI system, and the joint optimization on the coded apertures in the hybrid system are demonstrated. The reconstruction is further improved with the optimization, indicated by the quantitative image quality metrics and the point spectra. Compared to the conventional unoptimized CASSI, the optimized hybrid CSI system improves the PSNR for more than 7 dB. This work may introduce opportunities for multipath compressive imaging systems toward high-efficiency, high-resolution future.

Superpixel-based linear reconstruction method for dual-camera compressed hyperspectral imaging system

Compressive spectral imaging systems have received significant research attention in recent years. Among these systems, the dual-camera compressive hyperspectral imaging (DCCHI) design, has the potential to improve imaging resolution while maintaining the snapshot advantage. In this paper, we propose superpixel-based linear reconstruction methods for DCCHI system. Specifically, we assume that the spectral intensity of the superpixels generated from the RGB image varies linearly in the spectral dimension. Instead of addressing an ill-posed problem, we reconstruct hyperspectral images via solving overdetermined equations using the least squares method, resulting in a significant reduction in both computation time and memory usage. To further enhance the reconstruction accuracy, we extend the spectra of superpixels from a 1D linear space into a 3D space with an RGB spectral response matrix. Despite the simplicity of our method, it offers improved reconstruction accuracy of hyperspectral images compared to other state-of-the-art methods. Furthermore, it is more efficient in terms of memory usage and computational time.

Grating-based coded aperture compressive spectral imaging to reconstruct over 190 spectral bands from a snapshot measurement

We report snapshot compressive spectral imaging using a grating and coded aperture. A vision-transformer-based deep learning algorithm is developed for high-quality reconstruction. Experimental results demonstrate that over 190 spectral bands can be reconstructed from a single measurement from 485–657 nm with spectral accuracy of about 2 nm and spectral resolution up to 1.5 nm.

D$^\text{2}$UF: Deep Coded Aperture Design and Unrolling Algorithm for Compressive Spectral Image Fusion

Compressive spectral imaging (CSI) has attracted significant attention since it employs synthetic apertures to codify spatial and spectral information, sensing only 2D projections of the 3D spectral image. However, these optical architectures suffer from a trade-off between the spatial and spectral resolution of the reconstructed image due to technology limitations. To overcome this issue, compressive spectral image fusion (CSIF) employs the projected measurements of two CSI architectures with different resolutions to estimate a high-spatial high-spectral resolution. This work presents the fusion of the compressive measurements of a low-spatial high-spectral resolution coded aperture snapshot spectral imager (CASSI) architecture and a high-spatial low-spectral resolution multispectral color filter array (MCFA) system. Unlike previous CSIF works, this paper proposes joint optimization of the sensing architectures and a reconstruction network in an end-to-end (E2E) manner. The trainable optical parameters are the coded aperture (CA) in the CASSI and the colored coded aperture in the MCFA system, employing a sigmoid activation function and regularization function to encourage binary values on the trainable variables for an implementation purpose. Additionally, an unrolling-based network inspired by the alternating direction method of multipliers (ADMM) optimization is formulated to address the reconstruction step and the acquisition systems design jointly. Finally, a spatial-spectral inspired loss function is employed at the end of each unrolling layer to increase the convergence of the unrolling network. The proposed method outperforms previous CSIF methods, and experimental results validate the method with real measurements.

A New Spectral Compression Method based on the Minimization of the Color Difference and Root Mean Square Error

This paper describes two new interim connection spaces (ICSs), P-2OC and P-3OC, for spectral image compression and reconstruction. For this type of ICS, the weighting table or compression matrix H is modelled as a variable. The associated reconstruction matrix N is chosen as the Weiner estimation matrix. The objective function f(H) is the combination of the averages of the CIEDE2000 color difference (&#x394;E00) and the root mean square error (RMSE) between the original and reconstructed reflectance values. Hence, the compression matrix H is determined by solving the nonlinear minimization problem based on the Munsell training dataset. The proposed ICSs, P-2OC and P-3OC, were tested and compared, respectively, with ICS-2SI and ICS-3SI developed by Zhang et&#xA0;al.&#xA0;(JOSAA, Vol. 29, pp. 1027&#x2013;1034) in 2012 using the NCS dataset, and 2 spectral images. Performance tests showed that the proposed P-2OC and P-3OC ICSs are better than the ICS-2SI and ICS-3SI ICSs, respectively, in terms of RMSE, goodness of fit coefficient (GFC), and &#x394;E00 under CIE illuminants D65, A, C and F11. Therefore, it is expected that the P-2OC and P-3OC ICSs can find applications in spectral image compression and cross-media reproduction.

Mid-Wave Infrared Snapshot Compressive Spectral Imager with Deep Infrared Denoising Prior

Although various infrared imaging spectrometers have been studied, most of them are developed under the Nyquist sampling theorem, which severely burdens 3D data acquisition, storage, transmission, and processing, in terms of both hardware and software. Recently, computational imaging, which avoids direct imaging, has been investigated for its potential in the visible field. However, it has been rarely studied in the infrared domain, as it suffers from inconsistency in spectral response and reconstruction. To address this, we propose a novel mid-wave infrared snapshot compressive spectral imager (MWIR-SCSI). This design scheme provides a high degree of randomness in the measurement projection, which is more conducive to the reconstruction of image information and makes spectral correction implementable. Furthermore, leveraging the explainability of model-based algorithms and the high efficiency of deep learning algorithms, we designed a deep infrared denoising prior plug-in for the optimization algorithm to perform in terms of both imaging quality and reconstruction speed. The system calibration obtains 111 real coded masks, filling the gap between theory and practice. Experimental results on simulation datasets and real infrared scenarios prove the efficacy of the designed deep infrared denoising prior plug-in and the proposed acquisition architecture that acquires mid-infrared spectral images of 640 pixels × 512 pixels × 111 spectral channels at an acquisition frame rate of 50 fps.

Compressive Spectral Imaging via Misalignment Induced Equivalent Grayscale Coded Aperture

Coded aperture snapshot spectral imager (CASSI) senses the spectral information of a 2D scene and captures a set of coded measurement data that can be used to reconstruct the 3D spatio-spectral datacube of the input scene by compressive sensing algorithms. The coded aperture (CA) in CASSI plays a crucial role in modulating the spatial information. The pixels in CA are typically square, switched binary on-off, and aligned with the pixels of focal plane array (FPA). Instead of this binary modulation, this paper explores a simple yet effective approach to enabling an equivalent grayscale modulation, which can increase the sensing degree of freedom in CASSI systems. In particular, we deliberately introduce misalignment between the CA pixels and the FPA pixels, such that the spatial modulation of one FPA pixel is determined by four adjacent CA pixels instead of one. Numerical experiments show that the proposed equivalent grayscale modulation induced by misalignment can significantly improve the CASSI reconstruction when compared with current methods, whether a random CA or an optimal blue noise CA is used. More importantly, it does not incur in any cost to the CASSI system.

High Resolution LED-Based Snapshot Compressive Spectral Video Imaging With Deep Neural Networks

High Resolution LED-Based Snapshot Compressive Spectral Video Imaging With Deep Neural Networks

A Refinement Boosted and Attention Guided Deep FISTA Reconstruction Framework for Compressive Spectral Imaging

Hyperspectral images (HSIs) contain rich spatial and spectral information. A double dispersers coded aperture snapshot spectral imaging (DD-CASSI) system takes advantage of compressive sensing (CS) theory to map 3D HSI data into a single 2D measurement. One of key components of DD-CASSI is to reconstruct high quality hyperspectral image from measurement. Traditional model-based methods use mathematical optimization to reconstruct hyperspectral images according to prior knowledge. Current deep learning based methods achieve pleasant results. But fully learned deep learning methods lack interpretability, and model-based deep learning methods cannot achieve pleasant performance. In this paper, we propose a novel HSI reconstruction framework named Refinement Boosted and Attention Guided Tensor FISTA(Fast Iterative Shrinkage-Thresholding Algorithm)-Net (ReAttFISTA-Net), which combines model-based deep learning and fully learned deep learning reconstruction strategies. In this framework, we introduces Attention Guided Fusion Mechanism which enhances spatial-spectral information, refinement sub-network and auxiliary loss terms to improve the reconstruction performance. Extensive experimental results show that the proposed reconstruction algorithm outperforms the state-of-the-art algorithms on both simulation and real-world datasets.