Coded Aperture Snapshot Spectral Imaging Research Articles

An interpretable and flexible fusion prior to boost hyperspectral imaging reconstruction

Hyperspectral image (HSI) reconstruction from the compressed measurement captured by the coded aperture snapshot spectral imager system remains a hot topic. Recently, deep-learning-based methods for HSI reconstruction have become the mainstream due to their high performance and efficiency in the testing inference. However, these learning methods do not fully utilize the abundant spectral information with proper physical spectral priors, resulting in complex architectures and unsatisfactory reconstruction performance. In this paper, we claim that the spectral low-rank property can still help these learning methods and propose a hyperspectral fusion theory, which demonstrates that full HSIs are mathematically equivalent to the closed-form combination of subspace images, mask, and measurements. Based on the above fusion theory, we propose the subspace distillation prior (SP) to efficiently cooperate with existing learning models to enhance the exploration of the spectral low-rank property. In detail, the SP can directly improve the testing inference of existing models (SP1, Section 4.1). Furthermore, SP can also be cooperated with exiting networks to formulate a new framework, which regularizes the existing models to learn the subspace images, and help to reconstruct the full HSIs from subspace images, mask, and measurements (SP2, Section 4.2). We choose six existing representative models for the HSI reconstruction experiments and find that SP1 and SP2 can, respectively, achieve improvements of 0.08 dB∼0.76 dB and 0.36 dB∼1.76 dB on the simulated datasets, demonstrating the advantage of the proposed hyperspectral fusion theory. The source code is available at https://github.com/prowDIY.

Snapshot spectral imaging based on aberration model-driven deep learning.

Coded aperture snapshot spectral imaging (CASSI) can capture hyperspectral images (HSIs) in one shot, but it suffers from optical aberrations that degrade the reconstruction quality. Existing deep learning methods for CASSI reconstruction lose some performance on real data due to aberrations. We propose a method to restore high-resolution HSIs from a low-resolution CASSI measurement. We first generate realistic training data that mimics the optical aberrations of CASSI using a spectral imaging simulation technique. A generative network is then trained on this data to recover HSIs from a blurred and distorted CASSI measurement. Our method adapts to the optical system degradation model and thus improves the reconstruction robustness. Experiments on both simulated and real data indicate that our method significantly enhances the image quality of reconstruction outcomes and can be applied to different CASSI systems.

Ultra-high-speed four-dimensional hyperspectral imaging

We propose, to the best of our knowledge, a novel deep learning–enabled four-dimensional spectral imaging system composed of a reflective coded aperture snapshot spectral imaging system and a panchromatic camera. The system simultaneously captures a compressively coded hyperspectral measurement and a panchromatic measurement. The hyperspectral data cube is recovered by the U-net-3D network. The depth information of the scene is then acquired by estimating a disparity map between the hyperspectral data cube and the panchromatic measurement through stereo matching. This disparity map is used to align the hyperspectral data cube and the panchromatic measurement. A designed fusion network is used to improve the spatial reconstruction of the hyperspectral data cube by fusing aligned panchromatic measurements. The hardware prototype of the proposed system demonstrates high-speed four-dimensional spectral imaging that allows for simultaneously acquiring depth and spectral images with an 8 nm spectral resolution between 450 and 700 nm, 2.5 mm depth accuracy, and a 1.83 s reconstruction time.

CasFormer: Cascaded transformers for fusion-aware computational hyperspectral imaging

Computational hyperspectral imaging (CHI) is a cutting-edge technique, which plays a pivotal role in breaking through the quality bottleneck of hyperspectral images (HSI). Among the techniques employed in this domain, the coded aperture snapshot spectral imaging (CASSI) system holds widespread recognition. Nevertheless, the imaging capability of CASSI remains limited due to the hardware conditions and the fragility of outcomes associated with the ill-posed blind reconstruction process. To this end, we propose a novel cascaded transformer architecture, termed CasFormer, specifically crafted for fusion-aware CHI by means of a dual-imaging mechanism. CasFormer facilitates the effective enhancement of hyperspectral imaging quality by fusing RGB images, with a focus on spatial and spectral domains. As the name suggests, CasFormer is primarily composed of a series of cascade-attention blocks, enabling the fusion of high-spatial-resolution RGB images through spatial coherence alignment and the recovery of spectrally sequential information more compactly and accurately. Furthermore, CasFormer incorporates physical constraints through a decoupling-based loss function, ensuring spatial consistency and spectral fidelity in the fusion-aware CHI process. Extensive experiments conducted across multiple datasets demonstrate the superiority of CasFormer in achieving high-quality imaging results compared to SOTA CHI algorithms. Our code and benchmark datasets will be openly accessible at https://github.com/danfenghong/Information_Fusion_CasFormer.

Dual-Window Multiscale Transformer for Hyperspectral Snapshot Compressive Imaging

Coded aperture snapshot spectral imaging (CASSI) system is an effective manner for hyperspectral snapshot compressive imaging. The core issue of CASSI is to solve the inverse problem for the reconstruction of hyperspectral image (HSI). In recent years, Transformer-based methods achieve promising performance in HSI reconstruction. However, capturing both long-range dependencies and local information while ensuring reasonable computational costs remains a challenging problem. In this paper, we propose a Transformer-based HSI reconstruction method called dual-window multiscale Transformer (DWMT), which is a coarse-to-fine process, reconstructing the global properties of HSI with the long-range dependencies. In our method, we propose a novel U-Net architecture using a dual-branch encoder to refine pixel information and full-scale skip connections to fuse different features, enhancing the extraction of fine-grained features. Meanwhile, we design a novel self-attention mechanism called dual-window multiscale multi-head self-attention (DWM-MSA), which utilizes two different-sized windows to compute self-attention, which can capture the long-range dependencies in a local region at different scales to improve the reconstruction performance. We also propose a novel position embedding method for Transformer, named con-abs position embedding (CAPE), which effectively enhances positional information of the HSIs. Extensive experiments on both the simulated and the real data are conducted to demonstrate the superior performance, stability, and generalization ability of our DWMT. Code of this project is at https://github.com/chenx2000/DWMT.

Anti-laser interference methods for compressive spectral imaging based on grayscale coded aperture

Coded aperture snapshot spectral imaging (CASSI) captures three-dimensional (3D) spectral images by one or several snapshots two-dimensional (2D) measurements. At present, in practical application of spectral imaging systems, artificial interference such as laser irradiation is wide-spread. When CASSI system is irradiated by intense laser, optoelectronic image sensor will inevitably be dazzled, leading to the loss of compressive sampling information, thereby reducing the reconstruction accuracy of spectral images. This paper establishes the imaging model of CASSI system under laser interference and proposes a grayscale coding design method. Through the design of a reasonable and feasible feedback model, the grayscale coded aperture is calculated according to the measurements of CASSI system in real time feedback. The intensity of the incident light of the system is modulated to reduce the power density on the sensor, thus avoiding sensor saturation, and improving the anti-laser interference threshold of the CASSI system. Simulation results show the improvement of reconstruction accuracy when grayscale coded apertures are employed.

Hyperspectral Compressive Snapshot Reconstruction via Coupled Low-Rank Subspace Representation and Self-Supervised Deep Network.

Coded aperture snapshot spectral imaging (CASSI) is an important technique for capturing three-dimensional (3D) hyperspectral images (HSIs), and involves an inverse problem of reconstructing the 3D HSI from its corresponding coded 2D measurements. Existing model-based and learning-based methods either could not explore the implicit feature of different HSIs or require a large amount of paired data for training, resulting in low reconstruction accuracy or poor generalization performance as well as interpretability. To remedy these deficiencies, this paper proposes a novel HSI reconstruction method, which exploits the global spectral correlation from the HSI itself through a formulation of model-driven low-rank subspace representation and learns the deep prior by a data-driven self-supervised deep learning scheme. Specifically, we firstly develop a model-driven low-rank subspace representation to decompose the HSI as the product of an orthogonal basis and a spatial representation coefficient, then propose a data-driven deep guided spatial-attention network (called DGSAN) to adaptively reconstruct the implicit spatial feature of HSI by learning the deep coefficient prior (DCP), and finally embed these implicit priors into an iterative optimization framework through a self-supervised training way without requiring any training data. Thus, the proposed method shall enhance the reconstruction accuracy, generalization ability, and interpretability. Extensive experiments on several datasets and imaging systems validate the superiority of our method. The source code and data of this article will be made publicly available at https://github.com/ChenYong1993/LRSDN.

RCUMP: Residual Completion Unrolling with Mixed Priors for Snapshot Compressive Imaging.

Deep unrolling-based snapshot compressive imaging (SCI) methods, which employ iterative formulas to construct interpretable iterative frameworks and embedded learnable modules, have achieved remarkable success in reconstructing 3-dimensional (3D) hyperspectral images (HSIs) from 2D measurement induced by coded aperture snapshot spectral imaging (CASSI). However, the existing deep unrolling-based methods are limited by the residuals associated with Taylor approximations and the poor representation ability of single hand-craft priors. To address these issues, we propose a novel HSI construction method named residual completion unrolling with mixed priors (RCUMP). RCUMP exploits a residual completion branch to solve the residual problem and incorporates mixed priors composed of a novel deep sparse prior and mask prior to enhance the representation ability. Our proposed CNN-based model can significantly reduce memory cost, which is an obvious improvement over previous CNN methods, and achieves better performance compared with the state-of-the-art transformer and RNN methods. In this work, our method is compared with the 9 most recent baselines on 10 scenes. The results show that our method consistently outperforms all the other methods while decreasing memory consumption by up to 80%.

High-Fidelity compressive spectral image reconstruction through a novel Non-Convex Non-Local Low-Rank tensor approximation model

Low-quality reconstruction is one main factor that greatly limits the development of snapshot compressive imaging (SCI) systems. The common algorithms usually unfold a 3D hyperspectral image (HSI) into a vector or matrix which inevitably corrupts the intrinsic structure of the HSI and insufficiently discovers its prior information. To fully exploit the underlying structures and internal correlations of the HSIs, this paper proposes a novel non-convex model based on Non-Local low-rank Tensor Approximation with Hyper-Laplacian priori (HL), named HL-NLTA, which effectively combines local and non-local similarity information from fused spectral and spatial perspectives. Specifically, a non-local low-rank tensor model based on the minimax concave plus (MCP) penalty and log sum (LS) penalty separately is developed which can simultaneously exploit two intrinsic properties of HSI, i.e., global correlation along the spectral domain (GCAS) and non-local self-similarity along the spatial domain (NSAS). Furthermore, the Hyper-Laplacian priori regularized through a non-convex ℓp (0 < p < 1) norm can preserve well the spectral and spatial structure. An optimization algorithm based on the alternating direction multiplier method (ADMM) framework is designed and accelerated by the Generalized shrinkage/thresholding (GST) algorithm and the fast Fourier transform (FFT) to solve the new nonconvex model. Extensive simulation results on public datasets and experimental results on our real CASSI system indicate that the proposed method enables high-fidelity HSI reconstructions in terms of simultaneous spatial structure detail recovery and spectral preservation. These results verify the effectiveness of the proposed model for snapshot compressive imaging (e.g., Coded aperture snapshot spectral imaging CASSI).

Multi-Scale CNN-Transformer Dual Network for Hyperspectral Compressive Snapshot Reconstruction

Coded aperture snapshot spectral imaging (CASSI) is a new imaging mode that captures the spectral characteristics of materials in real scenes. It encodes three-dimensional spatial–spectral data into two-dimensional snapshot measurements, and then recovers the original hyperspectral image (HSI) through a reconstruction algorithm. Hyperspectral data have multi-scale coupling correlations in both spatial and spectral dimensions. Designing a network architecture that effectively represents this coupling correlation is crucial for enhancing reconstruction quality. Although the convolutional neural network (CNN) can effectively represent local details, it cannot capture long-range correlation well. The Transformer excels at representing long-range correlation within the local window, but there are also issues of over-smoothing and loss of details. In order to cope with these problems, this paper proposes a dual-branch CNN-Transformer complementary module (DualCT). Its CNN branch mainly focuses on learning the spatial details of hyperspectral images, and the Transformer branch captures the global correlation between spectral bands. These two branches are linked through bidirectional interactions to promote the effective fusion of spatial–spectral features of the two branches. By utilizing characteristics of CASSI imaging, the residual mask attention is also designed and encapsulated in the DualCT module to refine the fused features. Furthermore, by using the DualCT module as a basic component, a multi-scale encoding and decoding model is designed to capture multi-scale spatial–spectral features of hyperspectral images and achieve end-to-end reconstruction. Experiments show that the proposed network can effectively improve reconstruction quality, and ablation experiments also verify the effectiveness of our network design.

Coded aperture snapshot hyperspectral light field tomography.

Multidimensional imaging has emerged as a powerful technology capable of simultaneously acquiring spatial, spectral, and depth information about a scene. However, existing approaches often rely on mechanical scanning or multi-modal sensing configurations, leading to prolonged acquisition times and increased system complexity. Coded aperture snapshot spectral imaging (CASSI) has introduced compressed sensing to recover three-dimensional (3D) spatial-spectral datacubes from single snapshot two-dimensional (2D) measurements. Despite its advantages, the reconstruction problem remains severely underdetermined due to the high compression ratio, resulting in limited spatial and spectral reconstruction quality. To overcome this challenge, we developed a novel two-stage cascaded compressed sensing scheme called coded aperture snapshot hyperspectral light field tomography (CASH-LIFT). By appropriately distributing the computation load to each stage, this method utilizes the compressibility of natural scenes in multiple domains, reducing the ill-posed nature of datacube recovery and achieving enhanced spatial resolution, suppressed aliasing artifacts, and improved spectral fidelity. Additionally, leveraging the snapshot 3D imaging capability of LIFT, our approach efficiently records a five-dimensional (5D) plenoptic function in a single snapshot.

Prior Image Guided Snapshot Compressive Spectral Imaging.

Spectral images with rich spatial and spectral information have wide usage, however, traditional spectral imaging techniques undeniably take a long time to capture scenes. We consider the computational imaging problem of the snapshot spectral spectrometer, i.e., the Coded Aperture Snapshot Spectral Imaging (CASSI) system. For the sake of a fast and generalized reconstruction algorithm, we propose a prior image guidance-based snapshot compressive imaging method. Typically, the prior image denotes the RGB measurement captured by the additional uncoded panchromatic camera of the dual-camera CASSI system. We argue that the RGB image as a prior image can provide valuable semantic information. More importantly, we design the Prior Image Semantic Similarity (PIDS) regularization term to enhance the reconstructed spectral image fidelity. In particular, the PIDS is formulated as the difference between the total variation of the prior image and the recovered spectral image. Then, we solve the PIDS regularized reconstruction problem by the Alternating Direction Method of Multipliers (ADMM) optimization algorithm. Comprehensive experiments on various datasets demonstrate the superior performance of our method.

Coded aperture snapshot spectral imaging fundus camera

Spectral imaging holds great promise for the non-invasive diagnosis of retinal diseases. However, to acquire a spectral datacube, conventional spectral cameras require extensive scanning, leading to a prolonged acquisition. Therefore, they are inapplicable to retinal imaging because of the rapid eye movement. To address this problem, we built a coded aperture snapshot spectral imaging fundus camera, which captures a large-sized spectral datacube in a single exposure. Moreover, to reconstruct a high-resolution image, we developed a robust deep unfolding algorithm using a state-of-the-art spectral transformer in the denoising network. We demonstrated the performance of the system through various experiments, including imaging standard targets, utilizing an eye phantom, and conducting in vivo imaging of the human retina.

Hybrid compressive spectral imaging with coded aperture optimization based on coherence minimization

Compressive spectral imaging (CSI) obtains spectral images through reconstruction from under-sampled low-dimensional projections with spatial and spectral coding. However, the simultaneous realization of high spatial–spectral resolution is challenged due to limitations of a single CSI architecture. In this paper, a hybrid CSI system combining the coded aperture snapshot spectral imaging (CASSI) and the spectrometer-based single-pixel spectral imaging (SPSI), with the corresponding coded aperture optimization for both constituent systems are proposed. The hybrid CSI system sustains the complementary advantages of the spatial resolution in CASSI and the spectral fidelity in SPSI, ensuring high reconstruction quality in both spatial and spectral dimensions with increased sampling efficiency. The derivable matrix operation of the hybrid CSI system is formulated; meanwhile, based on coherence minimization, the individual optimization on the coded apertures in the CASSI or the SPSI system, and the joint optimization on the coded apertures in the hybrid system are demonstrated. The reconstruction is further improved with the optimization, indicated by the quantitative image quality metrics and the point spectra. Compared to the conventional unoptimized CASSI, the optimized hybrid CSI system improves the PSNR for more than 7 dB. This work may introduce opportunities for multipath compressive imaging systems toward high-efficiency, high-resolution future.

Hyperspectral image reconstruction via patch attention driven network.

Coded aperture snapshot spectral imaging (CASSI) captures 3D hyperspectral images (HSIs) with 2D compressive measurements. The recovery of HSIs from these measurements is an ill-posed problem. This paper proposes a novel, to our knowledge, network architecture for this inverse problem, which consists of a multilevel residual network driven by patch-wise attention and a data pre-processing method. Specifically, we propose the patch attention module to adaptively generate heuristic clues by capturing uneven feature distribution and global correlations of different regions. By revisiting the data pre-processing stage, we present a complementary input method that effectively integrates the measurements and coded aperture. Extensive simulation experiments illustrate that the proposed network architecture outperforms state-of-the-art methods.

Dual-camera compressive hyperspectral imaging based on deep image prior and a guided filter.

Coded aperture snapshot spectral imaging (CASSI) aims to capture the high-dimensional (usually 3D) data cube using a 2D sensor in a single snapshot. Due to the ill-posed snapshot, the reconstruction results are not ideal. One feasible solution is to utilize additional information such as the panchromatic measurement in CASSI. In this paper, we propose a dual-camera hyperspectral reconstruction method based on the deep image prior (DIP) and a guided filter. In particular, the panchromatic measurements are used to estimate spatial detail, and spectral details are provided using CASSI measurements. These measurements are used as a priori learning by the self-supervised network. Using iteration combined with DIP, the hyperspectral reconstruction is continuously updated iteratively. Finally, the panchromatic measurement is used as the guidance image, and the reconstruction result is optimized by guide filtering. A large number of experimental results demonstrate that our method without training data can reconstruct spectral data with both high spectral accuracy and spatial resolution.

Dual camera snapshot high-resolution-hyperspectral imaging system with parallel joint optimization via physics-informed learning.

The hardware architecture of the coded aperture snapshot spectral imaging (CASSI) system is based on a coded mask design, resulting in a poor spatial resolution of the system. Therefore, we consider the use of a physical model of optical imaging and a jointly optimized mathematical model to design a self-supervised framework to solve the high-resolution-hyperspectral imaging problem. In this paper, we design a parallel joint optimization architecture based on a two-camera system. This framework combines the physical model of optical system and a joint optimization mathematical model, which takes full advantage of the spatial detail information provided by the color camera. The system has a strong online self-learning capability for high-resolution-hyperspectral image reconstruction, and gets rid of the dependence of supervised learning neural network methods on training data sets.

D$^\text{2}$UF: Deep Coded Aperture Design and Unrolling Algorithm for Compressive Spectral Image Fusion

Compressive spectral imaging (CSI) has attracted significant attention since it employs synthetic apertures to codify spatial and spectral information, sensing only 2D projections of the 3D spectral image. However, these optical architectures suffer from a trade-off between the spatial and spectral resolution of the reconstructed image due to technology limitations. To overcome this issue, compressive spectral image fusion (CSIF) employs the projected measurements of two CSI architectures with different resolutions to estimate a high-spatial high-spectral resolution. This work presents the fusion of the compressive measurements of a low-spatial high-spectral resolution coded aperture snapshot spectral imager (CASSI) architecture and a high-spatial low-spectral resolution multispectral color filter array (MCFA) system. Unlike previous CSIF works, this paper proposes joint optimization of the sensing architectures and a reconstruction network in an end-to-end (E2E) manner. The trainable optical parameters are the coded aperture (CA) in the CASSI and the colored coded aperture in the MCFA system, employing a sigmoid activation function and regularization function to encourage binary values on the trainable variables for an implementation purpose. Additionally, an unrolling-based network inspired by the alternating direction method of multipliers (ADMM) optimization is formulated to address the reconstruction step and the acquisition systems design jointly. Finally, a spatial-spectral inspired loss function is employed at the end of each unrolling layer to increase the convergence of the unrolling network. The proposed method outperforms previous CSIF methods, and experimental results validate the method with real measurements.

Compressive Spectral Imaging via Misalignment Induced Equivalent Grayscale Coded Aperture

Coded aperture snapshot spectral imager (CASSI) senses the spectral information of a 2D scene and captures a set of coded measurement data that can be used to reconstruct the 3D spatio-spectral datacube of the input scene by compressive sensing algorithms. The coded aperture (CA) in CASSI plays a crucial role in modulating the spatial information. The pixels in CA are typically square, switched binary on-off, and aligned with the pixels of focal plane array (FPA). Instead of this binary modulation, this paper explores a simple yet effective approach to enabling an equivalent grayscale modulation, which can increase the sensing degree of freedom in CASSI systems. In particular, we deliberately introduce misalignment between the CA pixels and the FPA pixels, such that the spatial modulation of one FPA pixel is determined by four adjacent CA pixels instead of one. Numerical experiments show that the proposed equivalent grayscale modulation induced by misalignment can significantly improve the CASSI reconstruction when compared with current methods, whether a random CA or an optimal blue noise CA is used. More importantly, it does not incur in any cost to the CASSI system.

Plug-and-Play Priors for Multi-Shot Compressive Hyperspectral Imaging.

Multi-shot coded aperture snapshot spectral imaging (CASSI) uses multiple measurement snapshots to encode the three-dimensional hyperspectral image (HSI). Increasing the number of snapshots will multiply the number of measurements, making CASSI system more appropriate for detailed spatial or spectrally rich scenes. However, the reconstruction algorithms still face the challenge of being ineffective or inflexible. In this paper, we propose a plug-and-play (PnP) method that uses denoiser as priors for multi-shot CASSI. Specifically, the proposed PnP method is based on the primal-dual algorithm with linesearch (PDAL), which makes it flexible and can be used for any multi-shot CASSI mechanisms. Furthermore, a new subspaced-based nonlocal reweighted low-rank (SNRL) denoiser is presented to utilize the global spectral correlation and nonlocal self-similarity priors of HSI. By integrating the SNRL denoiser into PnP-PDAL, we show the balloons (512 × 512 × 31) in CAVE dataset recovered from two snapshots compressive measurements with MPSNR above 50 dB. Experimental results demonstrate that our proposed method leads to significant improvements compared to the current state-of-the-art methods.