Snapshot Spectral Imaging System Research Articles

A MLP architecture fusing RGB and CASSI for computational spectral imaging

The coded Aperture Snapshot Spectral Imaging (CASSI) system offers significant advantages in dynamically acquiring hyper-spectral images compared to traditional measurement methods. However, it faces the following challenges: (1) Traditional masks rely on random patterns or analytical design, limiting CASSI’s performance improvement. (2) Existing CASSI reconstruction algorithms do not fully utilize RGB information. (3) High-quality reconstruction algorithms are often slow and limited to offline scene reconstruction. To address these issues, this paper proposes a new MLP architecture, Spectral-Spatial MLP (SSMLP), which replaces the transformer structure with a network using CASSI measurements and RGB as multimodal inputs. This maintains reconstruction quality while significantly improving reconstruction speed. Additionally, we constructed a teacher-student network (SSMLP with a teacher, SSMLP-WT) to transfer the knowledge learned from a large model to a smaller network, further enhancing the smaller network’s accuracy. Extensive experiments show that SSMLP matches the performance of transformer-based structures in spectral image reconstruction while improving inference speed by at least 50%. The reconstruction quality of SSMLP-WT is further improved by knowledge transfer without changing the network, and the teacher boosts the performance by 0.92 dB (44.73 dB vs. 43.81 dB).

Dual-channel aperture-splitting snapshot spectral imaging detection system

The contemporary spectral imaging detection systems commonly employed, such as pushbroom and stare systems, often necessitate motion-based imaging mechanisms such as scanning motors. This reliance on motion renders the imaging process vulnerable to platform vibrations, resulting in intricate post-image correction procedures and precluding dynamic target detection. Consequently, the advent of snapshot spectral imaging detection systems has ensued. Currently, there are significant challenges in the miniaturization design and rapid data acquisition aspects of snapshot spectral imaging systems. In this study, linear variable filters were employed as spectral components, and, through optical system simulation and design, the design of the nonspherical monolithic lens system and telescope system in the dual-channel aperture-splitting snapshot spectral imaging detection system were separately completed. The spectral range covered 400–1000 nm, with a spectral resolution of 27.3 nm, and each channel had a spatial sampling of 409×409 pixels. Additionally, based on the optical system design results, the system structure design and assembly were completed. Performance testing and preliminary spectral image fusion research were conducted on the assembled prototype. The instrument demonstrated excellent spectral imaging performance, thereby enhancing the efficiency of spectral imaging detection in snapshot spectral imaging.

Ultra-high-speed four-dimensional hyperspectral imaging

We propose, to the best of our knowledge, a novel deep learning–enabled four-dimensional spectral imaging system composed of a reflective coded aperture snapshot spectral imaging system and a panchromatic camera. The system simultaneously captures a compressively coded hyperspectral measurement and a panchromatic measurement. The hyperspectral data cube is recovered by the U-net-3D network. The depth information of the scene is then acquired by estimating a disparity map between the hyperspectral data cube and the panchromatic measurement through stereo matching. This disparity map is used to align the hyperspectral data cube and the panchromatic measurement. A designed fusion network is used to improve the spatial reconstruction of the hyperspectral data cube by fusing aligned panchromatic measurements. The hardware prototype of the proposed system demonstrates high-speed four-dimensional spectral imaging that allows for simultaneously acquiring depth and spectral images with an 8 nm spectral resolution between 450 and 700 nm, 2.5 mm depth accuracy, and a 1.83 s reconstruction time.

Meta-Attention Network Based Spectral Reconstruction with Snapshot Near-Infrared Metasurface.

Near-infrared (NIR) spectral information is important for detecting and analyzing material compositions. However, snapshot NIR spectral imaging systems still pose significant challenges owing to the lack of high-performance NIR filters and bulky setups, preventing effective encoding and integration with mobile devices. This study introduces a snapshot spectral imaging system that employs a compact NIR metasurface featuring 25 distinct C4 symmetry structures. Benefitting from the sufficient spectral variety and low correlation coefficient among these structures, center-wavelength accuracy of 0.05nm and full width at half maximum accuracy of 0.13nm are realized. The system maintains good performance within an incident angle of 1°. A novel meta-attention network prior iterative denoising reconstruction (MAN-IDR) algorithm is developed to achieve high-quality NIR spectral imaging. By leveraging the designed metasurface and MAN-IDR, the NIR spectral images, exhibiting precise textures, minimal artifacts in the spatial dimension, and little crosstalk between spectral channels, are reconstructed from a single grayscale recording image. The proposed NIR metasurface and MAN-IDR hold great promise for further integration with smartphones and drones, guaranteeing the adoption of NIR spectral imaging in real-world scenarios such as aerospace, health diagnostics, and machine vision.

CasFormer: Cascaded transformers for fusion-aware computational hyperspectral imaging

Computational hyperspectral imaging (CHI) is a cutting-edge technique, which plays a pivotal role in breaking through the quality bottleneck of hyperspectral images (HSI). Among the techniques employed in this domain, the coded aperture snapshot spectral imaging (CASSI) system holds widespread recognition. Nevertheless, the imaging capability of CASSI remains limited due to the hardware conditions and the fragility of outcomes associated with the ill-posed blind reconstruction process. To this end, we propose a novel cascaded transformer architecture, termed CasFormer, specifically crafted for fusion-aware CHI by means of a dual-imaging mechanism. CasFormer facilitates the effective enhancement of hyperspectral imaging quality by fusing RGB images, with a focus on spatial and spectral domains. As the name suggests, CasFormer is primarily composed of a series of cascade-attention blocks, enabling the fusion of high-spatial-resolution RGB images through spatial coherence alignment and the recovery of spectrally sequential information more compactly and accurately. Furthermore, CasFormer incorporates physical constraints through a decoupling-based loss function, ensuring spatial consistency and spectral fidelity in the fusion-aware CHI process. Extensive experiments conducted across multiple datasets demonstrate the superiority of CasFormer in achieving high-quality imaging results compared to SOTA CHI algorithms. Our code and benchmark datasets will be openly accessible at https://github.com/danfenghong/Information_Fusion_CasFormer.

Dual-Window Multiscale Transformer for Hyperspectral Snapshot Compressive Imaging

Coded aperture snapshot spectral imaging (CASSI) system is an effective manner for hyperspectral snapshot compressive imaging. The core issue of CASSI is to solve the inverse problem for the reconstruction of hyperspectral image (HSI). In recent years, Transformer-based methods achieve promising performance in HSI reconstruction. However, capturing both long-range dependencies and local information while ensuring reasonable computational costs remains a challenging problem. In this paper, we propose a Transformer-based HSI reconstruction method called dual-window multiscale Transformer (DWMT), which is a coarse-to-fine process, reconstructing the global properties of HSI with the long-range dependencies. In our method, we propose a novel U-Net architecture using a dual-branch encoder to refine pixel information and full-scale skip connections to fuse different features, enhancing the extraction of fine-grained features. Meanwhile, we design a novel self-attention mechanism called dual-window multiscale multi-head self-attention (DWM-MSA), which utilizes two different-sized windows to compute self-attention, which can capture the long-range dependencies in a local region at different scales to improve the reconstruction performance. We also propose a novel position embedding method for Transformer, named con-abs position embedding (CAPE), which effectively enhances positional information of the HSIs. Extensive experiments on both the simulated and the real data are conducted to demonstrate the superior performance, stability, and generalization ability of our DWMT. Code of this project is at https://github.com/chenx2000/DWMT.

Research on the Processing of Image and Spectral Information in an Infrared Polarization Snapshot Spectral Imaging System

In order to solve the problems of a low target recognition rate and poor real-time performance brought about by conventional infrared imaging spectral detection technology under complex background conditions or in the detection of targets of weak radiation or long distance, a kind of infrared polarization snapshot spectral imaging system (PSIFTIS) and a spectrum information processing method based on micro-optical devices are proposed in this paper, where the synchronous acquisition of polarization spectrum information is realized through the spatial modulation of phase with a rooftop-shaped multi-stage micro-mirror and the modulation of the polarization state of light with a micro-nanowire array. For the polarization interference image information obtained, the infrared polarization spectrum decoupling is realized by image segmentation, optical path difference matching, and image registration methods, the infrared polarization spectrum reconstruction is realized by Fourier transform spectral demodulation, and the infrared polarization image fusion is realized by decomposing and reconstructing the high- and low-frequency components of the polarization image based on the Haar wavelet transform. The maximum spectral peak wavenumber error of the four polarization channels of the polarization spectrum reconstruction is less than 2 cm−1, and the polarization angle error is within 1°. Ultimately, compared with the unprocessed polarization image unit, the peak signal-to-noise ratio is improved by 45.67%, the average gradient is improved by 8.03%, and the information entropy is improved by 56.98%.

High-resolution calibration and data reconstruction method for image mapping spectrometer

The image mapping spectrometer (IMS) is a kind of snapshot spectral imaging system that modulates the 3D spatial-spectral datacube (x,y,λ) to a 2D sensor for simultaneous measurement. Since the IMS utilizes an image mapper for mapping the spatial information, a calibration method is required to generate the lookup table between the measured raw image pixels and the desired datacube voxels. However, the current calibration methods are either time-consuming or low-accurate. We present a combined slit scan calibration and data processing approach that can balance the calibration efficiency and accuracy, by taking in account the irregularity of field of view (FOV) of the image mapper in the calibrating setup. Additionally, we analyze the spatial-spectral mixing phenomenon of the IMS datacube recovered from the calibrated lookup table. To reconstruct high-quality spectral datacube from the spatial-spectral mixed datacube, a spatial-spectral unmixing network (SSUNet) based on multi-head multi-attention mechanism is presented. Experimental results validated the effectiveness of both the calibration strategy and the datacube restoration network.

Prior Image Guided Snapshot Compressive Spectral Imaging.

Spectral images with rich spatial and spectral information have wide usage, however, traditional spectral imaging techniques undeniably take a long time to capture scenes. We consider the computational imaging problem of the snapshot spectral spectrometer, i.e., the Coded Aperture Snapshot Spectral Imaging (CASSI) system. For the sake of a fast and generalized reconstruction algorithm, we propose a prior image guidance-based snapshot compressive imaging method. Typically, the prior image denotes the RGB measurement captured by the additional uncoded panchromatic camera of the dual-camera CASSI system. We argue that the RGB image as a prior image can provide valuable semantic information. More importantly, we design the Prior Image Semantic Similarity (PIDS) regularization term to enhance the reconstructed spectral image fidelity. In particular, the PIDS is formulated as the difference between the total variation of the prior image and the recovered spectral image. Then, we solve the PIDS regularized reconstruction problem by the Alternating Direction Method of Multipliers (ADMM) optimization algorithm. Comprehensive experiments on various datasets demonstrate the superior performance of our method.

Multi-Parameter Model-Based Polarimetric Calibration for Dual-Coded Spectral Polarization Imaging System

A polarization analysis method based on a multi-parameter model is proposed to address the polarization effect analysis and calibration requirements of a dual-coded snapshot spectral polarization imaging system. A full-link polarization effect model for a spectral polarization imaging system is established that includes a digital micromirror array (DMD), prism grating prism (PGP), micro-polarizer array detector (MPA), and multi-film. The influence of parameters such as the refractive index, incident angle, grating refractive index, constant, prism refractive index, vertex angle, multi-layer film complex refractive index, and film thickness on the optical transmittance of the system are analyzed. Using a dynamic data exchange mechanism to perform full-link, full-FOV, and full-pupil ray tracing on the optical system, the polarization effect distribution of the system under different degrees of polarization (DOP) and wavelengths is obtained. A calibration experiment for the controllable incident wavelength and DOP using narrowband filters and glass stacks is established. The experimental results show that in the 420 nm, 532 nm, and 635 nm wavelength bands, the MSEs of the calibrated values are 1.3924 × 10−4, 1.6852 × 10−4, and 1.6735 × 10−4, respectively. It is proven that the calibration method based on a multi-parameter model is feasible. Finally, the spectral polarization image at 532 nm is calibrated. The contrast ratio of metallic aluminum is calibrated from 7.13 to 15.33. This study provides a theoretical basis for the analysis and calibration of polarization effects in a dual-coded snapshot spectral polarization imaging system.

Polarization Snapshot Imaging Spectrometer for Infrared Range

Infrared imaging spectrometers detect and identify targets by collecting spectral and image information. However, when detecting small temperature differences and dynamic targets, the accuracy of infrared detection is reduced, the traditional scanning structure detection time is longer, the real-time performance is poor and it is easy to introduce motion artifacts. This paper proposes an infrared polarization snapshot spectral imaging system (PSIFTIS) based on a polarizer array, a lens array and a roof-shaped stepped micromirror. Polarized light can solve the problem of small-temperature-difference target recognition by characterizing the surface properties of materials. Lens arrays utilize multi-aperture imaging to achieve snapshot detection of targets. The system can obtain 4D data information, including polarization, in a single measurement cycle. This study completed the overall optical design of a PSIFTIS and an optical simulation experiment using it. Finally, a system prototype was built in the laboratory and a polarization spectrum detection experiment was carried out. The experimental results show that the PSIFTIS could accurately obtain the polarization spectrum information for the target, the spectral resolution reached 7.8 cm−1 and the Stokes measurement error was less than 5%.

Dual camera snapshot high-resolution-hyperspectral imaging system with parallel joint optimization via physics-informed learning.

The hardware architecture of the coded aperture snapshot spectral imaging (CASSI) system is based on a coded mask design, resulting in a poor spatial resolution of the system. Therefore, we consider the use of a physical model of optical imaging and a jointly optimized mathematical model to design a self-supervised framework to solve the high-resolution-hyperspectral imaging problem. In this paper, we design a parallel joint optimization architecture based on a two-camera system. This framework combines the physical model of optical system and a joint optimization mathematical model, which takes full advantage of the spatial detail information provided by the color camera. The system has a strong online self-learning capability for high-resolution-hyperspectral image reconstruction, and gets rid of the dependence of supervised learning neural network methods on training data sets.

Modular snapshot multispectral-panchromatic imager (MSPI) with customized filter arrays.

As one of the simplest methods to construct snapshot spectral imagers, multispectral filter array (MSFA) has been applied to commercial miniatured spectral imagers. While most of them have fixed configurations of spectral channels, lacking flexibility and replaceability. Moreover, conventional MSFA only comprises filtering channels but lacks the panchromatic channel which is essential in detecting dim and indistinct objects. Here, we propose a modular assembly method for snapshot imager which can simultaneously acquire the object's multispectral and panchromatic information based on a customized filter array. The multispectral-panchromatic filter array is batch fabricated and integrated with the imaging senor through a modular mode. Five-band spectral images and a broadband intensity image can be efficiently acquired in a single snapshot photographing. The efficacy and accuracy of the imager are experimentally verified in imaging and spectral measurements. Owing to the modular architecture, our proposed assembly method owns the advantages of compactness, simple assembling, rapid replacement, and customized designing, which overcomes the expensiveness and complexity of scientific-level snapshot spectral imaging systems.

A Refinement Boosted and Attention Guided Deep FISTA Reconstruction Framework for Compressive Spectral Imaging

Hyperspectral images (HSIs) contain rich spatial and spectral information. A double dispersers coded aperture snapshot spectral imaging (DD-CASSI) system takes advantage of compressive sensing (CS) theory to map 3D HSI data into a single 2D measurement. One of key components of DD-CASSI is to reconstruct high quality hyperspectral image from measurement. Traditional model-based methods use mathematical optimization to reconstruct hyperspectral images according to prior knowledge. Current deep learning based methods achieve pleasant results. But fully learned deep learning methods lack interpretability, and model-based deep learning methods cannot achieve pleasant performance. In this paper, we propose a novel HSI reconstruction framework named Refinement Boosted and Attention Guided Tensor FISTA(Fast Iterative Shrinkage-Thresholding Algorithm)-Net (ReAttFISTA-Net), which combines model-based deep learning and fully learned deep learning reconstruction strategies. In this framework, we introduces Attention Guided Fusion Mechanism which enhances spatial-spectral information, refinement sub-network and auxiliary loss terms to improve the reconstruction performance. Extensive experimental results show that the proposed reconstruction algorithm outperforms the state-of-the-art algorithms on both simulation and real-world datasets.

Fourier coded aperture transform hyperspectral imaging system

We consider using Fourier coded aperture transform to solve the acquisition and reconstruction problem of three-dimensional (3-D) hyperspectral imaging. In recent years, Coded aperture snapshot spectral imaging (CASSI) system has received increasing attention. However, reconstruction errors are unavoidable due to its ill-posed problems and the number of dispersion channels is limited. The Hadamard transform spectral (HTS) imaging method has no limitation on the number of dispersion channels, but it usually requires a long data acquisition time. By exploring the process of coded spectral imaging, in this paper, we propose Fourier coded aperture transform spectral imaging (FCTS). We compared the performance of HTS and FCTS by theoretical analysis and experiments. The results show that FCTS has better concentrated energy than HTS on natural hyperspectral data. Compared with the Hadamard transform method, the Fourier basis mode has better physical correlation with the natural image, and the reconstruction quality of FCTS is better than that of HTS in the case of under-sampling.

Deep learning enabled reflective coded aperture snapshot spectral imaging.

Coded aperture snapshot spectral imaging (CASSI) can acquire rich spatial and spectral information at ultra-high speed, which shows extensive application prospects. CASSI innovatively employed the idea of compressive sensing to capture the spatial-spectral data cube using a monochromatic detector and used reconstruction algorithms to recover the desired spatial-spectral information. Based on the optical design, CASSI currently has two different implementations: single-disperser (SD) CASSI and dual-disperser (DD) CASSI. However, SD-CASSI has poor spatial resolution naturally while DD-CASSI increases size and cost because of the extra prism. In this work, we propose a deep learning-enabled reflective coded aperture snapshot spectral imaging (R-CASSI) system, which uses a mask and a beam splitter to receive the reflected light by utilizing the reflection of the mask. The optical path design of R-CASSI makes the optical system compact, using only one prism as two dispersers. Furthermore, an encoder-decoder structure with 3D convolution kernels is built for the reconstruction, dubbed U-net-3D. The designed U-net-3D network achieves both spatial and spectral consistency, leading to state-of-the-art reconstruction results. The real data is released and can serve as a benchmark dataset to test new reconstruction algorithms.

Fast non-iterative blind restoration of hyperspectral images with spectrally-varying PSFs

Hyperspectral imaging is the basis for many data analysis techniques. As more application scenarios demand the portability and snapshot imaging capabilities of hyperspectral imaging systems, snapshot spectral imaging system based on diffracted rotation has gained increasing attention and development. However, in this system, the traditional iterative optimization-based unrolled network architectures require the assistance of point spread functions (PSFs) in the reconstruction process and are costly in terms of time and computational resources. Aiming to improve the quality of the reconstruction results, reduce the consumption of computational resources and time, and broaden the spectral range, we firstly construct a visible to near-infrared hyperspectral (VNH) dataset. Then we propose a convolutional neural network (CNN) based blind restoration network for hyperspectral images. In this work, we take Unet as the initial framework and propose using the spectral upsampling block, the weight adaptive residual (WAR) block, and the hybrid loss function to enhance the network performance. Experiments show that our proposed method can effectively reconstruct hyperspectral images from rotational diffraction blurred images, consuming less computational resources while retaining a competitive spatial resolution and spectral precision.

Optical Design of Novel Snapshot Spectral Imaging System Based on Dyson Structure

Optical Design of Novel Snapshot Spectral Imaging System Based on Dyson Structure

Optical design of an Offner coded aperture snapshot spectral imaging system based on dual-DMDs in the mid-wave infrared band.

With the advantages of flexible encoding and high frame rate, the digital micromirror devices (DMD) have been used as a binary encoding mask in the coded aperture snapshot spectral imaging (CASSI) systems. But the use of DMD will cause the image plane to tilt at a specific angle, so it is almost impossible to realize the strict matching between the optical system of CASSI and the cold stop of the infrared focal plane array in the mid-wave infrared band. In this paper, a CASSI system with two DMDs based on the Offner spectrometer is proposed to solve the above problem. The concept and working principle are described in detail. Under the premise of the matching optical parameters, the telescopic system, Offner spectroscopic system and microscopic system are designed independently. Then the integrated optimization design method is adopted, and the aberration of the microscopic system is used to offset the astigmatic aberration of the Offner spectroscopic system, and the imaging quality of the system is improved. The results of performance measurements confirm that the system has desirable spatial resolution and spectral response functions. Thus, the concept and optical design of the proposed system are verified to be effective and valuable.

Shift-variant color-coded diffractive spectral imaging system

State-of-the-art snapshot spectral imaging (SI) systems introduce color-coded apertures (CCAs) into their setups to obtain a flexible spatial-spectral modulation, allowing spectral information to be reconstructed from a set of coded measurements. Besides the CCA, other optical elements, such as lenses, prisms, or beam splitters, are usually employed, making systems large and impractical. Recently, diffractive optical elements (DOEs) have partially replaced refractive lenses to drastically reduce the size of the SI devices. The sensing model of these systems is represented as a projection modeled by a spatially shift-invariant convolution between the unknown scene and a point spread function (PSF) at each spectral band. However, the height maps of the DOE are the only free parameters that offer changes in the spectral modulation, which causes the ill-posedness of the reconstruction to increase significantly. To overcome this challenge, our work explores the advantages of the spectral modulation of an optical setup composed of a DOE and a CCA. Specifically, the light is diffracted by the DOE and then filtered by the CCA, located close to the sensor. A shift-variant property of the proposed system is clearly evidenced, resulting in a different PSF for each pixel, where a symmetric structure constraint is imposed on the CCA to reduce the high number of resulting PSFs. Additionally, we jointly design the DOE and the CCA parameters with a fully differentiable image formation model using an end-to-end approach to minimize the deviation between the true and reconstructed image over a large set of images. Simulation shows that the proposed system improves the spectral reconstruction quality in up to 4 dB compared with current state-of-the-art systems. Finally, experimental results with a fabricated prototype in indoor and outdoor scenes validate the proposed system, where it can recover up to 49 high-fidelity spectral bands in the 420–660 nm.