Coded Aperture Snapshot Spectral Imaging Research Articles

Plug-and-Play Priors for Multi-Shot Compressive Hyperspectral Imaging.

Multi-shot coded aperture snapshot spectral imaging (CASSI) uses multiple measurement snapshots to encode the three-dimensional hyperspectral image (HSI). Increasing the number of snapshots will multiply the number of measurements, making CASSI system more appropriate for detailed spatial or spectrally rich scenes. However, the reconstruction algorithms still face the challenge of being ineffective or inflexible. In this paper, we propose a plug-and-play (PnP) method that uses denoiser as priors for multi-shot CASSI. Specifically, the proposed PnP method is based on the primal-dual algorithm with linesearch (PDAL), which makes it flexible and can be used for any multi-shot CASSI mechanisms. Furthermore, a new subspaced-based nonlocal reweighted low-rank (SNRL) denoiser is presented to utilize the global spectral correlation and nonlocal self-similarity priors of HSI. By integrating the SNRL denoiser into PnP-PDAL, we show the balloons (512 × 512 × 31) in CAVE dataset recovered from two snapshots compressive measurements with MPSNR above 50 dB. Experimental results demonstrate that our proposed method leads to significant improvements compared to the current state-of-the-art methods.

Sub-Pixel Dispersion Model for Coded Aperture Snapshot Spectral Imaging

Sub-Pixel Dispersion Model for Coded Aperture Snapshot Spectral Imaging

Fourier coded aperture transform hyperspectral imaging system

We consider using Fourier coded aperture transform to solve the acquisition and reconstruction problem of three-dimensional (3-D) hyperspectral imaging. In recent years, Coded aperture snapshot spectral imaging (CASSI) system has received increasing attention. However, reconstruction errors are unavoidable due to its ill-posed problems and the number of dispersion channels is limited. The Hadamard transform spectral (HTS) imaging method has no limitation on the number of dispersion channels, but it usually requires a long data acquisition time. By exploring the process of coded spectral imaging, in this paper, we propose Fourier coded aperture transform spectral imaging (FCTS). We compared the performance of HTS and FCTS by theoretical analysis and experiments. The results show that FCTS has better concentrated energy than HTS on natural hyperspectral data. Compared with the Hadamard transform method, the Fourier basis mode has better physical correlation with the natural image, and the reconstruction quality of FCTS is better than that of HTS in the case of under-sampling.

Deep learning enabled reflective coded aperture snapshot spectral imaging.

Coded aperture snapshot spectral imaging (CASSI) can acquire rich spatial and spectral information at ultra-high speed, which shows extensive application prospects. CASSI innovatively employed the idea of compressive sensing to capture the spatial-spectral data cube using a monochromatic detector and used reconstruction algorithms to recover the desired spatial-spectral information. Based on the optical design, CASSI currently has two different implementations: single-disperser (SD) CASSI and dual-disperser (DD) CASSI. However, SD-CASSI has poor spatial resolution naturally while DD-CASSI increases size and cost because of the extra prism. In this work, we propose a deep learning-enabled reflective coded aperture snapshot spectral imaging (R-CASSI) system, which uses a mask and a beam splitter to receive the reflected light by utilizing the reflection of the mask. The optical path design of R-CASSI makes the optical system compact, using only one prism as two dispersers. Furthermore, an encoder-decoder structure with 3D convolution kernels is built for the reconstruction, dubbed U-net-3D. The designed U-net-3D network achieves both spatial and spectral consistency, leading to state-of-the-art reconstruction results. The real data is released and can serve as a benchmark dataset to test new reconstruction algorithms.

Dictionary Learning- and Total Variation-Based High-Light-Efficiency Snapshot Multi-Aperture Spectral Imaging

Conventional multispectral imaging systems based on bandpass filters struggle to record multispectral videos with high spatial resolutions because of their limited light efficiencies. This paper proposes a multi-aperture multispectral imaging system based on notch filters that overcomes this limitation by allowing light from most of the spectrum to pass through. Based on this imaging principle, a prototype multi-aperture multispectral imaging system comprising notch filters was built and demonstrated. Further, a dictionary learning- and total variation-based spectral super-resolution algorithm was developed to reconstruct spectral images. The simulation results obtained using public multispectral datasets showed that, compared to the dictionary learning-based spectral super-resolution algorithm, the proposed algorithm reconstructed the spectral information with a higher accuracy and removed noise, and the verification experiments confirmed the performance efficiency of the prototype system. The experimental results showed that the proposed imaging system can capture images with high spatial and spectral resolutions under low illumination conditions. The proposed algorithm improved the spectral resolution of the acquired data from 9 to 31 bands, and the average peak signal-to-noise ratio remained above 43 dB, which is 13 dB higher than those of the state-of-the-art coded aperture snapshot spectral imaging methods. Simultaneously, the frame rate of the imaging system was up to 5000 frames/s under natural daylight.

Single-shot compressed optical field topography

Femtosecond lasers are powerful in studying matter’s ultrafast dynamics within femtosecond to attosecond time scales. Drawing a three-dimensional (3D) topological map of the optical field of a femtosecond laser pulse including its spatiotemporal amplitude and phase distributions, allows one to predict and understand the underlying physics of light interaction with matter, whose spatially resolved transient dielectric function experiences ultrafast evolution. However, such a task is technically challenging for two reasons: first, one has to capture in single-shot and squeeze the 3D information of an optical field profile into a two-dimensional (2D) detector; second, typical detectors are only sensitive to intensity or amplitude information rather than phase. Here we have demonstrated compressed optical field topography (COFT) drawing a 3D map for an ultrafast optical field in single-shot, by combining the coded aperture snapshot spectral imaging (CASSI) technique with a global 3D phase retrieval procedure. COFT can, in single-shot, fully characterize the spatiotemporal coupling of a femtosecond laser pulse, and live stream the light-speed propagation of an air plasma ionization front, unveiling its potential applications in ultrafast sciences.

Adaptive local sparse representation for compressive hyperspectral imaging

Coded aperture snapshot spectral imaging (CASSI) is an effective way for hyperspectral imaging. In CASSI, the key issue is to accurately and efficiently reconstruct the 3D hyperspectral image from its corresponding coded 2D image. Due to the ill-posed nature, reconstruction errors are inevitable, a feasible solution is to add an RGB camera for complementary sampling to reduce the reconstruction error. In this paper, we investigate the structural changes of local image patches in different bands and their correlation with RGB observation, propose a reconstruction method for dual-camera CASSI system. Specifically, we learn an adaptive dictionary with RGB observation, then use RGB observation to guide the selection of the adaptive dictionary for each local image patch of the reconstruction target, and finally reconstruct the original hyperspectral image through an iterative numerical algorithm. This method fuses the spatial and spectral information obtained from RGB observations into the reconstruction process, experimental results show that the proposed method can greatly improve the reconstruction quality, especially the reconstruction of the details, and reduce more time compared with past dictionary-based reconstruction methods.

Residual image recovery method based on the dual-camera design of a compressive hyperspectral imaging system.

Compressive hyperspectral imaging technology can quickly detect the encoded two-dimensional measurements and reconstruct the three-dimensional hyperspectral images offline, which is of great significance for object detection and analysis. To provide more information for reconstruction and improve the reconstruction quality, some of the latest compressive hyperspectral imaging systems adopt a dual-camera design. To utilize the information from additional camera more efficiently, this paper proposes a residual image recovery method. The proposed method takes advantage of the structural similarity between the image captured by the additional camera and the hyperspectral image, combining the measurements from the additional camera and coded aperture snapshot spectral imaging (CASSI) sensor to construct an estimated hyperspectral image. Then, the component of the estimated hyperspectral image is subtracted from the measurement of the CASSI sensor to obtain the residual data. The residual data is used to reconstruct the residual hyperspectral image. Finally, the reconstructed hyperspectral image is the sum of the estimated and residual image. Compared with some state-of-the-art algorithms based on such systems, the proposed method can significantly improve the reconstruction quality of hyperspectral image.

Differential coded aperture single-snapshot spectral imaging.

We propose a novel, to the best of our knowledge, concept of the differential coded aperture snapshot spectral imaging (D-CASSI) technique exploiting the benefits of using a {-1,+1} random mask, which is demonstrated by a broadband single-snapshot hyperspectral camera using compressed sensing. To double the information, we encode the image with two complementary random masks, which proved to be superior to two independent patterns. We utilize dispersed and non-dispersed encoded images captured in parallel on a single detector. We explored several different approaches to processing the measured data, which demonstrates significant improvement in retrieving complex hyperspectral scenes. The experiments were completed by simulations in order to quantify the reconstruction fidelity. The concept of differential CASSI could be easily implemented also by multi-snapshot CASSI without any need for optical system modification.

Off-axis aberration correction for a reflective coded aperture snapshot spectral imager.

The reflective optical configuration of the coded aperture snapshot spectral imager (CASSI) based on a digital micromirror device (DMD) features both high quality and high flexibility of intensity modulation, but the image quality is always subject to off-axis aberration. This letter analyzes the formation of the off-axis aberration, and then proposes an optical-axis-shift (OAS) configuration to compensate for this adverse effect in the DMD-based reflective CASSI system. Experimental results demonstrate that the proposed design can effectively eliminate the off-axis aberration. The average peak signal-to-noise ratio of the reconstructed spectral images is improved by approximately 2.8 dB. In addition, other methods to alleviate the off-axis aberration are discussed, and the superiority of the proposed design is signified. The proposed OAS design is easy to implement with commercially available components, and provides an initial configuration of an optical design to implement a high-performance CASSI instrument.

Single-shot compressive hyperspectral imaging with dispersed and undispersed light using a generally available grating.

Commercially available hyperspectral cameras are useful for remote sensing, but in most cases snapshot imaging is difficult due to the need for scanning. The coded aperture snapshot spectral imager (CASSI) has been proposed to simultaneously acquire a target scene's spatial and spectral dimensional data, employing a refractive prism as a disperser. This paper proposes a CASSI-based technique using a generally available diffraction grating of a Ronchi ruling and blazed grating and its improvement using the undispersed zeroth-order light. The feasibility and performance of the proposed technique are experimentally validated, and the grating parameters are identified.

Total variation regularized nonlocal low-rank tensor train for spectral compressive imaging

In this paper, we consider the reconstruction problem of hyperspectral images (HSIs) based on a new type of imaging system termed as coded aperture snapshot spectral imaging (CASSI). CASSI compresses multiple spectral bands of HSI into a single snapshot measurement which enjoys low cost, low bandwidth, and high-speed sensing rate. Despite the promising applications of CASSI systems, the bottleneck lies in the reconstruction algorithms. The existing CASSI reconstruction methods could not fully exploit the underlying structures of HSIs, and thereby demonstrate unsatisfactory reconstruction quality. To remedy this issue, we propose in this paper a novel tensor-based method by modeling two intrinsic HSI priors, namely nonlocal low-rank tensor train (NLTT) and 3-D weighted total variation (3DWTV). Taking into consideration that the snapshot measurement and targeted HSI should share similar spatial structures, we use the snapshot measurement to help group nonlocal spatially similar cubes within the unknown HSI to form 4-D tensors. And the NLTT prior employed on these 4-D tensors can effectively learn the correlations among the spatial, spectral and nonlocal modes thanks to the well-balanced matricization scheme of TT rank. Meanwhile, we employ the 3DWTV prior to characterize the local smoothness of the spatial and spectral modes. The resulting model can be solved efficiently by using the alternative direction method of multipliers (ADMM). Extensive experiments on both simulated and real CASSI datasets validate the improved reconstruction performance of our method over several other competing ones.

High-Accuracy Image Formation Model for Coded Aperture Snapshot Spectral Imaging

Coded aperture snapshot spectral imaging (CASSI) is based on the binary modulation of the spatial-spectral scene, which allows for hyperspectral image reconstruction from 2D compressive measurement. However, the actual optical modulation does not match the current image formation model due to the extra optical phenomena, such as diffraction, distortion, optical misalignment, and dispersion, inside the system. It is a long-lasting problem that the gap between the simplified image formation model and the actual optical modulation degrades the reconstruction quality. In this paper, we propose a high-accuracy image formation model to reduce this gap in CASSI. Specifically, we first reformulate the spectral modulation as channel-wise convolution, in which the convolution kernel represents the point-spread-function (PSF) of each spectral channel. Then, according to our key observation that the calibration images are the blurred versions of the coded aperture, we propose to estimate the PSF by exploring the relationship between these blurred and non-blurred pairs. In addition, we also provide a theoretical analysis of the PSF’s influences on the reconstruction quality, which can serve as a guide for CASSI system implementation. Our simulations and real system experiments demonstrate the effectiveness of the proposed model.

Optical design of a short-wave infrared coded aperture snapshot spectral imager based on Offner–Wynne imaging spectrometer

A calculation model to obtain the optimal combination of Offner–Wynne imaging spectrometer parameters to balance the RMS spot radius of the chief ray over the entire field is established. On the basis of the calculation model, an Offner–Wynne imaging spectrometer is designed, and then it is applied in a short-wave infrared coded aperture snapshot spectral imager (CASSI). The optical system of the short-wave infrared CASSI is designed by a combination method of independent design and integrated optimization. The spectral smile and spectral keystone of the optical system which operates in the 900–1700 nm band are respectively less than half a pixel. The focal length of the optical system is 1200 mm, the total optical system length is 775 mm, and its average spectral resolution is 16 nm. The optical system offers the advantages of excellent imaging quality, compact optical structure, high optical transmittance, reduced spectral smile and spectral keystone.

Learning Deep Subspace Projection Prior for Dual-Camera Compressive Hyperspectral Imaging

Coded aperture snapshot spectral imaging (CASSI) captures the 3-D hyperspectral images (HSI) in the form of 2-D coded images. The dual-camera compressive hyperspectral imaging (DCCHI) can effectively improve the reconstruction quality by adding a parallel complementary panchromatic camera. Several regularization-based methods have been proposed for dual-camera reconstruction. However, the handcrafted priors of these methods are limited in representing the complex intrinsic structure of HSI. In this letter, we propose to learn deep subspace projection prior for dual-camera compressive reconstruction. We first design a deep subspace projection prior regularized dual-camera compressive reconstruction model and minimize it with alternative optimization. Then, we unfold the optimization process into a network. Specifically, the deep subspace projection prior learning leads to features with low-rank characteristics, which could efficiently exploit the spectral correlation of HSI. The dual-camera compressive reconstruction network is learned in an end-to-end manner. Extensive experiments substantiate the performance and efficiency of other start-of-the-art algorithms.

Optical design of an Offner coded aperture snapshot spectral imaging system based on dual-DMDs in the mid-wave infrared band.

With the advantages of flexible encoding and high frame rate, the digital micromirror devices (DMD) have been used as a binary encoding mask in the coded aperture snapshot spectral imaging (CASSI) systems. But the use of DMD will cause the image plane to tilt at a specific angle, so it is almost impossible to realize the strict matching between the optical system of CASSI and the cold stop of the infrared focal plane array in the mid-wave infrared band. In this paper, a CASSI system with two DMDs based on the Offner spectrometer is proposed to solve the above problem. The concept and working principle are described in detail. Under the premise of the matching optical parameters, the telescopic system, Offner spectroscopic system and microscopic system are designed independently. Then the integrated optimization design method is adopted, and the aberration of the microscopic system is used to offset the astigmatic aberration of the Offner spectroscopic system, and the imaging quality of the system is improved. The results of performance measurements confirm that the system has desirable spatial resolution and spectral response functions. Thus, the concept and optical design of the proposed system are verified to be effective and valuable.

Optical Design of a Shortwave Infrared Dual Camera CASSI Based on Improved Offner-Wynne Imaging Spectrometer

Based on the optical system characteristics of coded aperture snapshot spectral imager (CASSI), an optimized optical system of shortwave infrared dual camera CASSI was designed based on improved Offner-Wynne imaging spectrometer. The operating wavelength of the optical system ranges from 900nm to 1700nm, and the focal length is 1200mm. It consists of two parts: the two dimensional imaging system and the multispectral CASSI imaging system. The key technical parameters of the two parts are the same and there is no visual axis difference. Therefore, the optimized optical system can effectively improve real-time performance, optical transmittance and compactness of the dual camera shortwave infrared CASSI, which is conducive to the application in optical measurement scenes in the shooting range.

Synthetic coded aperture snapshot spectral imaging based on coprime sub-aperture sampling.

The imaging of a large area scene is difficult to achieve for a single camera. Alternatively, a virtual large aperture can be synthesized by sub-aperture cooperation. We propose a solution through the combination of a coded aperture snapshot spectral imager and coprime array. This method reduces the amount of data by using a smaller sub-aperture for sampling. The position of the sub-aperture is determined according to the element distribution of the coprime array, so that the data obtained from each sampling are about the target object and its adjacent area, which can ensure high reconstruction accuracy. The feasibility of coprime sub-aperture sampling is verified by numerical simulation.

Compressive hyperspectral image classification using a 3D coded convolutional neural network.

Hyperspectral image classification (HIC) is an active research topic in remote sensing. Hyperspectral images typically generate large data cubes posing big challenges in data acquisition, storage, transmission and processing. To overcome these limitations, this paper develops a novel deep learning HIC approach based on compressive measurements of coded-aperture snapshot spectral imagers (CASSI), without reconstructing the complete hyperspectral data cube. A new kind of deep learning strategy, namely 3D coded convolutional neural network (3D-CCNN), is proposed to efficiently solve for the classification problem, where the hardware-based coded aperture is regarded as a pixel-wise connected network layer. An end-to-end training method is developed to jointly optimize the network parameters and the coded apertures with periodic structures. The accuracy of classification is effectively improved by exploiting the synergy between the deep learning network and coded apertures. The superiority of the proposed method is assessed over the state-of-the-art HIC methods on several hyperspectral datasets.

Compressive spectral image reconstruction using deep prior and low-rank tensor representation.

Compressive spectral imaging (CSI) has emerged as an alternative spectral image acquisition technology, which reduces the number of measurements at the cost of requiring a recovery process. In general, the reconstruction methods are based on handcrafted priors used as regularizers in optimization algorithms or recent deep neural networks employed as an image generator to learn a non-linear mapping from the low-dimensional compressed measurements to the image space. However, these deep learning methods need many spectral images to obtain good performance. In this work, a deep recovery framework for CSI without training data is presented. The proposed method is based on the fact that the structure of some deep neural networks and an appropriated low-dimensional structure are sufficient to impose a structure of the underlying spectral image from CSI. We analyzed the low-dimensional structure via the Tucker representation, modeled in the first net layer. The proposed scheme is obtained by minimizing the ${\ell _2}$-norm distance between the compressive measurements and the predicted measurements, and the desired recovered spectral image is formed just before the forward operator. Simulated and experimental results verify the effectiveness of the proposed method for the coded aperture snapshot spectral imaging.