Compressive Hyperspectral Imaging Research Articles

Advancements in Remote Compressive Hyperspectral Imaging: Adaptive Sampling with Low-Rank Tensor Image Reconstruction

We advanced the practical development of compressive hyperspectral cameras for remote sensing scenarios with a design that simultaneously compresses and captures high-quality spectral information of a scene via configurable measurements. We built a prototype imaging system that is compatible with light-modulation devices that encode the incoming spectrum. The sensing approach enables a substantial reduction in the volume of data collected and transmitted, facilitating large-scale remote hyperspectral imaging. A main advantage of our sensing design is that it allows for adaptive sampling. When prior information of a survey region is available or gained, the modulation patterns can be re-programmed to efficiently sample and detect desired endmembers. Given target spectral signatures, we propose an optimization scheme that guides the encoding process. The approach severely reduces the number of required sampling patterns, with the ability to achieve image segmentation and correct distortions. Additionally, to decode the modulated data, we considered a novel reconstruction algorithm suited for large-scale images. The computational methodology leverages the multidimensional structure and redundant representation of hyperspectral images via the canonical polyadic decomposition of multiway arrays. Under realistic remote sensing scenarios, we demonstrated the efficiency of our approach with several data sets collected by our prototype camera and reconstructed by our low-rank tensor decoder.

Enhanced deep unrolling networks for snapshot compressive hyperspectral imaging

Snapshot compressive hyperspectral imaging necessitates the reconstruction of a complete hyperspectral image from its compressive snapshot measurement, presenting a challenging inverse problem. This paper proposes an enhanced deep unrolling neural network, called EDUNet, to tackle this problem. The EDUNet is constructed via the deep unrolling of a proximal gradient descent algorithm and introduces two innovative modules for gradient-driven update and proximal mapping reflectivity. The gradient-driven update module leverages a memory-assistant descent approach inspired by momentum-based acceleration techniques, for enhancing the unrolled reconstruction process and improving convergence. The proximal mapping is modeled by a sub-network with a cross-stage spectral self-attention, which effectively exploits the inherent self-similarities present in hyperspectral images along the spectral axis. It also enhances feature flow throughout the network, contributing to reconstruction performance gain. Furthermore, we introduce a spectral geometry consistency loss, encouraging EDUNet to prioritize the geometric layouts of spectral curves, leading to a more precise capture of spectral information in hyperspectral images. Experiments are conducted using three benchmark datasets including KAIST, ICVL, and Harvard, along with some real data, comprising a total of 73 samples. The experimental results demonstrate that EDUNet outperforms 15 competing models across four metrics including PSNR, SSIM, SAM, and ERGAS.

Snapshot volumetric imaging using rainbow-sheet illumination

Traditional lens-based three-dimensional imaging methods struggle with speed, spatial resolution, field of view, and depth of field (DOF). Here, we propose a volumetric imaging method that combines rainbow-sheet illumination, chromatic-aberration-induced DOF extension, and compressive hyperspectral imaging to optically section transparent objects over 200 depth slices in a single snapshot. A proof-of-concept mesoscopic system with a lateral resolution of 12.7 line pairs per millimeter and a depth resolution of roughly 140 μm in a volume of 10 × 10 × 10 mm3 is constructed. The practicality of the suggested method is demonstrated by dynamic volumetric imaging of a transparent jellyfish at a rate of 15 volumes per second.

Exploiting sparsity of hyperspectral image: A novel approach for compressive hyperspectral image reconstruction using deep learning

Compressive hyperspectral imaging is an emerging technology that captures compressed two-dimensional (2D) measurements and subsequently reconstructs three-dimensional (3D) hyperspectral images, capitalizing on the sparsity prior. Numerous end-to-end networks have been proposed, yielding notable improvements in reconstruction speed and quality compared to traditional algorithms. However, these networks fall short in exploiting the sparsity of hyperspectral images in their direct mapping from 2D measurements to 3D hyperspectral images. To address this limitation, this paper introduces a novel approach for compressive hyperspectral image reconstruction using deep learning. The deep learning network comprises two components: the sparse coefficient recovery net and the dictionary training net. The dictionary training net incorporates a pre-trained dictionary, whose parameters can also be updated during the backpropagation process. Concurrently, the sparse coefficient recovery net generates sparse coefficients under this dictionary. Our approach draws inspiration from conventional dictionary learning algorithms, enabling effective utilization of signal sparsity. In comparison to several state-of-the-art methods, our method achieves superior reconstruction quality and demonstrates enhanced robustness against noise. The proposed approach is envisioned to offer fresh insights into hyperspectral images reconstruction using deep learning models.

Multishot Compressive Hyperspectral Imaging Based on Tensor Fibered Rank Minimization and Its Primal-Dual Algorithm

Multishot Compressive Hyperspectral Imaging Based on Tensor Fibered Rank Minimization and Its Primal-Dual Algorithm

Compressive hyperspectral imaging based on Images Structure Similarity and deep image prior

In this paper, the structural similarity between RGB image and spectral image is studied, and a non-iterative Images Structure Similarity(ISS) method for fast reconstruction of spectral image is proposed. At the same time, the input of the Deep Image Prior (DIP) method is optimized for the first time by using the initial spectral data reconstructed by ISS. It raises the starting value of the iteration. Specifically, we take the RGB image data as the base of the spectral data, and solve the base coefficient by the least square method to quickly estimate the initial hyperspectral image. The Gaussian noise data is replaced by the estimated initial spectral data to constrain the solution space of the network and reduce the number of iterations. Finally, the structural similarity between RGB image and spectral image is used. The RGB three-channel graph is used to filter the iterative results to improve the reconstruction quality. Experimental results show that compared with other hyperspectral imaging methods, the proposed method can improve the quality of reconstruction in both spectral resolution and spatial resolution. In addition, compared with other methods based on Deep Image Prior (DIP), our improvement greatly reduces the reconstruction time and is more suitable for actual snapshot spectral imaging.

Bi-channel compressive hyperspectral imager based on polarization conversion metasurface

Metasurfaces have demonstrated remarkable capabilities in manipulating light fields and reducing the size of conventional optical elements to ultrathin devices, and compressive hyperspectral imaging systems can effectively utilize the spectral modulations of metasurfaces through spectral transmission tuning. In this paper, a bi-channel compressive hyperspectral imager based on polarization conversion metasurfaces is proposed. The metasurface layer of this device is able to modulate and convert the polarized incidence into two outputs of different transmission spectra with orthogonal polarizations, providing two channels of hyperspectral information that can be detected by a polarization image sensor in one snapshot. Additionally, this design offers a compact size, high speed, and accurate reconstruction through the utilization of a macro-pixel segmentation method for compressive sensing algorithms, and it is firmly demonstrated that the proposed hyperspectral imaging technology surpasses other traditional compressive hyperspectral image recovery methods in terms of improved reconstruction quality. Therefore, it is envisioned that this technology will find significant applications in various fields, including food safety, biomedical imaging, precision agriculture, object detection and so on.

InSPECtor: an end-to-end design framework for compressive pixelated hyperspectral instruments.

Classic designs of hyperspectral instrumentation densely sample the spatial and spectral information of the scene of interest. Data may be compressed after the acquisition. In this paper, we introduce a framework for the design of an optimized, micropatterned snapshot hyperspectral imager that acquires an optimized subset of the spatial and spectral information in the scene. The data is thereby already compressed at the sensor level but can be restored to the full hyperspectral data cube by the jointly optimized reconstructor. This framework is implemented with TensorFlow and makes use of its automatic differentiation for the joint optimization of the layout of the micropatterned filter array as well as the reconstructor. We explore the achievable compression ratio for different numbers of filter passbands, number of scanning frames, and filter layouts using data collected by the Hyperscout instrument. We show resulting instrument designs that take snapshot measurements without losing significant information while reducing the data volume, acquisition time, or detector space by a factor of 40 as compared to classic, dense sampling. The joint optimization of a compressive hyperspectral imager design and the accompanying reconstructor provides an avenue to substantially reduce the data volume from hyperspectral imagers.

Scanning-based compressive hyperspectral imaging via spectral-coded illumination.

In this Letter, we present a novel, to the best of our knowledge, scanning-based compressive hyperspectral imaging method via spectral-coded illumination. We achieve efficient and flexible spectral modulation by spectral coding of a dispersive light source while spatial information is obtained by point-wise scanning, which can be applied to optical scanning imaging systems such as lidar. In addition, we propose a new tensor-based joint hyperspectral image reconstruction algorithm that considers spectral correlation and spatial self-similarity to recover three-dimensional hyperspectral data from compressive sampled data. Both simulated and real experiments show that our method has superior performance in visual quality and quantitative analysis.

Superpixel-based linear reconstruction method for dual-camera compressed hyperspectral imaging system

Compressive spectral imaging systems have received significant research attention in recent years. Among these systems, the dual-camera compressive hyperspectral imaging (DCCHI) design, has the potential to improve imaging resolution while maintaining the snapshot advantage. In this paper, we propose superpixel-based linear reconstruction methods for DCCHI system. Specifically, we assume that the spectral intensity of the superpixels generated from the RGB image varies linearly in the spectral dimension. Instead of addressing an ill-posed problem, we reconstruct hyperspectral images via solving overdetermined equations using the least squares method, resulting in a significant reduction in both computation time and memory usage. To further enhance the reconstruction accuracy, we extend the spectra of superpixels from a 1D linear space into a 3D space with an RGB spectral response matrix. Despite the simplicity of our method, it offers improved reconstruction accuracy of hyperspectral images compared to other state-of-the-art methods. Furthermore, it is more efficient in terms of memory usage and computational time.

Dual-camera compressive hyperspectral imaging based on deep image prior and a guided filter.

Coded aperture snapshot spectral imaging (CASSI) aims to capture the high-dimensional (usually 3D) data cube using a 2D sensor in a single snapshot. Due to the ill-posed snapshot, the reconstruction results are not ideal. One feasible solution is to utilize additional information such as the panchromatic measurement in CASSI. In this paper, we propose a dual-camera hyperspectral reconstruction method based on the deep image prior (DIP) and a guided filter. In particular, the panchromatic measurements are used to estimate spatial detail, and spectral details are provided using CASSI measurements. These measurements are used as a priori learning by the self-supervised network. Using iteration combined with DIP, the hyperspectral reconstruction is continuously updated iteratively. Finally, the panchromatic measurement is used as the guidance image, and the reconstruction result is optimized by guide filtering. A large number of experimental results demonstrate that our method without training data can reconstruct spectral data with both high spectral accuracy and spatial resolution.

Plug-and-Play Priors for Multi-Shot Compressive Hyperspectral Imaging.

Multi-shot coded aperture snapshot spectral imaging (CASSI) uses multiple measurement snapshots to encode the three-dimensional hyperspectral image (HSI). Increasing the number of snapshots will multiply the number of measurements, making CASSI system more appropriate for detailed spatial or spectrally rich scenes. However, the reconstruction algorithms still face the challenge of being ineffective or inflexible. In this paper, we propose a plug-and-play (PnP) method that uses denoiser as priors for multi-shot CASSI. Specifically, the proposed PnP method is based on the primal-dual algorithm with linesearch (PDAL), which makes it flexible and can be used for any multi-shot CASSI mechanisms. Furthermore, a new subspaced-based nonlocal reweighted low-rank (SNRL) denoiser is presented to utilize the global spectral correlation and nonlocal self-similarity priors of HSI. By integrating the SNRL denoiser into PnP-PDAL, we show the balloons (512 × 512 × 31) in CAVE dataset recovered from two snapshots compressive measurements with MPSNR above 50 dB. Experimental results demonstrate that our proposed method leads to significant improvements compared to the current state-of-the-art methods.

Push-broom compressive hyperspectral imaging method based on self-fusion refinement

Compressive hyperspectral imaging (CHI) with random encoding mask usually suffers from various noises and artifacts. Inspired by the dual-camera CHI techniques based on hyperspectral (HS) and multispectral (MS) image fusion, herein, we present a single-camera push-broom CHI method based on self-fusion refinement (SFR). In this work, the MS guidance image for data fusion is derived directly from the raw solved HS data cube itself rather than any additional data source, which turns cross-fusion into self-fusion; furthermore, a modified joint bilateral filtering (JBF) fusion algorithm is developed to adapt this self-fusion problem, and an adaptive range Gaussian radius is adopted to avoid the invalidation or over-smoothing effects so as to ensure spatial and spectral improvement. The visualized and quantitative assessment results both demonstrate that the proposed method achieves high-quality HS imaging in terms of noise and artifact removal and spatial–spectral fidelity. Furthermore, the proposed method has a great flexibility and extensibility, whose performances highly depend on the exact fusion algorithm adopted, and a more suitable fusion algorithm will lead to better reconstruction quality; herein, the SFR process by the modified JBF achieves better performances than SFR by guided filtering (GF) or Markov random field (MRF).

Feasibility of a Real-Time Embedded Hyperspectral Compressive Sensing Imaging System.

Hyperspectral imaging has been attracting considerable interest as it provides spectrally rich acquisitions useful in several applications, such as remote sensing, agriculture, astronomy, geology and medicine. Hyperspectral devices based on compressive acquisitions have appeared recently as an alternative to conventional hyperspectral imaging systems and allow for data-sampling with fewer acquisitions than classical imaging techniques, even under the Nyquist rate. However, compressive hyperspectral imaging requires a reconstruction algorithm in order to recover all the data from the raw compressed acquisition. The reconstruction process is one of the limiting factors for the spread of these devices, as it is generally time-consuming and comes with a high computational burden. Algorithmic and material acceleration with embedded and parallel architectures (e.g., GPUs and FPGAs) can considerably speed up image reconstruction, making hyperspectral compressive systems suitable for real-time applications. This paper provides an in-depth analysis of the required performance in terms of computing power, data memory and bandwidth considering a compressive hyperspectral imaging system and a state-of-the-art reconstruction algorithm as an example. The results of the analysis show that real-time application is possible by combining several approaches, namely, exploitation of system matrix sparsity and bandwidth reduction by appropriately tuning data value encoding.

Refinement method for compressive hyperspectral data cubes based on self-fusion.

Compressive hyperspectral images often suffer from various noises and artifacts, which severely degrade the imaging quality and limit subsequent applications. In this paper, we present a refinement method for compressive hyperspectral data cubes based on self-fusion of the raw data cubes, which can effectively reduce various noises and improve the spatial and spectral details of the data cubes. To verify the universality, flexibility, and extensibility of the self-fusion refinement (SFR) method, a series of specific simulations and practical experiments were conducted, and SFR processing was performed through different fusion algorithms. The visual and quantitative assessments of the results demonstrate that, in terms of noise reduction and spatial-spectral detail restoration, the SFR method generally is much better than other typical denoising methods for hyperspectral data cubes. The results also indicate that the denoising effects of SFR greatly depend on the fusion algorithm used, and SFR implemented by joint bilateral filtering (JBF) performs better than SRF by guided filtering (GF) or a Markov random field (MRF). The proposed SFR method can significantly improve the quality of a compressive hyperspectral data cube in terms of noise reduction, artifact removal, and spatial and spectral detail improvement, which will further benefit subsequent hyperspectral applications.

Design and Dispersion Calibration of Direct-Vision Push-Broom Compressive Double-Amici-Prism Hyperspectral Imager

The design and calibration of the dispersive device in a hyperspectral imager significantly affect the performance of hyperspectral imaging, especially the spectral accuracy. To achieve high-accuracy hyperspectral imaging over the visible band, firstly, the geometric and dispersive parameters of the double Amici prism (DAP) that serves as a dispersive device in the direct-vision push-broom compressive hyperspectral imager (PBCHI) are designed and optimized; secondly, a calibration method based on the numerical calculation of the DAP model is put forward, which can turn the conventional pixel-wise dispersive shift calibration by a monochromator into a group of numerical calculations; lastly, a PBCHI prototype is built to test the performances of the designed and calibrated DAP and the hyperspectral imager. The calibration experiments demonstrate that the mean squared error (MSE) of the dispersive pixel shifts calibrated by the proposed numerical method is 0.1774, which indicates the calibration result of the proposed method is consistent with the directly calibrated result. Furthermore, after this numerical calculation, the spectral signatures of the reconstructed cubes of the DAP-based PBCHI system show consistency with the ground truth. This work will benefit the design and calibration of the DAP-based hyperspectral imager.

Learned Design of a Compressive Hyperspectral Imager for Remote Sensing by a Physics-Constrained Autoencoder

Designing and optimizing systems by end-to-end deep learning is a recently emerging field. We present a novel physics-constrained autoencoder (PyCAE) for the design and optimization of a physically realizable sensing model. As a case study, we design a compressive hyperspectral imaging system for remote sensing based on this approach, which allows capturing hundreds of spectral bands with as few as four compressed measurements. We demonstrate our deep learning approach to design spectral compression with a spectral light modulator (SpLM) encoder and a reconstruction neural network decoder. The SpLM consists of a set of modified Fabry–Pérot resonator (mFPR) etalons that are designed to have a staircase-shaped geometry. Each stair occupies a few pixel columns of a push-broom-like spectral imager. The mFPR’s stairs can sample the earth terrain in along-track scanning from an airborne or spaceborne moving platform. The SpLM is jointly designed with an autoencoder by a data-driven approach, while spectra from remote sensing databases are used to train the system. The SpLM’s parameters are optimized by integrating its physically realizable sensing model in the encoder part of the PyCAE. The decoder part of the PyCAE implements the spectral reconstruction.

Spectral DefocusCam: Compressive Hyperspectral Imaging from Defocus Measurements

Hyperspectral imaging is used for a wide range of tasks from medical diagnostics to crop monitoring, but traditional imagers are prohibitively expensive for widespread use. This research strives to democratize hyperspectral imaging by using machine learning to reconstruct hyperspectral volumes from snapshot imagers. I propose a tunable lens with varying amounts of defocus paired with 31-channel spectral filter array mounted on a CMOS camera. These images are then fed into a reconstruction network that aims to recover the full 31-channel hyperspectral volume from a few encoded images with different amounts of defocus.

Residual image recovery method based on the dual-camera design of a compressive hyperspectral imaging system.

Compressive hyperspectral imaging technology can quickly detect the encoded two-dimensional measurements and reconstruct the three-dimensional hyperspectral images offline, which is of great significance for object detection and analysis. To provide more information for reconstruction and improve the reconstruction quality, some of the latest compressive hyperspectral imaging systems adopt a dual-camera design. To utilize the information from additional camera more efficiently, this paper proposes a residual image recovery method. The proposed method takes advantage of the structural similarity between the image captured by the additional camera and the hyperspectral image, combining the measurements from the additional camera and coded aperture snapshot spectral imaging (CASSI) sensor to construct an estimated hyperspectral image. Then, the component of the estimated hyperspectral image is subtracted from the measurement of the CASSI sensor to obtain the residual data. The residual data is used to reconstruct the residual hyperspectral image. Finally, the reconstructed hyperspectral image is the sum of the estimated and residual image. Compared with some state-of-the-art algorithms based on such systems, the proposed method can significantly improve the reconstruction quality of hyperspectral image.

Backtracking Reconstruction Network for Three-Dimensional Compressed Hyperspectral Imaging

Compressed sensing (CS) has been widely used in hyperspectral (HS) imaging to obtain hyperspectral data at a sub-Nyquist sampling rate, lifting the efficiency of data acquisition. Yet, reconstructing the acquired HS data via iterative algorithms is time consuming, which hinders the real-time application of compressed HS imaging. To alleviate this problem, this paper makes the first attempt to adopt convolutional neural networks (CNNs) to reconstruct three-dimensional compressed HS data by backtracking the entire imaging process, leading to a simple yet effective network, dubbed the backtracking reconstruction network (BTR-Net). Concretely, we leverage the divide-and-conquer method to divide the imaging process based on coded aperture tunable filter (CATF) spectral imager into steps, and build a subnetwork for each step to specialize in its reverse process. Consequently, BTR-Net introduces multiple built-in networks which performs spatial initialization, spatial enhancement, spectral initialization and spatial–spectral enhancement in an independent and sequential manner. Extensive experiments show that BTR-Net can reconstruct compressed HS data quickly and accurately, which outperforms leading iterative algorithms both quantitatively and visually, while having superior resistance to noise.