Snapshot Hyperspectral Imaging Research Articles

ChromaFlash: Snapshot Hyperspectral Imaging Using Rolling Shutter Cameras

Hyperspectral imaging captures scene information across narrow, contiguous bands of the electromagnetic spectrum. Despite its proven utility in industrial and biomedical applications, its ubiquity has been limited by bulky form factors, slow capture times, and prohibitive costs. In this work, we propose a generalized approach to snapshot hyperspectral imaging that only requires a standard rolling shutter camera and wavelength-adjustable lighting. The crux of this approach entails using the rolling shutter as a spatiotemporal mask, varying incoming light quicker than the camera's frame rate in order for the captured image to contain rows of pixels illuminated at different wavelengths. An image reconstruction pipeline then converts this coded image into a complete hyperspectral image using sparse optimization. We demonstrate the feasibility of this approach by deploying a low-cost system called ChromaFlash, which uses a smartphone's camera for image acquisition and a series of LEDs to change the scene's illumination. We evaluated ChromaFlash through simulations on two public hyperspectral datasets and assessed its spatial and spectral accuracy across various system parameters. We also tested the real-world performance of our prototype by capturing diverse scenes under varied ambient lighting conditions. In both experiments, ChromaFlash outperformed state-of-the-art techniques that use deep learning to convert RGB images into hyperspectral ones, achieving snapshot performance not demonstrated by prior attempts at accessible hyperspectral imaging.

Uncertainties in radiometric measurements of the ocean surface from above water and helicopter hyperspectral and polarimetric observations

A new system was assembled with a snapshot hyperspectral imager and a polarimetric camera for quantifying uncertainties in aquatic remote sensing applications. The hyperspectral imager measures radiances from a field-of-view (FOV) creating imagery for each of its spectral bands. The polarimetric camera measures the Stokes vector components of the radiance. A combination of polarimetric and hyperspectral measurements provides data about the wind-roughened surface in various water and sky conditions. Uncertainties in the total radiance L t in unpolarized and polarized modes are estimated by observations from the ships in two ocean cruises, from a nearshore platform, and a helicopter in the Chesapeake Bay showing that uncertainties are a combination of the effects of the reflected sky from the surface (ρL s ), water leaving radiance (L w ), and Rayleigh scattering; the impact of the latter increases with the height.

Optical design and development of an underwater dual-channel microlens array integral field snapshot hyperspectral imager

Compared to push-scan hyperspectral imagers, snapshot hyperspectral imagers offer an advantage by minimizing sensitivity to attitude jitter in underwater mobile platforms. Here we present the optical design and development of an underwater microlens array integral field hyperspectral imager. The system comprises a panchromatic imaging channel with a high spatial resolution and a spectral imaging channel with a lower spatial resolution. Through the fusion of high-resolution panchromatic images and low-resolution spectral images, we achieve high spatial resolution hyperspectral images. Both the panchromatic imaging channel and the spectral imaging channel share a common front objective, featuring a 25 mm focal length and a wide 36° field of view angle. Utilizing prism dispersion, the spectral imaging system spans a band range from 465 to 700 nm with a spectral resolution of less than 10 nm. Specialized algorithms for spectral image reconstruction and image fusion have been developed. The experimental results across diverse scenes confirm the exemplary spectral imaging performance of the system, positioning it as a robust solution for underwater snapshot hyperspectral imaging.

Water pollution classification and detection by hyperspectral imaging.

This study utilizes spectral analysis to quantify water pollutants by analyzing the images of biological oxygen demand (BOD). In this study, a total of 2545 images depicting water quality pollution were generated due to the absence of a standardized water pollution detection method. A novel snap-shot hyperspectral imaging (HSI) conversion algorithm has been developed to conduct spectral analysis on traditional RGB images. In order to demonstrate the effectiveness of the developed HSI algorithm, two distinct three-dimensional convolution neural networks (3D-CNN) are employed to train two separate datasets. One dataset is based on the HSI conversion algorithm (HSI-3DCNN), while the other dataset is the traditional RGB dataset (RGB-3DCNN). The images depicting water quality pollution were categorized into three distinct groups: Good, Normal, and Severe, based on the extent of pollution severity. A comparison was conducted between the HSI and RGB models, focusing on precision, recall, F1-score, and accuracy. The water pollution model's accuracy improved from 76% to 80% when the RGB-3DCNN was substituted with the HSI-3DCNN. The results suggest that the HSI has the capacity to enhance the effectiveness of water pollution detection compared to the RGB model.

Super-resolved snapshot hyperspectral imaging of solid-state quantum emitters for high-throughput integrated quantum technologies

Super-resolved snapshot hyperspectral imaging of solid-state quantum emitters for high-throughput integrated quantum technologies

Deep learning-enhanced snapshot hyperspectral confocal microscopy imaging system

Laser-scanning confocal hyperspectral microscopy is a powerful technique to identify the different sample constituents and their spatial distribution in three-dimensional (3D). However, it suffers from low imaging speed because of the mechanical scanning methods. To overcome this challenge, we propose a snapshot hyperspectral confocal microscopy imaging system (SHCMS). It combined coded illumination microscopy based on a digital micromirror device (DMD) with a snapshot hyperspectral confocal neural network (SHCNet) to realize single-shot confocal hyperspectral imaging. With SHCMS, high-contrast 160-bands confocal hyperspectral images of potato tuber autofluorescence can be collected by only single-shot, which is almost 5 times improvement in the number of spectral channels than previously reported methods. Moreover, our approach can efficiently record hyperspectral volumetric imaging due to the optical sectioning capability. This fast high-resolution hyperspectral imaging method may pave the way for real-time highly multiplexed biological imaging.

Identification of Skin Lesions by Snapshot Hyperspectral Imaging.

This study pioneers the application of artificial intelligence (AI) and hyperspectral imaging (HSI) in the diagnosis of skin cancer lesions, particularly focusing on Mycosis fungoides (MF) and its differentiation from psoriasis (PsO) and atopic dermatitis (AD). By utilizing a comprehensive dataset of 1659 skin images, including cases of MF, PsO, AD, and normal skin, a novel multi-frame AI algorithm was used for computer-aided diagnosis. The automatic segmentation and classification of skin lesions were further explored using advanced techniques, such as U-Net Attention models and XGBoost algorithms, transforming images from the color space to the spectral domain. The potential of AI and HSI in dermatological diagnostics was underscored, offering a noninvasive, efficient, and accurate alternative to traditional methods. The findings are particularly crucial for early-stage invasive lesion detection in MF, showcasing the model's robust performance in segmenting and classifying lesions and its superior predictive accuracy validated through k-fold cross-validation. The model attained its optimal performance with a k-fold cross-validation value of 7, achieving a sensitivity of 90.72%, a specificity of 96.76%, an F1-score of 90.08%, and an ROC-AUC of 0.9351. This study marks a substantial advancement in dermatological diagnostics, thereby contributing significantly to the early and precise identification of skin malignancies and inflammatory conditions.

Non-Serial Quantization-Aware Deep Optics for Snapshot Hyperspectral Imaging.

Deep optics has been endeavoring to capture hyperspectral images of dynamic scenes, where the optical encoder plays an essential role in deciding the imaging performance. Our key insight is that the optical encoder of a deep optics system is expected to keep fabrication-friendliness and decoder-friendliness, to be faithfully realized in the implementation phase and fully interacted with the decoder in the design phase, respectively. In this paper, we propose the non-serial quantization-aware deep optics (NSQDO), which consists of the fabrication-friendly quantization-aware model (QAM) and the decoder-friendly non-serial manner (NSM). The QAM integrates the quantization process into the optimization and adaptively adjusts the physical height of each quantization level, reducing the deviation of the physical encoder from the numerical simulation through the awareness of and adaptation to the quantization operation of the DOE physical structure. The NSM bridges the encoder and the decoder with full interaction through bidirectional hint connections and flexibilize the connections with a gating mechanism, boosting the power of joint optimization in deep optics. The proposed NSQDO improves the fabrication-friendliness and decoder-friendliness of the encoder and develops the deep optics framework to be more practical and powerful. Extensive synthetic simulation and real hardware experiments demonstrate the superior performance of the proposed method.

Lenslet-array-based snapshot hyperspectral imaging polarimeter using opposite spectral modulation

We present a lenslet-array-based snapshot hyperspectral imaging polarimeter (LSHIP) that integrates spectral polarization modulation (SPM) into integral field imaging spectrometry. The LSHIP can simultaneously acquire three-dimensional spatial and spectral data-cubes for linear Stokes parameters in a single snapshot without any moving parts. The dual-scan technique is achieved through a custom-fabricated Wollaston prism that can simultaneously generate pairs of micro-spectra with opposite spectral modulations, maintaining the system's intrinsic spectral resolution during intensity spectrum reconstruction. We describe the principle and experimental setup of the LSHIP, as well as the system calibration and validation using measurements of polarized color blocks and a sphere made of polytetrafluoroethylene material. The prototype can cover the spectral range of 450–650 nm with 157 sample points, capturing approximately 3700 SPM micro-spectra spatially distributed on the detector in a single integration time. These results demonstrate the potential of the LSHIP for various applications, including remote sensing and biomedical diagnosis.

Metasurface-empowered snapshot hyperspectral imaging with convex/deep (CODE) small-data learning theory

Hyperspectral imaging is vital for material identification but traditional systems are bulky, hindering the development of compact systems. While previous metasurfaces address volume issues, the requirements of complicated fabrication processes and significant footprint still limit their applications. This work reports a compact snapshot hyperspectral imager by incorporating the meta-optics with a small-data convex/deep (CODE) deep learning theory. Our snapshot hyperspectral imager comprises only one single multi-wavelength metasurface chip working in the visible window (500-650 nm), significantly reducing the device area. To demonstrate the high performance of our hyperspectral imager, a 4-band multispectral imaging dataset is used as the input. Through the CODE-driven imaging system, it efficiently generates an 18-band hyperspectral data cube with high fidelity using only 18 training data points. We expect the elegant integration of multi-resonant metasurfaces with small-data learning theory will enable low-profile advanced instruments for fundamental science studies and real-world applications.

Pediatric Brain Tissue Segmentation Using a Snapshot Hyperspectral Imaging (sHSI) Camera and Machine Learning Classifier.

Pediatric brain tumors are the second most common type of cancer, accounting for one in four childhood cancer types. Brain tumor resection surgery remains the most common treatment option for brain cancer. While assessing tumor margins intraoperatively, surgeons must send tissue samples for biopsy, which can be time-consuming and not always accurate or helpful. Snapshot hyperspectral imaging (sHSI) cameras can capture scenes beyond the human visual spectrum and provide real-time guidance where we aim to segment healthy brain tissues from lesions on pediatric patients undergoing brain tumor resection. With the institutional research board approval, Pro00011028, 139 red-green-blue (RGB), 279 visible, and 85 infrared sHSI data were collected from four subjects with the system integrated into an operating microscope. A random forest classifier was used for data analysis. The RGB, infrared sHSI, and visible sHSI models achieved average intersection of unions (IoUs) of 0.76, 0.59, and 0.57, respectively, while the tumor segmentation achieved a specificity of 0.996, followed by the infrared HSI and visible HSI models at 0.93 and 0.91, respectively. Despite the small dataset considering pediatric cases, our research leveraged sHSI technology and successfully segmented healthy brain tissues from lesions with a high specificity during pediatric brain tumor resection procedures.

Snapshot hyperspectral imaging of intracellular lasers.

Intracellular lasers are emerging as powerful biosensors for multiplexed tracking and precision sensing of cells and their microenvironment. This sensing capacity is enabled by quantifying their narrow-linewidth emission spectra, which is presently challenging to do at high speeds. In this work, we demonstrate rapid snapshot hyperspectral imaging of intracellular lasers. Using integral field mapping with a microlens array and a diffraction grating, we obtain images of the spatial and spectral intensity distribution from a single camera acquisition. We demonstrate widefield hyperspectral imaging over a 3 × 3 mm2 field of view and volumetric imaging over 250 × 250 × 800 µm3 (XYZ) volumes with a lateral (XY) resolution of 5 µm, axial (Z) resolution of 10 µm, and a spectral resolution of less than 0.8 nm. We evaluate the performance and outline the challenges and strengths of snapshot methods in the context of characterizing the emission from intracellular lasers. This method offers new opportunities for a diverse range of applications, including high-throughput and long-term biosensing with intracellular lasers.

InSPECtor: an end-to-end design framework for compressive pixelated hyperspectral instruments.

Classic designs of hyperspectral instrumentation densely sample the spatial and spectral information of the scene of interest. Data may be compressed after the acquisition. In this paper, we introduce a framework for the design of an optimized, micropatterned snapshot hyperspectral imager that acquires an optimized subset of the spatial and spectral information in the scene. The data is thereby already compressed at the sensor level but can be restored to the full hyperspectral data cube by the jointly optimized reconstructor. This framework is implemented with TensorFlow and makes use of its automatic differentiation for the joint optimization of the layout of the micropatterned filter array as well as the reconstructor. We explore the achievable compression ratio for different numbers of filter passbands, number of scanning frames, and filter layouts using data collected by the Hyperscout instrument. We show resulting instrument designs that take snapshot measurements without losing significant information while reducing the data volume, acquisition time, or detector space by a factor of 40 as compared to classic, dense sampling. The joint optimization of a compressive hyperspectral imager design and the accompanying reconstructor provides an avenue to substantially reduce the data volume from hyperspectral imagers.

Snapshot hyperspectral imaging based on equalization designed DOE.

Hyperspectral imaging attempts to determine distinctive information in spatial and spectral domain of a target. Over the past few years, hyperspectral imaging systems have developed towards lighter and faster. In phase-coded hyperspectral imaging systems, a better coding aperture design can improve the spectral accuracy relatively. Using wave optics, we post an equalization designed phase-coded aperture to achieve desired equalization point spread functions (PSFs) which provides richer features for subsequent image reconstruction. During the reconstruction of images, our raised hyperspectral reconstruction network, CAFormer, achieves better results than the state-of-the-art networks with less computation by substituting self-attention with channel-attention. Our work revolves around the equalization design of the phase-coded aperture and optimizes the imaging process from three aspects: hardware design, reconstruction algorithm, and PSF calibration. Our work is putting snapshot compact hyperspectral technology closer to a practical application.

Aperture-encoded snapshot hyperspectral imaging with a lensless camera

We present a lensless snapshot hyperspectral camera that is capable of hyperspectral imaging over a broad spectrum using a compact and low-cost hardware configuration. We leverage the multiplexing capability of a lensless camera, a novel type of computational imaging device that replaces the lens with a thin mask. Our device utilizes a linear variable filter and a phase mask to encode spectral information onto a monochromatic image sensor, enabling recovery of hyperspectral image stacks from a single measurement by utilizing spectral information encoded in different parts of the 2D point spread function. We perform spectral calibration using a reference color chart and verify the prototype device’s spectral and spatial resolution, as well as its imaging field of view. We report on the design and construction of the device, the image reconstruction algorithm, and spectral calibration methods and present hyperspectral images ranging from 410 to 800 nm obtained with our prototype device.

On-chip hyperspectral image segmentation with fully convolutional networks for scene understanding in autonomous driving

Most of current computer vision-based advanced driver assistance systems (ADAS) perform detection and tracking of objects quite successfully under regular conditions. However, under adverse weather and changing lighting conditions, and in complex situations with many overlapping objects, these systems are not completely reliable. The spectral reflectance of the different objects in a driving scene beyond the visible spectrum can offer additional information to increase the reliability of these systems, especially under challenging driving conditions. Furthermore, this information may be significant enough to develop vision systems that allow for a better understanding and interpretation of the whole driving scene. In this work we explore the use of snapshot, video-rate hyperspectral imaging (HSI) cameras in ADAS on the assumption that the near infrared (NIR) spectral reflectance of different materials can help to better segment the objects in real driving scenarios. To do this, we have used the HSI-Drive 1.1 dataset to perform various experiments on spectral classification algorithms. However, the information retrieval of hyperspectral recordings in natural outdoor scenarios is challenging, mainly because of deficient color constancy and other inherent shortcomings of current snapshot HSI technology, which poses some limitations to the development of pure spectral classifiers. In consequence, in this work we analyze to what extent the spatial features codified by standard, tiny fully convolutional network (FCN) models can improve the performance of HSI segmentation systems for ADAS applications. In order to be realistic from an engineering viewpoint, this research is focused on the development of a feasible HSI segmentation system for ADAS, which implies considering implementation constraints and latency specifications throughout the algorithmic development process. For this reason, it is of particular importance to include the study of the raw image preprocessing stage into the data processing pipeline. Accordingly, this paper describes the development and deployment of a complete machine learning-based HSI segmentation system for ADAS, including the characterization of its performance on different embedded computing platforms, including a single board computer, an embedded GPU SoC and a programmable system on chip (PSoC) with embedded FPGA. We verify the superiority of the FPGA-PSoC over the GPU-SoC in terms of energy consumption and, particularly, processing latency, and demonstrate that it is feasible to achieve segmentation speeds within the range of ADAS industry specifications using standard development tools.

Optical Design and Simulation of Snapshot Hyperspectral Dental Imaging Spectrometer with One-Dimensional Random Coding

To achieve perfect teeth color reproduction, we propose the design of a snapshot hyperspectral imaging spectrometer that can accurately measure the teeth spectrum. In this paper, the optical system of a snapshot hyperspectral dental imaging spectrometer is designed. In particular, to encode images in both the spatial and spectral dimensions, a one-dimensional random coding mask rotated 45 degrees around the optical axis is used. The system has a spectral resolution of approximately 2.2∼10 nm in the 450∼700 nm spectral range, and the total length of the system is less than 150 mm. The whole design can be used as a miniaturized handheld instrument. To verify the feasibility of whiteness measuring, a simulation experiment is carried out. The target spectral information is reconstructed through the reconstruction algorithm. The simulation result shows that spectral imaging has the potential to improve the accuracy and precision of teeth whiteness measurements.

A birefringent spectral demultiplexer enables fast hyper-spectral imaging of protoporphyrin IX during neurosurgery

Hyperspectral imaging and spectral analysis quantifies fluorophore concentration during fluorescence-guided surgery1–6. However, acquisition of the multiple wavelengths required to implement these methods can be time-consuming and hinder surgical workflow. To this end, a snapshot hyperspectral imaging system capable of acquiring 64 channels of spectral data simultaneously was developed for rapid hyperspectral imaging during neurosurgery. The system uses a birefringent spectral demultiplexer to split incoming light and redirect wavelengths to different sections of a large format microscope sensor. Its configuration achieves high optical throughput, accepts unpolarized input light and exceeds channel count of prior image-replicating imaging spectrometers by 4-fold. Tissue-simulating phantoms consisting of serial dilutions of the fluorescent agent characterize system linearity and sensitivity, and comparisons to performance of a liquid crystal tunable filter based hyperspectral imaging device are favorable. The new instrument showed comparable, if not improved, sensitivity at low fluorophore concentrations; yet, acquired wide-field images at more than 70-fold increase in frame rate. Image data acquired in the operating room during human brain tumor resection confirm these findings. The new device is an important advance in achieving real-time quantitative imaging of fluorophore concentration for guiding surgery.

MHealth hyperspectral learning for instantaneous spatiospectral imaging of hemodynamics.

Hyperspectral imaging acquires data in both the spatial and frequency domains to offer abundant physical or biological information. However, conventional hyperspectral imaging has intrinsic limitations of bulky instruments, slow data acquisition rate, and spatiospectral trade-off. Here we introduce hyperspectral learning for snapshot hyperspectral imaging in which sampled hyperspectral data in a small subarea are incorporated into a learning algorithm to recover the hypercube. Hyperspectral learning exploits the idea that a photograph is more than merely a picture and contains detailed spectral information. A small sampling of hyperspectral data enables spectrally informed learning to recover a hypercube from a red-green-blue (RGB) image without complete hyperspectral measurements. Hyperspectral learning is capable of recovering full spectroscopic resolution in the hypercube, comparable to high spectral resolutions of scientific spectrometers. Hyperspectral learning also enables ultrafast dynamic imaging, leveraging ultraslow video recording in an off-the-shelf smartphone, given that a video comprises a time series of multiple RGB images. To demonstrate its versatility, an experimental model of vascular development is used to extract hemodynamic parameters via statistical and deep learning approaches. Subsequently, the hemodynamics of peripheral microcirculation is assessed at an ultrafast temporal resolution up to a millisecond, using a conventional smartphone camera. This spectrally informed learning method is analogous to compressed sensing; however, it further allows for reliable hypercube recovery and key feature extractions with a transparent learning algorithm. This learning-powered snapshot hyperspectral imaging method yields high spectral and temporal resolutions and eliminates the spatiospectral trade-off, offering simple hardware requirements and potential applications of various machine learning techniques.

CTIS-GAN: computed tomography imaging spectrometry based on a generative adversarial network.

Computed tomography imaging spectrometry (CTIS) is a snapshot hyperspectral imaging technique that can obtain a three-dimensional (2D spatial+1D spectral) data cube of the scene captured within a single exposure. The CTIS inversion problem is typically highly ill-posed and is usually solved by time-consuming iterative algorithms. This work aims to take the full advantage of the recent advances in deep-learning algorithms to dramatically reduce the computational cost. For this purpose, a generative adversarial network is developed and integrated with self-attention, which cleverly exploits the clearly utilizable features of zero-order diffraction of CTIS. The proposed network is able to reconstruct a CTIS data cube (containing 31 spectral bands) in milliseconds with a higher quality than traditional methods and the state-of-the-art (SOTA). Simulation studies based on real image data sets confirmed the robustness and efficiency of the method. In numerical experiments with 1000 samples, the average reconstruction time for a single data cube was ∼16m s. The robustness of the method against noise is also confirmed by numerical experiments with different levels of Gaussian noise. The CTIS generative adversarial network framework can be easily extended to solve CTIS problems with larger spatial and spectral dimensions, or migrated to other compressed spectral imaging modalities.