Snapshot Hyperspectral Imaging Research Articles

Non-serial Quantization-aware Deep Optics for Snapshot Hyperspectral Imaging.

Deep optics has been endeavoring to capture hyperspectral images of dynamic scenes, where the optical encoder plays an essential role in deciding the imaging performance. Our key insight is that the optical encoder of a deep optics system is expected to keep fabrication-friendliness and decoder-friendliness, to be faithfully realized in the implementation phase and fully interacted with the decoder in the design phase, respectively. In this paper, we propose the non-serial quantization-aware deep optics (NSQDO), which consists of the fabrication-friendly quantization-aware model (QAM) and the decoder-friendly non-serial manner (NSM). The QAM integrates the quantization process into the optimization and adaptively adjusts the physical height of each quantization level, reducing the deviation of the physical encoder from the numerical simulation through the awareness of and adaptation to the quantization operation of the DOE physical structure. The NSM bridges the encoder and the decoder with full interaction through bidirectional hint connections and flexibilize the connections with a gating mechanism, boosting the power of joint optimization in deep optics. The proposed NSQDO improves the fabrication-friendliness and decoder-friendliness of the encoder and develops the deep optics framework to be more practical and powerful. Extensive synthetic simulation and real hardware experiments demonstrate the superior performance of the proposed method.

Optical design and development of underwater dualchannel microlens array integral field snapshot hyperspectral imager

Optical design and development of underwater dualchannel microlens array integral field snapshot hyperspectral imager

Super-resolved snapshot hyperspectral imaging of solid-state quantum emitters for high-throughput integrated quantum technologies

Super-resolved snapshot hyperspectral imaging of solid-state quantum emitters for high-throughput integrated quantum technologies

Deep learning-enhanced snapshot hyperspectral confocal microscopy imaging system

Laser-scanning confocal hyperspectral microscopy is a powerful technique to identify the different sample constituents and their spatial distribution in three-dimensional (3D). However, it suffers from low imaging speed because of the mechanical scanning methods. To overcome this challenge, we propose a snapshot hyperspectral confocal microscopy imaging system (SHCMS). It combined coded illumination microscopy based on a digital micromirror device (DMD) with a snapshot hyperspectral confocal neural network (SHCNet) to realize single-shot confocal hyperspectral imaging. With SHCMS, high-contrast 160-bands confocal hyperspectral images of potato tuber autofluorescence can be collected by only single-shot, which is almost 5 times improvement in the number of spectral channels than previously reported methods. Moreover, our approach can efficiently record hyperspectral volumetric imaging due to the optical sectioning capability. This fast high-resolution hyperspectral imaging method may pave the way for real-time highly multiplexed biological imaging.

Identification of Skin Lesions by Snapshot Hyperspectral Imaging.

This study pioneers the application of artificial intelligence (AI) and hyperspectral imaging (HSI) in the diagnosis of skin cancer lesions, particularly focusing on Mycosis fungoides (MF) and its differentiation from psoriasis (PsO) and atopic dermatitis (AD). By utilizing a comprehensive dataset of 1659 skin images, including cases of MF, PsO, AD, and normal skin, a novel multi-frame AI algorithm was used for computer-aided diagnosis. The automatic segmentation and classification of skin lesions were further explored using advanced techniques, such as U-Net Attention models and XGBoost algorithms, transforming images from the color space to the spectral domain. The potential of AI and HSI in dermatological diagnostics was underscored, offering a noninvasive, efficient, and accurate alternative to traditional methods. The findings are particularly crucial for early-stage invasive lesion detection in MF, showcasing the model's robust performance in segmenting and classifying lesions and its superior predictive accuracy validated through k-fold cross-validation. The model attained its optimal performance with a k-fold cross-validation value of 7, achieving a sensitivity of 90.72%, a specificity of 96.76%, an F1-score of 90.08%, and an ROC-AUC of 0.9351. This study marks a substantial advancement in dermatological diagnostics, thereby contributing significantly to the early and precise identification of skin malignancies and inflammatory conditions.

Non-serial Quantization-aware Deep Optics for Snapshot Hyperspectral Imaging.

Deep optics has been endeavoring to capture hyperspectral images of dynamic scenes, where the optical encoder plays an essential role in deciding the imaging performance. Our key insight is that the optical encoder of a deep optics system is expected to keep fabrication-friendliness and decoder-friendliness, to be faithfully realized in the implementation phase and fully interacted with the decoder in the design phase, respectively. In this paper, we propose the non-serial quantization-aware deep optics (NSQDO), which consists of the fabrication-friendly quantization-aware model (QAM) and the decoder-friendly non-serial manner (NSM). The QAM integrates the quantization process into the optimization and adaptively adjusts the physical height of each quantization level, reducing the deviation of the physical encoder from the numerical simulation through the awareness of and adaptation to the quantization operation of the DOE physical structure. The NSM bridges the encoder and the decoder with full interaction through bidirectional hint connections and flexibilize the connections with a gating mechanism, boosting the power of joint optimization in deep optics. The proposed NSQDO improves the fabrication-friendliness and decoder-friendliness of the encoder and develops the deep optics framework to be more practical and powerful. Extensive synthetic simulation and real hardware experiments demonstrate the superior performance of the proposed method.

Lenslet-array-based snapshot hyperspectral imaging polarimeter using opposite spectral modulation

We present a lenslet-array-based snapshot hyperspectral imaging polarimeter (LSHIP) that integrates spectral polarization modulation (SPM) into integral field imaging spectrometry. The LSHIP can simultaneously acquire three-dimensional spatial and spectral data-cubes for linear Stokes parameters in a single snapshot without any moving parts. The dual-scan technique is achieved through a custom-fabricated Wollaston prism that can simultaneously generate pairs of micro-spectra with opposite spectral modulations, maintaining the system's intrinsic spectral resolution during intensity spectrum reconstruction. We describe the principle and experimental setup of the LSHIP, as well as the system calibration and validation using measurements of polarized color blocks and a sphere made of polytetrafluoroethylene material. The prototype can cover the spectral range of 450–650 nm with 157 sample points, capturing approximately 3700 SPM micro-spectra spatially distributed on the detector in a single integration time. These results demonstrate the potential of the LSHIP for various applications, including remote sensing and biomedical diagnosis.

Metasurface-empowered snapshot hyperspectral imaging with convex/deep (CODE) small-data learning theory

Hyperspectral imaging is vital for material identification but traditional systems are bulky, hindering the development of compact systems. While previous metasurfaces address volume issues, the requirements of complicated fabrication processes and significant footprint still limit their applications. This work reports a compact snapshot hyperspectral imager by incorporating the meta-optics with a small-data convex/deep (CODE) deep learning theory. Our snapshot hyperspectral imager comprises only one single multi-wavelength metasurface chip working in the visible window (500-650 nm), significantly reducing the device area. To demonstrate the high performance of our hyperspectral imager, a 4-band multispectral imaging dataset is used as the input. Through the CODE-driven imaging system, it efficiently generates an 18-band hyperspectral data cube with high fidelity using only 18 training data points. We expect the elegant integration of multi-resonant metasurfaces with small-data learning theory will enable low-profile advanced instruments for fundamental science studies and real-world applications.

Pediatric Brain Tissue Segmentation Using a Snapshot Hyperspectral Imaging (sHSI) Camera and Machine Learning Classifier.

Pediatric brain tumors are the second most common type of cancer, accounting for one in four childhood cancer types. Brain tumor resection surgery remains the most common treatment option for brain cancer. While assessing tumor margins intraoperatively, surgeons must send tissue samples for biopsy, which can be time-consuming and not always accurate or helpful. Snapshot hyperspectral imaging (sHSI) cameras can capture scenes beyond the human visual spectrum and provide real-time guidance where we aim to segment healthy brain tissues from lesions on pediatric patients undergoing brain tumor resection. With the institutional research board approval, Pro00011028, 139 red-green-blue (RGB), 279 visible, and 85 infrared sHSI data were collected from four subjects with the system integrated into an operating microscope. A random forest classifier was used for data analysis. The RGB, infrared sHSI, and visible sHSI models achieved average intersection of unions (IoUs) of 0.76, 0.59, and 0.57, respectively, while the tumor segmentation achieved a specificity of 0.996, followed by the infrared HSI and visible HSI models at 0.93 and 0.91, respectively. Despite the small dataset considering pediatric cases, our research leveraged sHSI technology and successfully segmented healthy brain tissues from lesions with a high specificity during pediatric brain tumor resection procedures.

Snapshot hyperspectral imaging of intracellular lasers.

Intracellular lasers are emerging as powerful biosensors for multiplexed tracking and precision sensing of cells and their microenvironment. This sensing capacity is enabled by quantifying their narrow-linewidth emission spectra, which is presently challenging to do at high speeds. In this work, we demonstrate rapid snapshot hyperspectral imaging of intracellular lasers. Using integral field mapping with a microlens array and a diffraction grating, we obtain images of the spatial and spectral intensity distribution from a single camera acquisition. We demonstrate widefield hyperspectral imaging over a 3 × 3 mm2 field of view and volumetric imaging over 250 × 250 × 800 µm3 (XYZ) volumes with a lateral (XY) resolution of 5 µm, axial (Z) resolution of 10 µm, and a spectral resolution of less than 0.8 nm. We evaluate the performance and outline the challenges and strengths of snapshot methods in the context of characterizing the emission from intracellular lasers. This method offers new opportunities for a diverse range of applications, including high-throughput and long-term biosensing with intracellular lasers.

InSPECtor: an end-to-end design framework for compressive pixelated hyperspectral instruments.

Classic designs of hyperspectral instrumentation densely sample the spatial and spectral information of the scene of interest. Data may be compressed after the acquisition. In this paper, we introduce a framework for the design of an optimized, micropatterned snapshot hyperspectral imager that acquires an optimized subset of the spatial and spectral information in the scene. The data is thereby already compressed at the sensor level but can be restored to the full hyperspectral data cube by the jointly optimized reconstructor. This framework is implemented with TensorFlow and makes use of its automatic differentiation for the joint optimization of the layout of the micropatterned filter array as well as the reconstructor. We explore the achievable compression ratio for different numbers of filter passbands, number of scanning frames, and filter layouts using data collected by the Hyperscout instrument. We show resulting instrument designs that take snapshot measurements without losing significant information while reducing the data volume, acquisition time, or detector space by a factor of 40 as compared to classic, dense sampling. The joint optimization of a compressive hyperspectral imager design and the accompanying reconstructor provides an avenue to substantially reduce the data volume from hyperspectral imagers.

Snapshot hyperspectral imaging based on equalization designed DOE.

Hyperspectral imaging attempts to determine distinctive information in spatial and spectral domain of a target. Over the past few years, hyperspectral imaging systems have developed towards lighter and faster. In phase-coded hyperspectral imaging systems, a better coding aperture design can improve the spectral accuracy relatively. Using wave optics, we post an equalization designed phase-coded aperture to achieve desired equalization point spread functions (PSFs) which provides richer features for subsequent image reconstruction. During the reconstruction of images, our raised hyperspectral reconstruction network, CAFormer, achieves better results than the state-of-the-art networks with less computation by substituting self-attention with channel-attention. Our work revolves around the equalization design of the phase-coded aperture and optimizes the imaging process from three aspects: hardware design, reconstruction algorithm, and PSF calibration. Our work is putting snapshot compact hyperspectral technology closer to a practical application.

Aperture-encoded snapshot hyperspectral imaging with a lensless camera

We present a lensless snapshot hyperspectral camera that is capable of hyperspectral imaging over a broad spectrum using a compact and low-cost hardware configuration. We leverage the multiplexing capability of a lensless camera, a novel type of computational imaging device that replaces the lens with a thin mask. Our device utilizes a linear variable filter and a phase mask to encode spectral information onto a monochromatic image sensor, enabling recovery of hyperspectral image stacks from a single measurement by utilizing spectral information encoded in different parts of the 2D point spread function. We perform spectral calibration using a reference color chart and verify the prototype device’s spectral and spatial resolution, as well as its imaging field of view. We report on the design and construction of the device, the image reconstruction algorithm, and spectral calibration methods and present hyperspectral images ranging from 410 to 800 nm obtained with our prototype device.

On-chip hyperspectral image segmentation with fully convolutional networks for scene understanding in autonomous driving

Most of current computer vision-based advanced driver assistance systems (ADAS) perform detection and tracking of objects quite successfully under regular conditions. However, under adverse weather and changing lighting conditions, and in complex situations with many overlapping objects, these systems are not completely reliable. The spectral reflectance of the different objects in a driving scene beyond the visible spectrum can offer additional information to increase the reliability of these systems, especially under challenging driving conditions. Furthermore, this information may be significant enough to develop vision systems that allow for a better understanding and interpretation of the whole driving scene. In this work we explore the use of snapshot, video-rate hyperspectral imaging (HSI) cameras in ADAS on the assumption that the near infrared (NIR) spectral reflectance of different materials can help to better segment the objects in real driving scenarios. To do this, we have used the HSI-Drive 1.1 dataset to perform various experiments on spectral classification algorithms. However, the information retrieval of hyperspectral recordings in natural outdoor scenarios is challenging, mainly because of deficient color constancy and other inherent shortcomings of current snapshot HSI technology, which poses some limitations to the development of pure spectral classifiers. In consequence, in this work we analyze to what extent the spatial features codified by standard, tiny fully convolutional network (FCN) models can improve the performance of HSI segmentation systems for ADAS applications. In order to be realistic from an engineering viewpoint, this research is focused on the development of a feasible HSI segmentation system for ADAS, which implies considering implementation constraints and latency specifications throughout the algorithmic development process. For this reason, it is of particular importance to include the study of the raw image preprocessing stage into the data processing pipeline. Accordingly, this paper describes the development and deployment of a complete machine learning-based HSI segmentation system for ADAS, including the characterization of its performance on different embedded computing platforms, including a single board computer, an embedded GPU SoC and a programmable system on chip (PSoC) with embedded FPGA. We verify the superiority of the FPGA-PSoC over the GPU-SoC in terms of energy consumption and, particularly, processing latency, and demonstrate that it is feasible to achieve segmentation speeds within the range of ADAS industry specifications using standard development tools.

Optical Design and Simulation of Snapshot Hyperspectral Dental Imaging Spectrometer with One-Dimensional Random Coding

To achieve perfect teeth color reproduction, we propose the design of a snapshot hyperspectral imaging spectrometer that can accurately measure the teeth spectrum. In this paper, the optical system of a snapshot hyperspectral dental imaging spectrometer is designed. In particular, to encode images in both the spatial and spectral dimensions, a one-dimensional random coding mask rotated 45 degrees around the optical axis is used. The system has a spectral resolution of approximately 2.2∼10 nm in the 450∼700 nm spectral range, and the total length of the system is less than 150 mm. The whole design can be used as a miniaturized handheld instrument. To verify the feasibility of whiteness measuring, a simulation experiment is carried out. The target spectral information is reconstructed through the reconstruction algorithm. The simulation result shows that spectral imaging has the potential to improve the accuracy and precision of teeth whiteness measurements.

A birefringent spectral demultiplexer enables fast hyper-spectral imaging of protoporphyrin IX during neurosurgery

Hyperspectral imaging and spectral analysis quantifies fluorophore concentration during fluorescence-guided surgery1–6. However, acquisition of the multiple wavelengths required to implement these methods can be time-consuming and hinder surgical workflow. To this end, a snapshot hyperspectral imaging system capable of acquiring 64 channels of spectral data simultaneously was developed for rapid hyperspectral imaging during neurosurgery. The system uses a birefringent spectral demultiplexer to split incoming light and redirect wavelengths to different sections of a large format microscope sensor. Its configuration achieves high optical throughput, accepts unpolarized input light and exceeds channel count of prior image-replicating imaging spectrometers by 4-fold. Tissue-simulating phantoms consisting of serial dilutions of the fluorescent agent characterize system linearity and sensitivity, and comparisons to performance of a liquid crystal tunable filter based hyperspectral imaging device are favorable. The new instrument showed comparable, if not improved, sensitivity at low fluorophore concentrations; yet, acquired wide-field images at more than 70-fold increase in frame rate. Image data acquired in the operating room during human brain tumor resection confirm these findings. The new device is an important advance in achieving real-time quantitative imaging of fluorophore concentration for guiding surgery.

MHealth hyperspectral learning for instantaneous spatiospectral imaging of hemodynamics.

Hyperspectral imaging acquires data in both the spatial and frequency domains to offer abundant physical or biological information. However, conventional hyperspectral imaging has intrinsic limitations of bulky instruments, slow data acquisition rate, and spatiospectral trade-off. Here we introduce hyperspectral learning for snapshot hyperspectral imaging in which sampled hyperspectral data in a small subarea are incorporated into a learning algorithm to recover the hypercube. Hyperspectral learning exploits the idea that a photograph is more than merely a picture and contains detailed spectral information. A small sampling of hyperspectral data enables spectrally informed learning to recover a hypercube from a red-green-blue (RGB) image without complete hyperspectral measurements. Hyperspectral learning is capable of recovering full spectroscopic resolution in the hypercube, comparable to high spectral resolutions of scientific spectrometers. Hyperspectral learning also enables ultrafast dynamic imaging, leveraging ultraslow video recording in an off-the-shelf smartphone, given that a video comprises a time series of multiple RGB images. To demonstrate its versatility, an experimental model of vascular development is used to extract hemodynamic parameters via statistical and deep learning approaches. Subsequently, the hemodynamics of peripheral microcirculation is assessed at an ultrafast temporal resolution up to a millisecond, using a conventional smartphone camera. This spectrally informed learning method is analogous to compressed sensing; however, it further allows for reliable hypercube recovery and key feature extractions with a transparent learning algorithm. This learning-powered snapshot hyperspectral imaging method yields high spectral and temporal resolutions and eliminates the spatiospectral trade-off, offering simple hardware requirements and potential applications of various machine learning techniques.

CTIS-GAN: computed tomography imaging spectrometry based on a generative adversarial network.

Computed tomography imaging spectrometry (CTIS) is a snapshot hyperspectral imaging technique that can obtain a three-dimensional (2D spatial+1D spectral) data cube of the scene captured within a single exposure. The CTIS inversion problem is typically highly ill-posed and is usually solved by time-consuming iterative algorithms. This work aims to take the full advantage of the recent advances in deep-learning algorithms to dramatically reduce the computational cost. For this purpose, a generative adversarial network is developed and integrated with self-attention, which cleverly exploits the clearly utilizable features of zero-order diffraction of CTIS. The proposed network is able to reconstruct a CTIS data cube (containing 31 spectral bands) in milliseconds with a higher quality than traditional methods and the state-of-the-art (SOTA). Simulation studies based on real image data sets confirmed the robustness and efficiency of the method. In numerical experiments with 1000 samples, the average reconstruction time for a single data cube was ∼16m s. The robustness of the method against noise is also confirmed by numerical experiments with different levels of Gaussian noise. The CTIS generative adversarial network framework can be easily extended to solve CTIS problems with larger spatial and spectral dimensions, or migrated to other compressed spectral imaging modalities.

Hyperspectral image segmentation: a preliminary study on the Oral and Dental Spectral Image Database (ODSI-DB)

ABSTRACT Visual discrimination of clinical tissue types remains challenging, with traditional RGB imaging providing limited contrast for such tasks. Hyperspectral imaging (HSI) is a promising technology providing rich spectral information that can extend far beyond three-channel RGB imaging. Moreover, recently developed snapshot HSI cameras enable real-time imaging with significant potential for clinical applications. Despite this, the investigation into the relative performance of HSI over RGB imaging for semantic segmentation purposes has been limited, particularly in the context of medical imaging. Here we compare the performance of state-of-the-art deep learning image segmentation methods when trained on hyperspectral images, RGB images, hyperspectral pixels (minus spatial context), and RGB pixels (disregarding spatial context). To achieve this, we employ the recently released Oral and Dental Spectral Image Database (ODSI-DB), which consists of 215 manually segmented dental reflectance spectral images with 35 different classes across 30 human subjects. The recent development of snapshot HSI cameras has made real-time clinical HSI a distinct possibility, though successful application requires a comprehensive understanding of the additional information HSI offers. Our work highlights the relative importance of spectral resolution, spectral range, and spatial information to both guide the development of HSI cameras and inform future clinical HSI applications.

Bio-inspired Compact, High-resolution Snapshot Hyperspectral Imaging System with 3D Printed Glass Lightguide Array.

To address the major challenges to obtain high spatial resolution in snapshot hyperspectral imaging, 3D printed glass lightguide array has been developed to sample the intermediate image in high spatial resolution and redistribute the pixels in the output end to achieve high spectral resolution. Curved 3D printed lightguide array can significantly simplify the snapshot hyperspectral imaging system, achieve better imaging performance, and reduce the system complexity and cost. We have developed two-photon polymerization process to print glass lightguide array, and demonstrated the system performance with biological samples. This new snapshot technology will catalyze new hyperspectral imaging system development and open doors for new applications from UV to IR.